# tutorials

topic: data-science
tutorial: snakemake
title: Make & Snakemake
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: d7846bceb4d9de84c7f1d6ac7af23c17
last_modified: 2026-02-26T21:12:14.909741
zenodo_link: https://zenodo.org/record/5562251
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/snakemake/tutorial.html

content:
Here you will learn to write both Make and Snakemake workflows. We teach two workflow engines because Snakemake uses a lot of the concepts of Make, and these concepts are somewhat complex and a very different way of thinking than you might be used to with workflow design.

This tutorial is aimed at students and Galaxy community members who might want to convert Snakemake workflows into Galaxy workflows, but need to understand how Snakemake workflows work.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

We're going to go through this tutorial as "the evolution of a pipeline" from a simple bash script that we might run as individual commands on the command line, all the way up to a Snakemake workflow. This should give you some perspective of why the systems exist, and what benefits each system brings.

# Bash

We've set up a simple bash pipeline. It downloads some read files from a website, decompresses them, builds an index for a `genome.fa` we already have, and then does the alignment against that genome.

```bash
# Downloading our datasets
wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz
wget https://zenodo.org/record/5562251/files/SRR2584866_1.fq.gz
wget https://zenodo.org/record/5562251/files/SRR2584866_2.fq.gz
wget https://zenodo.org/record/5562251/files/SRR2589044_1.fq.gz
wget https://zenodo.org/record/5562251/files/SRR2589044_2.fq.gz

# Generate FastQC Report
fastqc *.fq

# Run Trimmomatic to trim the bad reads out.
trimmomatic PE SRR2589044_1.fq.gz SRR2589044_2.fq.gz \
               SRR2589044_1.trim.fq SRR2589044_1un.trim.fq \
               SRR2589044_2.trim.fq SRR2589044_2un.trim.fq \
               SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15

trimmomatic PE SRR2584866_1.fq.gz SRR2589044_2.fq.gz \
               SRR2584866_1.trim.fq SRR2589044_1un.trim.fq \
               SRR2584866_2.trim.fq SRR2589044_2un.trim.fq \
               SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15

# Generate Updated FastQC Report
fastqc *.trim.fq

# Generate the genome index
bwa index GCA_000017985.1_ASM1798v1_genomic.fna

# Align reads to the genome
bwa mem GCA_000017985.1_ASM1798v1_genomic.fna SRR2589044_1.trim.fq SRR2589044_2.trim.fq | samtools sort -O bam -o SRR2589044.bam
bwa mem GCA_000017985.1_ASM1798v1_genomic.fna SRR2584866_1.trim.fq SRR2584866_2.trim.fq | samtools sort -O bam -o SRR2584866.bam
```

This is a fine start to our analysis pipeline, and might simply be summarising
the commands we executed interactively at the command line.

# Make

Make is a build automation tool that has been around since 1976. It lets you describe build processes, writing down the individual processes which will occur, and then providing you a single entrypoint to run your workflow, a lot like how {SciWMS} work! But importantly it is declarative, rather than imperative which is a big change if you're familiar with programming languages like bash or python.

Instead of defining step by step all of the steps that should be executed like the bash example above,
you instead write more general rules on how to produce individual files. We'll look at them one-by-one.

In bash scripts you write something like:
- `command input` and it implicitly creates the output file automatically, often by adding a suffix (e.g. `bwa index genome.fa` produces an index you refer to with `genome`)
- `command input > output` redirecting output of the command to a file
- `command input -O output` where we explicitly state where the output is

In Make you'll need to do that, but you'll also need to declare the inputs and outputs of every step!

<pre class="highlight"><code><span class="nb">output</span>: <span class="kt">input</span>
	<span class="s2">command [and here the correct variant from above]</span>
</code></pre>

This may seem tedious, to annotate the inputs and outputs of each step, but because Make knows what input files need to exist before it can execute this step, and it knows precisely which output files you will generate. It needs this information so that when you have multiple rules, it can decide which dependencies need to be executed.

## Line-By-Line Comparison

### Downloading Data

> <code-in-title>Bash</code-in-title>
> <pre class="highlight"><code><span class="s2">wget https://.../GCA_000017985.1_ASM1798v1_genomic.fna.gz
> wget https://.../SRR2584866_1.fq.gz</span>
> </code></pre>
> With bash we download each read file one by one
{: .code-in}
> <code-out-title>Make</code-out-title>
> <pre class="highlight"><code><span class="nb">%.gz</span>:
> 	<span class="s2">wget https://.../$@ -O $@</span>
> </code></pre>
> But with Make we write is a generic rule which can be used any time you need to download a file for this project. `$@` is used as the name of the output file and in the templated source url we will download from. If you ran `make SRR2584866_1.fq.gz` it would template out the `wget` command and run it. **Speed bonus**: It can parallelise these download jobs for us!
{: .code-out}

### FastQC

> <code-in-title>Bash</code-in-title>
> <pre class="highlight"><code><span class="s2">fastqc *.fq</span></code></pre>
> Generate ALL of the FastQC reports
{: .code-in}
> <code-out-title>Make</code-out-title>
> <pre class="highlight"><code><span class="nb">%.fastqc.html</span>: <span class="kt">%.fq</span>
> 	<span class="s2">fastqc $<</span>
> </code></pre>
> Here is a rule to generate a single FastQC report from a single FastQ file
{: .code-out}

### Trimming Data

> <code-in-title>Bash</code-in-title>
> <pre class="highlight"><code><span class="s2">trimmomatic PE</span> <span class="kt">SRR2589044_1.fq.gz SRR2589044_2.fq.gz</span> \
>                <span class="nb">SRR2589044_1.trim.fq SRR2589044_1un.trim.fq \
>                SRR2589044_2.trim.fq SRR2589044_2un.trim.fq</span> \
>                <span class="s2">SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
>
> <span class="s2">trimmomatic PE</span> <span class="kt">SRR2584866_1.fq.gz SRR2584866_2.fq.gz</span> \
>                <span class="nb">SRR2584866_1.trim.fq SRR2584866_1un.trim.fq \
>                SRR2584866_2.trim.fq SRR2584866_2un.trim.fq</span> \
>                <span class="s2">SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
> </code></pre>
> Run these two individual trimmomatic commands with these hardcoded filenames. If you need to run a new file make sure to carefully change all of the uses of it!
{: .code-in}
> <code-out-title>Make</code-out-title>
> <pre class="highlight"><code><span class="nb">%_1.trim.fq %_2.trim.fq %_1un.trim.fq %_2un.trim.fq</span>: <span class="kt">%_1.fq.gz %_2.fq.gz</span>
> 	<span class="s2">trimmomatic PE $^ \
> 		$(shell basename $(word 1,$^) .fq).trim.fq \
> 		$(shell basename $(word 1,$^) .fq)un.trim.fq \
> 		$(shell basename $(word 2,$^) .fq).trim.fq \
> 		$(shell basename $(word 2,$^) .fq)un.trim.fq \
> 		SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
> </code></pre>
>
> A more generic rule to generate the trimmed fastq files from whichever pair of sequence identifiers were provided. By using `%` we also get some safety in this function that we didn't have in the bash version! We know for sure those are identical values, and no one accidentally made a small typo in any one of the **6** times the same identifier was repeated. <u>But it comes at the cost</u> of using some complex Make statements like `$(shell)` which lets us execute arbitrary shell commands inside our pipeline.
>
> You'll notice that what we're really doing is extracting the inputs and outputs from the command and making them much more uniform
{: .code-out}

So when we translated from Bash to Make, every time

- we annotated inputs
- we annotated outputs
- we made the command a 'template'

> <comment-title>DO NOT REMEMBER THIS</comment-title>
> There is a lot of Make specific things going on in the above examples. You do not need to know it! We just wanted to provide working, correct examples. This tutorial is focused on Snakemake, so read these for context and understanding of why we use Snakemake, not reading them to learn Makefiles.
{: .comment}

## Why Make?

So now comes the question, why Make? Why would you want to write the rules in this declarative way, rather than the imperative way that is so much easier and requires so much less work? *Speed!* Why would you want to write these generic rules that require learning a lot of Make's syntax? *Reusability*.

Speed
:  Every time you want to run your declarative pipeline, you either need to run it from start to finish every time (re-downloading sequence data, re-building genome indicies, re-mapping reads) or to write a lot of bash code to check if those files exist and only downloading/indexing/alinging as-needed.

Reusability
:  That pipeline can only download those specific files, unless you write additional code to template out the name of the reads and genome you want to align. By writing more generic rules, as soon as someone gives you new datasets, you can immediately start processing those with your re-usable pipeline.

However with `make`, you've written generic rules which can be used to download any fastq files. And best of all, make can check if the files already exist, and if they do, it skips the step and goes on to the next step, rather than re-creating the file. Make does this by checking the

Aspect | Bash | Make
--- |--- | ---
Language style | Individual steps performed line-by-line | Generic rules are written which say how to do the operation, but not on which data
Partial re-run | You must run the entire script every time (or write extra code) | Files are only created when you ask for them
How it is invoked | `bash script.sh` | `make`
Running with different data? | Edit the script to replace the identifiers, or support templated identifiers. | `make read1111.sam read2222.sam read3333.same read4444.sam`
Paralellisation? | None by default, you must edit the script to add it. | `make -j 8` runs each build step on one thread, with 8 threads available, until all tasks are finished.
Filename | Anything ending in `.sh` | `Makefile` is the default name, and you should name your makefile this, unless you want people to have to type `make -f other-file.mk`
Dependencies | Up to you to manage | Up to you to manage
Multiple output files per step | n/a | [Very tricky to get completely right](https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html), Makefiles really expect one rule produces one output file (and can only check e.g. update times of a single file.)
Cluster/HPC Friendliness | Everything is manual | Everything is manual

## Backwards

A full makefile example

```
all: SRR2589044.bam

# Here we've hardcoded the genome name because it's less likely to change for a
# single pipeline than the individual data files are.
%.bam: %_1.trim.fq %_1.trim.fq GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt
	bwa mem GCA_000017985.1_ASM1798v1_genomic.fna $(word 1,$^) $(word 2,$^) | \
		samtools sort -O bam -o $@

# This indexing step however will work for any possible
%.fna.gz.bwt: %.fna.gz
	bwa index $<

# This handles ALL fastqc reporting, so we don't have to do it in two sections,
# but, unless we ask for these reports they won't be generated.
%.fastqc.html: %.fq
	fastqc $<

# This rule violates some Makefile internal expectations by having multiple
# outputs which is not handled well by all implementations
%_1.trim.fq %_2.trim.fq %_1un.trim.fq %_2un.trim.fq: %_1.fq.gz %_2.fq.gz
	trimmomatic PE $^ \
		$(shell basename $(word 1,$^) .fq).trim.fq \
		$(shell basename $(word 1,$^) .fq)un.trim.fq \
		$(shell basename $(word 2,$^) .fq).trim.fq \
		$(shell basename $(word 2,$^) .fq)un.trim.fq \
		SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15

# And here's finally our download step
%:
	https://zenodo.org/record/5562251/files/$(shell basename $@) \
		-O $@
```

You'll notice a couple things about this above example:

1. There is a new rule called `all`. In a makefile, by default, the very first rule is executed when you run `make`. If you want to execute other rules you can, but it defaults to the first one. By convention, many people name it `all`.
2. Inside there we've also written a file we'd like created, `SRR2589044.bam`, which doesn't exist yet. Make sees this as a dependency to finishing the (empty) all rule, and then goes on to figure out how to create it.
3. It is written backwards, we've started with what we want to output, and for each line, we figured out what we needed for that, and wrote a rule on how to create it. This is relatively common in makefiles.

You'll also notice some weird additional things we've had to do like `$(word 2,$^)` to get the second input file to that rule, this is really kind of ugly and hard to understand and is a great motivation for learning Snakemake which helps address these issues.

Make will read the above makefile like so:

1. You're running `make` so I will find the first task and run it
2. `all` is the first task and it depends on this BAM file
3. The BAM file needs me to run bwa-mem and bwa-index, in two separate steps
	1. For bwa index I need to index the genome
	2. And I need to download it first!
4. Also I need the trimmed fastq files, both forward and reverse files.
5. And I need to have the untrimmed files to trim them
6. And I need to download those first!

Notice that Make isn't running every task, it's reading the one task you asked for, and seeing what's required for that based on your annotations of inputs and outputs.

> <question-title>Check your understanding</question-title>
>
> 1. If a file is not part of the final output, or not the requested task, will it be created?
> 2. Do you spot any omissions in the above pipeline?
>
> > <solution-title></solution-title>
> > 1. No, Make only runs the tasks for files it needs, it won't run any of the other tasks. This is part of what makes `make` fast and easily parallelisable.
> > 2. Yes! FastQC is missing, it won't be created unless we ask for it.
> {: .solution}
{: .question}

Reading the above you should be able to imagine a tree of tasks that Make is creating internally:

TODO

This is called a {DAG}, it is a graph of nodes (tasks that need to be executed), with connections between nodes that have a direction (this task depends on outputs of that task), and there are no cycles (no outputs depend on inputs.) These are very common in {SciWMS}s because they make computation faster. Instead of executing step-by-step, we can build this graph of what work needs to be done, and then starting with the leaves of the graph (the end nodes without other dependencies) we can start executing and removing them.

This also how we can really easily parallelise workflows: because we know the dependencies of each step, we know which can be executed right now, and we can execute them in parallel because we know for sure they do not depend on each other.

# Snakemake

> The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language. They can be seamlessly scaled to server, cluster, grid and cloud environments, without the need to modify the workflow definition. Finally, Snakemake workflows can entail a description of required software, which will be automatically deployed to any execution environment.
{: .quote cite="https://snakemake.readthedocs.io/en/stable/index.html"}

Snakemake addresses a lot of the issues with make for use in scientific contexts: clearer pipelines and dependencies. We did not talk about it in the previous section, but where did `bowtie2` and `bowtie2-build` come from? How did those get installed? What versions are they? None of that information is included in the Makefile

Snakemake rules are a bit more complex, in Snakemake you will write rules that follow this form:

<pre class="highlight"><code>rule my-rule-name
	<span class="kt">input:
		"something",
		"something-else"</span>
	<span class="nb">output:
		"output-1",
		"output-2"</span>
	<span class="s2">shell:
		"cat {input} > {output}"</span>
</code></pre>

## Line-By-Line Comparison

### Downloading Data

> <code-in-title>Make</code-in-title>
> <pre class="highlight"><code><span class="nb">%.fq.gz</span>:
> 	<span class="s2">wget https://zenodo.org/record/5562251/files/$@</span>
> </code></pre>
>
> Generic download rule, the <code>$@</code> and <code>%</code> used are a bit opaque, you need to know what they mean to understand how the rule works.
{: .code-in}
> <code-out-title>Snakemake</code-out-title>
> <pre class="highlight"><code>rule download:
> 	<span class="nb">output:
> 		"{sample}.fq.gz"</span>
> 	<span class="s2">shell:
> 		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output}"</span>
> </code></pre>
>
> This is much more explicit, the outputs are listed and `{sample}` is used as the variable to be templated out, a lot like you might recognise from Python's `format` function or `f""` strings. The rule also has a name which serves as really nice documentation for what that step does, you don't have to read the command to figure it out.
{: .code-out}

### FastQC

> <code-in-title>Make</code-in-title>
> <pre class="highlight"><code><span class="nb">%.fastqc.html</span>: <span class="kt">%.fq</span>
> 	<span class="s2">fastqc $<</span>
> </code></pre>
>
> Here is a rule to generate a single FastQC report from a single FastQ file
{: .code-in}
> <code-out-title>Snakemake</code-out-title>
> <pre class="highlight"><code>rule fastqc:
> 	<span class="kt">input:
> 		"{sample}.fq.gz"</span>
> 	<span class="nb">output:
> 		"{sample}_fastqc.html"</span>
> 	<span class="s2">shell:
> 		"fastqc {input} --outdir fastqc/"</span>
> </code></pre>
>
> Essentially the same, but now we've also added a Conda environment in which our job will run. This makes dependency management a lot simpler.
{: .code-out}


### Trimming Data


> <code-in-title>Make</code-in-title>
> <pre class="highlight"><code><span class="nb">%_1.trim.fq %_2.trim.fq %_1un.trim.fq %_2un.trim.fq</span>: <span class="kt">%_1.fq.gz %_2.fq.gz</span>
> 	<span class="s2">trimmomatic PE $^ \
> 		$(shell basename $(word 1,$^) .fq).trim.fq \
> 		$(shell basename $(word 1,$^) .fq)un.trim.fq \
> 		$(shell basename $(word 2,$^) .fq).trim.fq \
> 		$(shell basename $(word 2,$^) .fq)un.trim.fq \
> 		SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15</span>
> </code></pre>
>
> Here we take our very complicated and hard to understand Make rule (shell? basename? word?) with ugly and potentially quite broken multiple output syntax
>
{: .code-in}
> <code-out-title>Snakemake</code-out-title>
> <pre class="highlight"><code>rule trimmomatic:
> 	<span class="kt">input:
> 		r1="{sample}_1.fq.gz",
> 		r2="{sample}_2.fq.gz"</span>
> 	<span class="nb">output:
> 		o1="{sample}_1.trim.fq",
> 		o2="{sample}_2.trim.fq",
> 		o1un="{sample}_1un.trim.fq",
> 		o2un="{sample}_2un.trim.fq"</span>
> 	<span class="s2">shell:
> 		"trimmomatic PE "
> 		"{input.r1} {input.r2} "
> 		"{output.o1} {output.o1un} "
> 		"{output.o2} {output.o2un} "
> 		"SLIDINGWINDOW:4:20 MINLEN:25 "
> 		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15"</span>
> </code></pre>
>
> And turn it into a much more readable and clear Snakemake step! Now instead of using more opaque terms like `$(word 1,$^)` we can just declare "our output should have this name and use {sample} as part of the name" and Snakemake takes care of the rest. Then in our commandline we can clearly reference exactly what we want.
>
> Note: We also use `fq.gz` as our input file because trimmomatic we know can accept gzipped fastq files, but we output plain fastq files because re-compressing those is another step and it's not necessary right now for our small datasets. In a real world situation you might also choose not to compress intermediate datasets because you want better performance and you know you'll throw away the intermediates and keep the source and final datasets.
{: .code-out}

Now that you have seen a few rules, let's write the rest.

> <hands-on-title>Save this Snakefile to the filesystem</hands-on-title>
> From here on you are going to finish writing this Snakefile on your own! We'll give you the bits you have seen up until now, and future tasks will require you to add your own rules and get to actually test out the pipeline!
>
> ```Snakemake
> rule download:
> 	output:
> 		"{sample}.fq.gz"
> 	shell:
> 		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output}"
>
> rule download_genome:
> 	output:
> 		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"
> 	shell:
> 		"wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O {output}"
>
> rule fastqc:
> 	input:
> 		"{sample}.fq.gz"
> 	output:
> 		"{sample}_fastqc.html"
> 	shell:
> 		"fastqc {input} --outdir fastqc/"
>
> rule trimmomatic:
> 	input:
> 		r1="{sample}_1.fq.gz",
> 		r2="{sample}_2.fq.gz"
> 	output:
> 		o1="{sample}_1.trim.fq",
> 		o2="{sample}_2.trim.fq",
> 		o1un="{sample}_1un.trim.fq",
> 		o2un="{sample}_2un.trim.fq"
> 	shell:
> 		"trimmomatic PE "
> 		"{input.r1} {input.r2} "
> 		"{output.o1} {output.o1un} "
> 		"{output.o2} {output.o2un} "
> 		"SLIDINGWINDOW:4:20 MINLEN:25 "
> 		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15"
> ```
{: .hands_on}

> <hands-on-title>Install Snakemake</hands-on-title>
> We're about to start doing things really with snakemake, so, it's time to install it.
> 1. [Install Miniconda](https://docs.conda.io/en/latest/miniconda.html)
> 2. Create an environment for Snakemake:
>
>    ```bash
>    conda create -n snakemake
>    ```
>
> 3. Activate it
>
>    ```bash
>    conda activate snakemake
>    ```
>
> 4. And install snakemake in this environment
>
>    ```bash
>    conda install snakemake=6.10.0
>    ```
>
{: .hands_on}


> <hands-on-title>Try running snakemake!</hands-on-title>
>
> 1. Try running snakemake
>
>    > <code-in-title>CLI</code-in-title>
>    > ```
>    > snakemake -c1
>    > ```
>    {: .code-in}
>
>    > <tip-title>"specify the maximum number of jobs"</tip-title>
>    > If you see an error like this it might be due to an outdated version of snakemake
>    > ```
>    > Error: you need to specify the maximum number of jobs to be queued or executed at the same time with --jobs.
>    > ```
>    {: .tip}
>
>    > <code-out-title></code-out-title>
>    > ```
>    > Building DAG of jobs...
>    > WorkflowError:
>    > Target rules may not contain wildcards. Please specify concrete files or a rule without wildcards at the command line, or have a rule without wildcards at the very top of your workflow (e.g. the typical "rule all" which just collects all results you want to generate in the end).
>    > ```
>    {: .code-out}
>
>    Wait, that didn't work! This is the `all` rule we also saw in the makefile. But we can use Snakemake to build individual rule outputs, so let's try that now.
>
> 2. Run snakemake to download `SRR2584866_1.fq.gz` and `SRR2584866_2.fq.gz`
>
>    > <code-in-title>CLI</code-in-title>
>    > ```
>    > snakemake -c1 SRR2584866_1.fq.gz SRR2584866_2.fq.gz
>    > ```
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```
>    > Building DAG of jobs...
>    > Using shell: /usr/bin/bash
>    > Provided cores: 1 (use --cores to define parallelism)
>    > Rules claiming more threads will be scaled down.
>    > Conda environments: ignored
>    > Job stats:
>    > job         count    min threads    max threads
>    > --------  -------  -------------  -------------
>    > download        2              1              1
>    > total           2              1              1
>    >
>    > Select jobs to execute...
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > rule download:
>    >     output: SRR2584866_2.fq.gz
>    >     jobid: 1
>    >     wildcards: sample=SRR2584866_2
>    >     resources: tmpdir=/tmp
>    >
>    > --2021-10-08 16:06:32--  https://zenodo.org/record/5562251/SRR2584866_2.fq.gz
>    > Resolving localhost (localhost)... 127.0.0.1
>    > Connecting to localhost (localhost)|127.0.0.1|:8000... connected.
>    > HTTP request sent, awaiting response... 200 OK
>    > Length: 6319969 (6,0M) [application/gzip]
>    > Saving to: ‘SRR2584866_2.fq.gz’
>    >
>    > 2021-10-08 16:06:32 (538 MB/s) - ‘SRR2584866_2.fq.gz’ saved [6319969/6319969]
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > Finished job 1.
>    > 1 of 2 steps (50%) done
>    > Select jobs to execute...
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > rule download:
>    >     output: SRR2584866_1.fq.gz
>    >     jobid: 0
>    >     wildcards: sample=SRR2584866_1
>    >     resources: tmpdir=/tmp
>    >
>    > --2021-10-08 16:06:32--  https://zenodo.org/record/5562251/SRR2584866_1.fq.gz
>    > Resolving localhost (localhost)... 127.0.0.1
>    > Connecting to localhost (localhost)|127.0.0.1|:8000... connected.
>    > HTTP request sent, awaiting response... 200 OK
>    > Length: 6586718 (6,3M) [application/gzip]
>    > Saving to: ‘SRR2584866_1.fq.gz’
>    >
>    > 2021-10-08 16:06:32 (537 MB/s) - ‘SRR2584866_1.fq.gz’ saved [6586718/6586718]
>    >
>    > [Fri Oct  8 16:06:32 2021]
>    > Finished job 0.
>    > 2 of 2 steps (100%) done
>    > Complete log: /tmp/snake.q5mqtdhfg6/.snakemake/log/2021-10-08T160632.213680.snakemake.log
>    > ```
>    {: .code-out.code-max-300}
>
> 3. Check that it worked. Do you see files in your directory?
{: .hands_on}


Now that we've got our pipeline started, let's do some more with it!

> <question-title></question-title>
>
> How would you write the following task in Snakemake?
>
> The command is
> <pre class="highlight"><code><span class="s2">bwa index</span> <span class="kt">GCA_000017985.1_ASM1798v1_genomic.fna</span></code></pre>
> and it creates `GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt`
>
> > <solution-title></solution-title>
> >
> > <pre class="highlight"><code>rule indexgenome:
> > 	<span class="kt">input:
> > 		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"</span>
> > 	<span class="nb">output:
> > 		"GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt"</span>
> > 	<span class="s2">shell:
> > 		"bwa index {input}"</span>
> > </code></pre>
> {: .solution}
{: .question}

> <question-title></question-title>
>
> The command is `bwa mem GCA_000017985.1_ASM1798v1_genomic.fna SRR2584866_1.trim.fq SRR2584866_2.trim.fq | samtools sort -O bam -o SRR2584866.bam`
>
> 1. What are the inputs?
> 2. What are the outputs?
> 3.  How would you write the following task in Snakemake?
>
>
> > <solution-title></solution-title>
> >
> > 1. `GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt`, the index file (but beware it does not get passed in as-is, the indexing tool expects just the `GCA_000017985.1_ASM1798v1_genomic.fna` portion.)
> >
> >    Also we have our two sequence files `SRR2584866_1.trim.fq SRR2584866_2.trim.fq`
> >
> > 2. Our output is `SRR2584866.bam`
> >
> > 3. There are a couple of options we have here, we can supply both the `.fna` and the `.fna.gz.bwt` file as inputs (not strictly true, we don't need the fasta file) and then just use the `fna` file in the command line, or we can pass in just the `.fna.gz.bwt` file and try and calculate the `.fna` version that is expected as the index name. We will show the second option as it is more complicated.
> > <pre class="highlight"><code>rule align:
> > 	<span class="kt">input:
> > 		r1="{sample}_1.trim.fq",
> > 		r2="{sample}_1.trim.fq",
> > 		index="{genome}.fna.gz.bwt"</span>
> > 	<span class="nb">output:
> > 		"{genome}/{sample}.bam"</span>
> > 	<span class="s2">shell:
> > 		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} | "
> > 		"samtools sort -O bam -o {output}"</span>
> > </code></pre>
> >
> > Here we used a number of features to accomplish what we need, and we'll now go through them. First is [Wildcards](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#wildcards) which can be used to take a portion of the output name or a portion of the input name and to re-use that in the command line. Here we declared that the first part of the index name up to `.fna.gz.bwt` was going to be the `genome` wildcard.
> >
> > Importantly, we also used this in our output. What would have happened if we didn't? It would be unresolvable! We would run `snakemake ... output.bam` and it would say "I don't know what value genome should be set to", so we need to have that value somewhere in our output filename in order to be able to figure that out.
> >
> > That isn't the only way to solve that problem, we could also hardcode this or write it in a [config file](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html) that is used by snakemake.
> {: .solution}
{: .question}

> <hands-on-title>Add the above outputs to your Pipeline</hands-on-title>
> If you haven't already, add the above outputs to your Snakemake pipeline. You should now have a few rules:
>
> - download
> - fastqc
> - trimmomatic
> - indexgenome
> - align
{: .hands_on}

## Best Practices

But this was our very first attempt at a workflow, so what might a best practice workflow look like?

### Conda for reproducibility

If you're not already using conda, you should be! Much scientific software you might be interested in using is already in there, probably mostly provided by the [BioConda](https://anaconda.org/bioconda/repo) repository. Writing an environment file can be pretty simple, you just need a file like this, which we customarily put in a folder named `envs/`

> <code-in-title>`envs/bwa.yaml`</code-in-title>
> ```yaml
> channels:
>   - conda-forge
>   - bioconda
>   - defaults
> dependencies:
>   - bwa=0.7.17
>   - samtools=1.13
> ```
{: .code-in}

In the above code sample you can see `bwa=0.7.17`, that's the version of the `bwa` mapper we want to use which was found on [the conda repository page](https://anaconda.org/bioconda/bwa). You'll notice that after it is `=h5bf99c6_8`, this is a specific revision of the package that has been 'pinned' (i.e. we want to ensure conda installs specifically the version we know works.) That step is not necessary, but if you really care about reproducibility it can be good in a pipeline

> <hands-on-title>Add conda</hands-on-title>
> Now you should add Conda environments where appropriate. But where is appropriate? Unix built in tools (wget, curl, gzip, cat, head, tail) generally do not need to be in conda, because they're very standard across all unix environments.
>
> Bioinformatics tools on the other hand, these need conda envs.
>
> Try and add these yourself, and check your work below.
>
> Please use the following versions:
>
> Package     | Version
> ---         | --
> fastqc      | 0.11.9
> trimmomatic | 0.39
> bwa         | 0.7.17
> samtools    | 1.13
>
> *Hints*
> 1. see above for what a conda environment looks like.
> 2. You'll need to install both `bwa` and `samtools` into the environment for the alignment step.
> 3. Create the `envs/` directory if it does not exist.
>
> > <solution-title></solution-title>
> >
> > `envs/bwa.yaml` should look like this:
> >
> > ```yaml
> > channels:
> >   - conda-forge
> >   - bioconda
> >   - defaults
> > dependencies:
> >   - bwa=0.7.17
> >   - samtools=1.13
> > ```
> >
> > And here is the `envs/fastqc.yaml`:
> >
> > ```yaml
> > channels:
> >   - conda-forge
> >   - bioconda
> >   - defaults
> > dependencies:
> >   - fastqc=0.11.9
> > ```
> >
> > And here is the `envs/trimmomatic.yaml`:
> >
> > ```yaml
> > channels:
> >   - conda-forge
> >   - bioconda
> >   - defaults
> > dependencies:
> >   - trimmomatic=0.39
> > ```
> >
> >
> > > <comment-title>How to read diffs</comment-title>
> > > This is a 'diff', it shows you the difference between two versions of a text file. Everything added is highlighed in light blue. Anything deleted is shown in black with a strikethrough. Importantly it shows you the context, the bits that are not highlighted, and this helps you know where the changes should go.
> > > The first line shows the original file, and the second line shows the new file. If the names are different, it means the file has been renamed.
> > >
> > > The `@@ ... @@` line shows the position in the file and how that changes, and often it will also show you the name of the function or in the case of Snakemake, the name of the rule that change appears in.
> > {: .comment}
> >
> > ```diff
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -9,6 +9,8 @@ rule fastqc:
> >  		"{sample}.fq.gz"
> >  	output:
> >  		"{sample}_fastqc.html"
> > +	conda:
> > +		"envs/fastqc.yaml"
> >  	shell:
> >  		"fastqc {input} --outdir fastqc/"
> >
> > @@ -21,6 +23,8 @@ rule trimmomatic:
> >  		o2="{sample}_2.trim.fq",
> >  		o1un="{sample}_1un.trim.fq",
> >  		o2un="{sample}_2un.trim.fq"
> > +	conda:
> > +		"envs/trimmomatic.yaml"
> >  	shell:
> >  		"trimmomatic PE "
> >  		"{input.r1} {input.r2} "
> > @@ -34,6 +38,8 @@ rule indexgenome:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"
> >  	output:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt"
> > +	conda:
> > +		"envs/bwa.yaml"
> >  	shell:
> >  		"bwa index {input}"
> >
> > @@ -44,6 +50,8 @@ rule align:
> >  		index="{genome}.fna.gz.bwt"
> >  	output:
> >  		"{genome}/{sample}.bam"
> > +	conda:
> > +		"envs/bwa.yaml"
> >  	shell:
> >  		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} | "
> >  		"samtools sort -O bam -o {output}"
> > ```
> >
> > Here in the last two steps we used a single environment, this is a technical decision we made. We could have used a single environment for every step, but with conda the more packages you add, the more complicated it is for Conda to find a version of all of those packages that work together OK. So by isolating packages to single environments, you speed up the installation process. But in two out of four steps above, we have used a shared environment. This is because `bwa` was required in both, and one just needs the additional `samtools` package. For this there is no real technical reason, just a feeling of "it's probably ok and won't be too difficult to resolve".
> {: .solution}
{: .hands_on}

### Log Files

Saving log files is key to making sure that you have a complete log of the execution of software. If you don't save the log files, and you're running a pipeline with 100 samples, it can be very easy to get confused about what went wrong and where. Unfortunately this is not a completely automatic process, and you'll need to manually configure it:

> <code-in-title>`Snakefile`</code-in-title>
> ```
> rule indexgenome:
> 	input:
> 		"{sample}.fna.gz"
> 	output:
> 		"{sample}.fna.gz.bwt"
> 	conda:
> 		"envs/bwa.yaml"
> 	log:
> 		out="logs/bwa.index.{sample}.out",
> 		err="logs/bwa.index.{sample}.err"
> 	shell:
> 		"bwa index {input} >{log.out} 2>{log.err}"
> ```
{: .code-in}

In the above rule we setup an `out` and `err` log file because every program that executes on Linux has two output streams known as `stderr` and `stdout`. By default these are just printed to your screen, and can be mixed together which is unhelpful when your program writes a lot of logging messages and you have to dig through it to find the one error message. So we setup the `out` and `err` files in a `logs/` directory with the name of the tool that will be executed as well.

Additionally we need to change the `shell` step to redirect `stdout` and `stderr` to the appropriate files. Please see [this bash tutorial](https://wiki.bash-hackers.org/howto/redirection_tutorial) for more information on how redirecting outputs works.

But **don't just copy/paste** the above example because:

- Some tools produce output on stdout, e.g. `bwa mem .. | samtools`, there `bwa mem` is writing to `stdout` which is getting piped to `samtools`, so, you should not redirect its output into the logging file or your pipeline will break. There you would only want to redirect `2>{log.err}`.
- Sometimes you have multiple commands in a pipeline, these all should go to different log files or you will have more trouble figuring out where your error came from.


> <hands-on-title>Add log files to every step</hands-on-title>
>
> Try and add log files everywhere (err and out where appropriate) to all of your rules. And use them in the `shell` sections as well. Also put your logs under the `logs/` folder!
>
> > <solution-title></solution-title>
> >
> > This is again a diff, things in blue were added to the file named at the top, things with black highlight were deleted. Think "track changes" mode in Google Docs or Word, except harder to read.
> >
> > ```diff
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -1,14 +1,20 @@
> >  rule download:
> >  	output:
> >  		"{sample}.fq.gz"
> > +	log:
> > +		out="logs/download.{sample}.out",
> > +		err="logs/download.{sample}.err"
> >  	shell:
> > -		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output}"
> > +		"wget https://zenodo.org/record/5562251/files/{wildcards.sample}.fq.gz -O {output} >{log.out} 2>{log.err}"
> >
> >  rule download_genome:
> >  	output:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz"
> > +	log:
> > +		out="logs/download.out",
> > +		err="logs/download.err"
> >  	shell:
> > -		"wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O {output}"
> > +		"wget https://zenodo.org/record/5562251/files/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O {output} >{log.out} 2>{log.err}"
> >
> >  rule fastqc:
> >  	input:
> > @@ -17,8 +23,11 @@ rule fastqc:
> >  		"{sample}_fastqc.html"
> >  	conda:
> >  		"envs/fastqc.yaml"
> > +	log:
> > +		out="logs/fastqc.{sample}.out",
> > +		err="logs/fastqc.{sample}.err"
> >  	shell:
> > -		"fastqc {input} --outdir fastqc/"
> > +		"fastqc {input} --outdir fastqc/ >{log.out} 2>{log.err}"
> >
> >  rule trimmomatic:
> >  	input:
> > @@ -31,13 +40,16 @@ rule trimmomatic:
> >  		o2un="{sample}_2un.trim.fq"
> >  	conda:
> >  		"envs/trimmomatic.yaml"
> > +	log:
> > +		out="logs/trimmomatic.{sample}.out",
> > +		err="logs/trimmomatic.{sample}.err"
> >  	shell:
> >  		"trimmomatic PE "
> >  		"{input.r1} {input.r2} "
> >  		"{output.o1} {output.o1un} "
> >  		"{output.o2} {output.o2un} "
> >  		"SLIDINGWINDOW:4:20 MINLEN:25 "
> > -		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15"
> > +		"ILLUMINACLIP:NexteraPE-PE.fa:2:40:15 >{log.out} 2>{log.err}"
> >
> >  rule indexgenome:
> >  	input:
> > @@ -46,8 +58,11 @@ rule indexgenome:
> >  		"GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt"
> >  	conda:
> >  		"envs/bwa.yaml"
> > +	log:
> > +		out="logs/bwa.index.out",
> > +		err="logs/bwa.index.err"
> >  	shell:
> > -		"bwa index {input}"
> > +		"bwa index {input} >{log.out} 2>{log.err}"
> >
> >  rule align:
> >  	input:
> > @@ -58,6 +73,10 @@ rule align:
> >  		"{genome}/{sample}.bam"
> >  	conda:
> >  		"envs/bwa.yaml"
> > +	log:
> > +		bwaerr="logs/bwa.{genome}.{sample}.err",
> > +		out="logs/samtools.{genome}.{sample}.out",
> > +		err="logs/samtools.{genome}.{sample}.err"
> >  	shell:
> > -		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} | "
> > -		"samtools sort -O bam -o {output}"
> > +		"bwa mem {wildcards.genome}.fna {input.r1} {input.r2} 2>{log.bwaerr} | "
> > +		"samtools sort -O bam -o {output} >{log.out} 2>{log.err}"
> > ```
> >
> > The last one is the most complicated. Here we have `bwa mem` which writes to `stdout`. We know that because a `|` character comes next which indicates we're piping the output from one program to another, so there we want to store the error. However `samtools` has a `-o {output}` so we can infer from this that it writes to that output file by name. Anything it writes on `stdout` or `stderr` migth be important.
> {: .solution}
{: .hands_on}



### Use Folders

You've seen a couple examples above but it's best to use folders to help keep your data organised. Separate individual pipeline steps into different folders so you can more easily keep track of e.g. whether you're working with trimmed or untrimmed data.

> <question-title></question-title>
> What sort of folders would you establish for the example pipeline in this tutorial?
>
> > <solution-title></solution-title>
> >
> > One possible solution you could consider is:
> >
> > Folder | Purpose
> > --- | ---
> > envs | Conda environment files
> > reads | Your sequencing data
> > trim | Your sequences after they have been trimmed and cleaned
> > fastqc | The FastQC reports
> > alignments | All of your final output alignments
> > reference | Any reference data, indicies, etc.
> {: .solution}
{: .question}

> <hands-on-title>Do it!</hands-on-title>
> Update your pipeline to have all of datasets stored nicely in folders.
>
> > <solution-title></solution-title>
> >
> > ```diff
> > diff --git a/Snakefile b/Snakefile
> > index 59b983c..31ebdff 100644
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -1,6 +1,6 @@
> >  rule download:
> >  	output:
> > -		"{sample}.fq.gz"
> > +		"reads/{sample}.fq.gz"
> >  	log:
> >  		out="logs/download.{sample}.out",
> >  		err="logs/download.{sample}.err"
> > @@ -9,7 +9,7 @@ rule download:
> >
> >  rule download_genome:
> >  	output:
> > -		"GCA_000017985.1_ASM1798v1_genomic.fa.gz"
> > +		"reference/GCA_000017985.1_ASM1798v1_genomic.fa.gz"
> >  	log:
> >  		out="logs/download.{sample}.out",
> >  		err="logs/download.{sample}.err"
> > @@ -18,9 +18,9 @@ rule download_genome:
> >
> >  rule fastqc:
> >  	input:
> > -		"{sample}.fq"
> > +		"reads/{sample}.fq.gz"
> >  	output:
> > -		"{sample}_fastqc.html"
> > +		"fastqc/{sample}_fastqc.html"
> >  	conda:
> >  		"envs/fastqc.yaml"
> >  	log:
> > @@ -31,13 +31,13 @@ rule fastqc:
> >
> >  rule trimmomatic:
> >  	input:
> > -		r1="{sample}_1.fq",
> > -		r2="{sample}_2.fq"
> > +		r1="reads/{sample}_1.fq",
> > +		r2="reads/{sample}_2.fq"
> >  	output:
> > -		o1="{sample}_1.trim.fq",
> > -		o2="{sample}_2.trim.fq",
> > -		o1un="{sample}_1un.trim.fq",
> > -		o2un="{sample}_2un.trim.fq"
> > +		o1="trim/{sample}_1.trim.fq",
> > +		o2="trim/{sample}_2.trim.fq",
> > +		o1un="trim/{sample}_1un.trim.fq",
> > +		o2un="trim/{sample}_2un.trim.fq"
> >  	conda:
> >  		"envs/trimmomatic.yaml"
> >  	log:
> > @@ -53,9 +53,9 @@ rule trimmomatic:
> >
> >  rule indexgenome:
> >  	input:
> > -		"GCA_000017985.1_ASM1798v1_genomic.fna"
> > +		"reference/GCA_000017985.1_ASM1798v1_genomic.fa"
> >  	output:
> > -		"GCA_000017985.1_ASM1798v1_genomic.fna.bwt"
> > +		"reference/GCA_000017985.1_ASM1798v1_genomic.fa.bwt"
> >  	conda:
> >  		"envs/bwa.yaml"
> >  	log:
> > @@ -66,11 +66,11 @@ rule indexgenome:
> >
> >  rule align:
> >  	input:
> > -		r1="{sample}_1.trim.fq",
> > -		r2="{sample}_1.trim.fq",
> > -		index="{genome}.fna.bwt"
> > +		r1="trim/{sample}_1.trim.fq",
> > +		r2="trim/{sample}_1.trim.fq",
> > +		index="reference/{genome}.fna.bwt"
> >  	output:
> > -		"{genome}/{sample}.bam"
> > +		"alignments/{genome}/{sample}.bam"
> >  	conda:
> >  		"envs/bwa.yaml"
> >  	log:
> > @@ -78,5 +78,5 @@ rule align:
> >  		out="logs/samtools.{genome}.{sample}.out",
> >  		err="logs/samtools.{genome}.{sample}.err"
> >  	shell:
> > -		"bwa mem {wildcard.genome}.fna {input.r1} {input.r2} 2>{log.bwaerr} | "
> > +		"bwa mem reference/{wildcards.genome}.fna.gz {input.r1} {input.r2} 2>{log.bwaerr} | "
> >  		"samtools sort -O bam -o {output} >{log.out} 2>{log.err}"
> > ```
> {: .solution}
{: .hands_on}

Notice how while updating data location, we only had to update the input and output boxes of the Snakefile, we didn't need to change any commands because there we used `{input}` or `{output}` which are automatically templated for us. Except for the last command where we had to use a wildcard and manually construct the path to work around the issue of 'fake' inputs where the tool expected a filename that was the common subset of all of the files it would produce.


### Set a Default Task

In the Makefile we had an `all` rule which was the first and default action to take. Let's reproduce that.

> <hands-on-title>Add an all task</hands-on-title>
> It should build `alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam`
>
> > <solution-title></solution-title>
> > We add this to the very top of our file.
> > ```diff
> > --- a/Snakefile
> > +++ b/Snakefile
> > @@ -1,3 +1,7 @@
> > +rule all:
> > +	input:
> > +		"alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam"
> > +
> >  rule download:
> >  	output:
> >  		"reads/{sample}.fq.gz"
> > ```
> >
> > > <tip-title>Why Input and not Output?</tip-title>
> > > Because rule all is taking in some other outputs as an input to it. If we used outputs, rule all would do nothing (it has no pre-requisite steps), and since it produces nothing by itself, snakemake would see that the outputs you expected have not been created and flag it as an error.
> > {: .tip}
> >
> {: .solution}
{: .hands_on}



## Final Pipeline

This is starting to look like a pretty good workflow! Let's preview how it will run:


> <code-in-title></code-in-title>
> ```
> $ snakemake -np
> ```
{: .code-in}
> <code-out-title></code-out-title>
> ```
> Building DAG of jobs...
> Job stats:
> job                count    min threads    max threads
> ---------------  -------  -------------  -------------
> align                  1              1              1
> all                    1              1              1
> download               2              1              1
> download_genome        1              1              1
> indexgenome            1              1              1
> trimmomatic            1              1              1
> total                  7              1              1
>
>
> [Fri Oct  8 16:59:39 2021]
> rule download:
>     output: reads/SRR2584863_2.fq.gz
>     log: logs/download.SRR2584863_2.out, logs/download.SRR2584863_2.err
>     jobid: 4
>     wildcards: sample=SRR2584863_2
>     resources: tmpdir=/tmp
>
> wget https://zenodo.org/record/5562251/SRR2584863_2.fq.gz -O reads/SRR2584863_2.fq.gz >logs/download.SRR2584863_2.out 2>logs/download.SRR2584863_2.err
>
> [Fri Oct  8 16:59:39 2021]
> rule download:
>     output: reads/SRR2584863_1.fq.gz
>     log: logs/download.SRR2584863_1.out, logs/download.SRR2584863_1.err
>     jobid: 3
>     wildcards: sample=SRR2584863_1
>     resources: tmpdir=/tmp
>
> wget https://zenodo.org/record/5562251/SRR2584863_1.fq.gz -O reads/SRR2584863_1.fq.gz >logs/download.SRR2584863_1.out 2>logs/download.SRR2584863_1.err
>
> [Fri Oct  8 16:59:39 2021]
> rule download_genome:
>     output: reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz
>     log: logs/download.out, logs/download.err
>     jobid: 6
>     resources: tmpdir=/tmp
>
> wget https://zenodo.org/record/5562251/GCA_000017985.1_ASM1798v1_genomic.fna.gz -O reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz >logs/download.out 2>logs/download.err
>
> [Fri Oct  8 16:59:39 2021]
> rule trimmomatic:
>     input: reads/SRR2584863_1.fq.gz, reads/SRR2584863_2.fq.gz
>     output: trim/SRR2584863_1.trim.fq, trim/SRR2584863_2.trim.fq, trim/SRR2584863_1un.trim.fq, trim/SRR2584863_2un.trim.fq
>     log: logs/trimmomatic.SRR2584863.out, logs/trimmomatic.SRR2584863.err
>     jobid: 2
>     wildcards: sample=SRR2584863
>     resources: tmpdir=/tmp
>
> trimmomatic PE reads/SRR2584863_1.fq.gz reads/SRR2584863_2.fq.gz trim/SRR2584863_1.trim.fq trim/SRR2584863_1un.trim.fq trim/SRR2584863_2.trim.fq trim/SRR2584863_2un.trim.fq SLIDINGWINDOW:4:20 MINLEN:25 ILLUMINACLIP:NexteraPE-PE.fa:2:40:15 >logs/trimmomatic.SRR2584863.out 2>logs/trimmomatic.SRR2584863.err
>
> [Fri Oct  8 16:59:39 2021]
> rule indexgenome:
>     input: reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz
>     output: reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt
>     log: logs/bwa.index.out, logs/bwa.index.err
>     jobid: 5
>     resources: tmpdir=/tmp
>
> bwa index reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz >logs/bwa.index.out 2>logs/bwa.index.err
>
> [Fri Oct  8 16:59:39 2021]
> rule align:
>     input: trim/SRR2584863_1.trim.fq, trim/SRR2584863_1.trim.fq, reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz.bwt
>     output: alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam
>     log: logs/bwa.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err, logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.out, logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err
>     jobid: 1
>     wildcards: genome=GCA_000017985.1_ASM1798v1_genomic, sample=SRR2584863
>     resources: tmpdir=/tmp
>
> bwa mem reference/GCA_000017985.1_ASM1798v1_genomic.fna.gz trim/SRR2584863_1.trim.fq trim/SRR2584863_1.trim.fq 2>logs/bwa.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err | samtools sort -O bam -o alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam >logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.out 2>logs/samtools.GCA_000017985.1_ASM1798v1_genomic.SRR2584863.err
>
> [Fri Oct  8 16:59:39 2021]
> localrule all:
>     input: alignments/GCA_000017985.1_ASM1798v1_genomic/SRR2584863.bam
>     jobid: 0
>     resources: tmpdir=/tmp
>
> Job stats:
> job                count    min threads    max threads
> ---------------  -------  -------------  -------------
> align                  1              1              1
> all                    1              1              1
> download               2              1              1
> download_genome        1              1              1
> indexgenome            1              1              1
> trimmomatic            1              1              1
> total                  7              1              1
>
> This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
> ```
{: .code-out.code-max-300}

Gosh that's a lot of output! Let's build the {DAG} to see a more concise representation of what is going to happen:

> <code-in-title></code-in-title>
> ```bash
> snakemake --dag | dot -Tsvg > out.svg
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ![Image of the snakemake dag. Conspicuously missing is FastQC jobs!](../../images/


topic: data-science
tutorial: cli-advanced
title: Advanced CLI in Galaxy
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["bash"]
content_hash: 277e98728dcea8b445b26ba455dfe126
last_modified: 2026-02-26T21:12:14.736989
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-advanced/tutorial.html

content:
This tutorial will walk you through the basics of how to use the Unix command line.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) ["The Unix Shell"](https://swcarpentry.github.io/shell-novice/) lesson, which is licensed CC-BY 4.0. Adaptations have been made to make this work better in a GTN/Galaxy environment.
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pipes and Filtering

Now that we know a few basic commands,
we can finally look at the shell's most powerful feature:
the ease with which it lets us combine existing programs in new ways.
We'll start with the directory called `shell-lesson-data/molecules`
that contains six files describing some simple organic molecules.
The `.pdb` extension indicates that these files are in Protein Data Bank format,
a simple text format that specifies the type and position of each atom in the molecule.

```bash
cd ~/Desktop/shell-lesson-data/
ls molecules
```


Let's go into that directory with `cd` and run an example command `wc cubane.pdb`:

```bash
cd molecules
wc cubane
```

`wc` is the 'word count' command:
it counts the number of lines, words, and characters in files (from left to right, in that order).

If we run the command `wc *.pdb`, the `*` in `*.pdb` matches zero or more characters,
so the shell turns `*.pdb` into a list of all `.pdb` files in the current directory:

```bash
wc *.pdb
```

Note that `wc *.pdb` also shows the total number of all lines in the last line of the output.

If we run `wc -l` instead of just `wc`,
the output shows only the number of lines per file:

```bash
wc -l *.pdb
```

The `-m` and `-w` options can also be used with the `wc` command, to show
only the number of characters or the number of words in the files.

> <tip-title>Why Isn't It Doing Anything?</tip-title>
>
> What happens if a command is supposed to process a file, but we
> don't give it a filename? For example, what if we type:
>
> ```bash
> $ wc -l
> ```
>
> but don't type `*.pdb` (or anything else) after the command?
> Since it doesn't have any filenames, `wc` assumes it is supposed to
> process input given at the command prompt, so it just sits there and waits for us to give
> it some data interactively. From the outside, though, all we see is it
> sitting there: the command doesn't appear to do anything.
>
> If you make this kind of mistake, you can escape out of this state by holding down
> the control key (<kbd>Ctrl</kbd>) and typing the letter <kbd>C</kbd> once and
> letting go of the <kbd>Ctrl</kbd> key.
> <kbd>Ctrl</kbd>+<kbd>C</kbd>
{: .tip}


## Capturing output from commands

Which of these files contains the fewest lines?
It's an easy question to answer when there are only six files,
but what if there were 6000?
Our first step toward a solution is to run the command:

```bash
wc -l *.pdb > lengths.txt
```

The greater than symbol, `>`, tells the shell to **redirect** the command's output
to a file instead of printing it to the screen. (This is why there is no screen output:
everything that `wc` would have printed has gone into the
file `lengths.txt` instead.)  The shell will create
the file if it doesn't exist. If the file exists, it will be
silently overwritten, which may lead to data loss and thus requires
some caution.

> <tip-title>No <code>&gt;</code> on an AZERTY keyboard?</tip-title>
> You can rewrite this using the tee command which writes out a file, while also showing the output to `stdout`.
> ```bash
> wc -l *.pdb | tee lengths.txt
> ```
> Or you can use copy and paste to copy the `>` character from the materials.
{: .tip}

`ls lengths.txt` confirms that the file exists:

```bash
ls lengths.txt
```

We can now send the content of `lengths.txt` to the screen using `cat lengths.txt`.
The `cat` command gets its name from 'concatenate' i.e. join together,
and it prints the contents of files one after another.
There's only one file in this case,
so `cat` just shows us what it contains:

```bash
cat lengths.txt
```

> <tip-title>Output Page by Page</tip-title>
>
> We'll continue to use `cat` in this lesson, for convenience and consistency,
> but it has the disadvantage that it always dumps the whole file onto your screen.
> More useful in practice is the command `less`,
> which you use with `less lengths.txt`.
> This displays a screenful of the file, and then stops.
> You can go forward one screenful by pressing the spacebar,
> or back one by pressing `b`.  Press `q` to quit.
{: .tip}


## Filtering output

Next we'll use the `sort` command to sort the contents of the `lengths.txt` file.
But first we'll use an exercise to learn a little about the sort command:

> <question-title>What Does `sort -n` Do?</question-title>
>
> The file `shell-lesson-data/numbers.txt`
> contains the following lines:
>
> ```
> 10
> 2
> 19
> 22
> 6
> ```
>
> If we run `sort` on this file, the output is:
>
> ```
> 10
> 19
> 2
> 22
> 6
> ```
>
> If we run `sort -n` on the same file, we get this instead:
>
> ```
> 2
> 6
> 10
> 19
> 22
> ```
>
> Explain why `-n` has this effect.
>
> > <solution-title></solution-title>
> > The `-n` option specifies a numerical rather than an alphanumerical sort.
> {: .solution}
{: .question}

We will also use the `-n` option to specify that the sort is
numerical instead of alphanumerical.
This does *not* change the file;
instead, it sends the sorted result to the screen:

```bash
sort -n lengths.txt
```


We can put the sorted list of lines in another temporary file called `sorted-lengths.txt`
by putting `> sorted-lengths.txt` after the command,
just as we used `> lengths.txt` to put the output of `wc` into `lengths.txt`.
Once we've done that,
we can run another command called `head` to get the first few lines in `sorted-lengths.txt`:

```bash
sort -n lengths.txt > sorted-lengths.txt
```

Using `-n 1` with `head` tells it that
we only want the first line of the file;
`-n 20` would get the first 20,
and so on.
Since `sorted-lengths.txt` contains the lengths of our files ordered from least to greatest,
the output of `head` must be the file with the fewest lines.

> <tip-title>Redirecting to the same file</tip-title>
>
> It's a very bad idea to try redirecting
> the output of a command that operates on a file
> to the same file. For example:
>
> ```
> $ sort -n lengths.txt > lengths.txt
> ```
>
> Doing something like this may give you
> incorrect results and/or delete
> the contents of `lengths.txt`.
{: .tip}

> <question-title>What Does `>>` Mean?</question-title>
>
> We have seen the use of `>`, but there is a similar operator `>>`
> which works slightly differently.
> We'll learn about the differences between these two operators by printing some strings.
> We can use the `echo` command to print strings e.g.
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ echo The echo command prints text
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > The echo command prints text
> > ```
> {: .code-out}
>
> Now test the commands below to reveal the difference between the two operators:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ echo hello > testfile01.txt
> > ```
> {: .code-in}
>
> and:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ echo hello >> testfile02.txt
> > ```
> {: .code-in}
>
> **Hint**: Try executing each command twice in a row and then examining the output files.
>
> > ## Solution
> > In the first example with `>`, the string 'hello' is written to `testfile01.txt`,
> > but the file gets overwritten each time we run the command.
> >
> > We see from the second example that the `>>` operator also writes 'hello' to a file
> > (in this case`testfile02.txt`),
> > but appends the string to the file if it already exists
> > (i.e. when we run it for the second time).
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Appending Data</question-title>
>
> We have already met the `head` command, which prints lines from the start of a file.
> `tail` is similar, but prints lines from the end of a file instead.
>
> Consider the file `shell-lesson-data/data/animals.txt`.
> After these commands, select the answer that
> corresponds to the file `animals-subset.txt`:
>
> ```
> $ head -n 3 animals.txt > animals-subset.txt
> $ tail -n 2 animals.txt >> animals-subset.txt
> ```
>
> 1. The first three lines of `animals.txt`
> 2. The last two lines of `animals.txt`
> 3. The first three lines and the last two lines of `animals.txt`
> 4. The second and third lines of `animals.txt`
>
> > <solution-title></solution-title>
> > Option 3 is correct.
> > For option 1 to be correct we would only run the `head` command.
> > For option 2 to be correct we would only run the `tail` command.
> > For option 4 to be correct we would have to pipe the output of `head` into `tail -n 2`
> > by doing `head -n 3 animals.txt | tail -n 2 > animals-subset.txt`
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```


## Passing output to another command
In our example of finding the file with the fewest lines,
we are using two intermediate files `lengths.txt` and `sorted-lengths.txt` to store output.
This is a confusing way to work because
even once you understand what `wc`, `sort`, and `head` do,
those intermediate files make it hard to follow what's going on.
We can make it easier to understand by running `sort` and `head` together:

```bash
sort -n lengths.txt | head -n 1
```

The vertical bar, `|`, between the two commands is called a **pipe**.
It tells the shell that we want to use
the output of the command on the left
as the input to the command on the right.

This has removed the need for the `sorted-lengths.txt` file.

## Combining multiple commands
Nothing prevents us from chaining pipes consecutively.
We can for example send the output of `wc` directly to `sort`,
and then the resulting output to `head`.
This removes the need for any intermediate files.

We'll start by using a pipe to send the output of `wc` to `sort`:

```bash
wc -l *.pdb | sort -n
```

We can then send that output through another pipe, to `head`, so that the full pipeline becomes:

```bash
wc -l *.pdb | sort -n | head -n 1
```

This is exactly like a mathematician nesting functions like *log(3x)*
and saying 'the log of three times *x*'.
In our case,
the calculation is 'head of sort of line count of `*.pdb`'.


The redirection and pipes used in the last few commands are illustrated below:

![Redirects and Pipes of different commands](../../images/carpentries-cli/redirects-and-pipes.svg)

`wc -l *.pdb` will direct the output to the shell. `wc -l *.pdb > lengths` will
direct output to the file lengths. `wc -l *.pdb | sort -n | head -n 1` will
build a pipeline where the output of the wc command is the input to the sort
command, the output of the sort command is the input to the head command and
the output of the head command is directed to the shell

> <question-title>Piping Commands Together</question-title>
>
> In our current directory, we want to find the 3 files which have the least number of
> lines. Which command listed below would work?
>
> 1. `wc -l * > sort -n > head -n 3`
> 2. `wc -l * | sort -n | head -n 1-3`
> 3. `wc -l * | head -n 3 | sort -n`
> 4. `wc -l * | sort -n | head -n 3`
>
> > <solution-title></solution-title>
> > Option 4 is the solution.
> > The pipe character `|` is used to connect the output from one command to
> > the input of another.
> > `>` is used to redirect standard output to a file.
> > Try it in the `shell-lesson-data/molecules` directory!
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```


## Tools designed to work together
This idea of linking programs together is why Unix has been so successful.
Instead of creating enormous programs that try to do many different things,
Unix programmers focus on creating lots of simple tools that each do one job well,
and that work well with each other.
This programming model is called 'pipes and filters'.
We've already seen pipes;
a **filter** is a program like `wc` or `sort`
that transforms a stream of input into a stream of output.
Almost all of the standard Unix tools can work this way:
unless told to do otherwise,
they read from standard input,
do something with what they've read,
and write to standard output.

The key is that any program that reads lines of text from standard input
and writes lines of text to standard output
can be combined with every other program that behaves this way as well.
You can *and should* write your programs this way
so that you and other people can put those programs into pipes to multiply their power.


> <question-title>Pipe Reading Comprehension</question-title>
>
> A file called `animals.txt` (in the `shell-lesson-data/data` folder) contains the following data:
>
> ```
> 2012-11-05,deer
> 2012-11-05,rabbit
> 2012-11-05,raccoon
> 2012-11-06,rabbit
> 2012-11-06,deer
> 2012-11-06,fox
> 2012-11-07,rabbit
> 2012-11-07,bear
> ```
>
> What text passes through each of the pipes and the final redirect in the pipeline below?
>
> ```
> $ cat animals.txt | head -n 5 | tail -n 3 | sort -r > final.txt
> ```
>
> Hint: build the pipeline up one command at a time to test your understanding
>
> > <solution-title></solution-title>
> > The `head` command extracts the first 5 lines from `animals.txt`.
> > Then, the last 3 lines are extracted from the previous 5 by using the `tail` command.
> > With the `sort -r` command those 3 lines are sorted in reverse order and finally,
> > the output is redirected to a file `final.txt`.
> > The content of this file can be checked by executing `cat final.txt`.
> > The file should contain the following lines:
> > ```
> > 2012-11-06,rabbit
> > 2012-11-06,deer
> > 2012-11-05,raccoon
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Pipe Construction</question-title>
>
> For the file `animals.txt` from the previous exercise, consider the following command:
>
> ```
> $ cut -d , -f 2 animals.txt
> ```
>
> The `cut` command is used to remove or 'cut out' certain sections of each line in the file,
> and `cut` expects the lines to be separated into columns by a <kbd>Tab</kbd> character.
> A character used in this way is a called a **delimiter**.
> In the example above we use the `-d` option to specify the comma as our delimiter character.
> We have also used the `-f` option to specify that we want to extract the second field (column).
> This gives the following output:
>
> ```
> deer
> rabbit
> raccoon
> rabbit
> deer
> fox
> rabbit
> bear
> ```
>
> The `uniq` command filters out adjacent matching lines in a file.
> How could you extend this pipeline (using `uniq` and another command) to find
> out what animals the file contains (without any duplicates in their
> names)?
>
> > <solution-title></solution-title>
> > ```
> > $ cut -d , -f 2 animals.txt | sort | uniq
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Which Pipe?</question-title>
>
> The file `animals.txt` contains 8 lines of data formatted as follows:
>
> ```
> 2012-11-05,deer
> 2012-11-05,rabbit
> 2012-11-05,raccoon
> 2012-11-06,rabbit
> ...
> ```
>
> The `uniq` command has a `-c` option which gives a count of the
> number of times a line occurs in its input.  Assuming your current
> directory is `shell-lesson-data/data/`, what command would you use to produce
> a table that shows the total count of each type of animal in the file?
>
> 1.  `sort animals.txt | uniq -c`
> 2.  `sort -t, -k2,2 animals.txt | uniq -c`
> 3.  `cut -d, -f 2 animals.txt | uniq -c`
> 4.  `cut -d, -f 2 animals.txt | sort | uniq -c`
> 5.  `cut -d, -f 2 animals.txt | sort | uniq -c | wc -l`
>
> > <solution-title></solution-title>
> > Option 4. is the correct answer.
> > If you have difficulty understanding why, try running the commands, or sub-sections of
> > the pipelines (make sure you are in the `shell-lesson-data/data` directory).
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

## Nelle's Pipeline: Checking Files

Nelle has run her samples through the assay machines
and created 17 files in the `north-pacific-gyre/2012-07-03` directory described earlier.
As a quick check, starting from her home directory, Nelle types:

```bash
cd ~/Desktop/shell-lesson-data/north-pacific-gyre/2012-07-03
wc -l *.txt
```

The output is 18 lines that look like this:

```
300 NENE01729A.txt
300 NENE01729B.txt
300 NENE01736A.txt
300 NENE01751A.txt
300 NENE01751B.txt
300 NENE01812A.txt
... ...
```

Now she types this:

```bash
wc -l *.txt | sort -n | head -n 5
```

Whoops: one of the files is 60 lines shorter than the others.
When she goes back and checks it,
she sees that she did that assay at 8:00 on a Monday morning --- someone
was probably in using the machine on the weekend,
and she forgot to reset it.
Before re-running that sample,
she checks to see if any files have too much data:

```bash
wc -l *.txt | sort -n | tail -n 5
```

Those numbers look good --- but what's that 'Z' doing there in the third-to-last line?
All of her samples should be marked 'A' or 'B';
by convention,
her lab uses 'Z' to indicate samples with missing information.
To find others like it, she does this:

```bash
ls *Z.txt
```

Sure enough,
when she checks the log on her laptop,
there's no depth recorded for either of those samples.
Since it's too late to get the information any other way,
she must exclude those two files from her analysis.
She could delete them using `rm`,
but there are actually some analyses she might do later where depth doesn't matter,
so instead, she'll have to be careful later on to select files using the wildcard expressions
`NENE*A.txt NENE*B.txt`.


> <question-title>Removing Unneeded Files</question-title>
>
> Suppose you want to delete your processed data files, and only keep
> your raw files and processing script to save storage.
> The raw files end in `.dat` and the processed files end in `.txt`.
> Which of the following would remove all the processed data files,
> and *only* the processed data files?
>
> 1. `rm ?.txt`
> 2. `rm *.txt`
> 3. `rm * .txt`
> 4. `rm *.*`
>
> > <solution-title></solution-title>
> > 1. This would remove `.txt` files with one-character names
> > 2. This is correct answer
> > 3. The shell would expand `*` to match everything in the current directory,
> > so the command would try to remove all matched files and an additional
> > file called `.txt`
> > 4. The shell would expand `*.*` to match all files with any extension,
> > so this command would delete all files
> {: .solution}
{: .question}


# Loops

**Loops** are a programming construct which allow us to repeat a command or set of commands
for each item in a list.
As such they are key to productivity improvements through automation.
Similar to wildcards and tab completion, using loops also reduces the
amount of typing required (and hence reduces the number of typing mistakes).

Suppose we have several hundred genome data files named `basilisk.dat`, `minotaur.dat`, and
`unicorn.dat`.
For this example, we'll use the `creatures` directory which only has three example files,
but the principles can be applied to many many more files at once. First, go
into the creatures directory.

```bash
# Change directories here!

```

The structure of these files is the same: the common name, classification, and updated date are
presented on the first three lines, with DNA sequences on the following lines.
Let's look at the files:

```bash
head -n 5 basilisk.dat minotaur.dat unicorn.dat
```

We would like to print out the classification for each species, which is given on the second
line of each file.
For each file, we would need to execute the command `head -n 2` and pipe this to `tail -n 1`.
We’ll use a loop to solve this problem, but first let’s look at the general form of a loop:

```
for thing in list_of_things
do
    operation_using $thing    # Indentation within the loop is not required, but aids legibility
done
```

and we can apply this to our example like this:

```bash
for filename in basilisk.dat minotaur.dat unicorn.dat
do
    head -n 2 $filename | tail -n 1
done
```


> <tip-title>Follow the Prompt</tip-title>
>
> The shell prompt changes from `$` to `>` and back again as we were
> typing in our loop. The second prompt, `>`, is different to remind
> us that we haven't finished typing a complete command yet. A semicolon, `;`,
> can be used to separate two commands written on a single line.
{: .tip}

When the shell sees the keyword `for`,
it knows to repeat a command (or group of commands) once for each item in a list.
Each time the loop runs (called an iteration), an item in the list is assigned in sequence to
the **variable**, and the commands inside the loop are executed, before moving on to
the next item in the list.
Inside the loop,
we call for the variable's value by putting `$` in front of it.
The `$` tells the shell interpreter to treat
the variable as a variable name and substitute its value in its place,
rather than treat it as text or an external command.

In this example, the list is three filenames: `basilisk.dat`, `minotaur.dat`, and `unicorn.dat`.
Each time the loop iterates, it will assign a file name to the variable `filename`
and run the `head` command.
The first time through the loop,
`$filename` is `basilisk.dat`.
The interpreter runs the command `head` on `basilisk.dat`
and pipes the first two lines to the `tail` command,
which then prints the second line of `basilisk.dat`.
For the second iteration, `$filename` becomes
`minotaur.dat`. This time, the shell runs `head` on `minotaur.dat`
and pipes the first two lines to the `tail` command,
which then prints the second line of `minotaur.dat`.
For the third iteration, `$filename` becomes
`unicorn.dat`, so the shell runs the `head` command on that file,
and `tail` on the output of that.
Since the list was only three items, the shell exits the `for` loop.

> <tip-title>Same Symbols, Different Meanings</tip-title>
>
> Here we see `>` being used as a shell prompt, whereas `>` is also
> used to redirect output.
> Similarly, `$` is used as a shell prompt, but, as we saw earlier,
> it is also used to ask the shell to get the value of a variable.
>
> If the *shell* prints `>` or `$` then it expects you to type something,
> and the symbol is a prompt.
>
> If *you* type `>` or `$` yourself, it is an instruction from you that
> the shell should redirect output or get the value of a variable.
{: .tip}

When using variables it is also
possible to put the names into curly braces to clearly delimit the variable
name: `$filename` is equivalent to `${filename}`, but is different from
`${file}name`. You may find this notation in other people's programs.

We have called the variable in this loop `filename`
in order to make its purpose clearer to human readers.
The shell itself doesn't care what the variable is called;
if we wrote this loop as:

```bash
for x in basilisk.dat minotaur.dat unicorn.dat
do
    head -n 2 $x | tail -n 1
done
```

or:

```bash
for temperature in basilisk.dat minotaur.dat unicorn.dat
do
    head -n 2 $temperature | tail -n 1
done
```

it would work exactly the same way.

# Don't do this.

Programs are only useful if people can understand them,
so meaningless names (like `x`) or misleading names (like `temperature`)
increase the odds that the program won't do what its readers think it does.

> <question-title>Variables in Loops</question-title>
>
> This exercise refers to the `shell-lesson-data/molecules` directory.
> `ls` gives the following output:
>
> ```
> cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> ```
>
> What is the output of the following code?
>
> ```
> for datafile in *.pdb
> do
>     ls *.pdb
> done
> ```
>
> Now, what is the output of the following code?
>
> ```
> for datafile in *.pdb
> do
>    ls $datafile
> done
> ```
>
> Why do these two loops give different outputs?
>
> > <solution-title></solution-title>
> > The first code block gives the same output on each iteration through
> > the loop.
> > Bash expands the wildcard `*.pdb` within the loop body (as well as
> > before the loop starts) to match all files ending in `.pdb`
> > and then lists them using `ls`.
> > The expanded loop would look like this:
> > ```
> > $ for datafile in cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > > do
> > >     ls cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > > done
> > ```
> >
> > ```
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > cubane.pdb  ethane.pdb  methane.pdb  octane.pdb  pentane.pdb  propane.pdb
> > ```
> >
> > The second code block lists a different file on each loop iteration.
> > The value of the `datafile` variable is evaluated using `$datafile`,
> > and then listed using `ls`.
> >
> > ```
> > cubane.pdb
> > ethane.pdb
> > methane.pdb
> > octane.pdb
> > pentane.pdb
> > propane.pdb
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Limiting Sets of Files</question-title>
>
> What would be the output of running the following loop in thei
> `shell-lesson-data/molecules` directory?
>
> ```
> for filename in c*
> do
>     ls $filename
> done
> ```
>
> 1.  No files are listed.
> 2.  All files are listed.
> 3.  Only `cubane.pdb`, `octane.pdb` and `pentane.pdb` are listed.
> 4.  Only `cubane.pdb` is listed.
>
> > <solution-title></solution-title>
> > 4 is the correct answer. `*` matches zero or more characters, so any file name starting with
> > the letter c, followed by zero or more other characters will be matched.
> {: .solution}
>
> How would the output differ from using this command instead?
>
> ```
> for filename in *c*
> do
>     ls $filename
> done
> ```
>
> 1.  The same files would be listed.
> 2.  All the files are listed this time.
> 3.  No files are listed this time.
> 4.  The files `cubane.pdb` and `octane.pdb` will be listed.
> 5.  Only the file `octane.pdb` will be listed.
>
> > <solution-title></solution-title>
> > 4 is the correct answer. `*` matches zero or more characters, so a file name with zero or more
> > characters before a letter c and zero or more characters after the letter c will be matched.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Saving to a File in a Loop - Part One</question-title>
>
> In the `shell-lesson-data/molecules` directory, what is the effect of this loop?
>
> ```
> for alkanes in *.pdb
> do
>     echo $alkanes
>     cat $alkanes > alkanes.pdb
> done
> ```
>
> 1.  Prints `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, `pentane.pdb` and
>    `propane.pdb`, and the text from `propane.pdb` will be saved to a file called `alkanes.pdb`.
> 2.  Prints `cubane.pdb`, `ethane.pdb`, and `methane.pdb`, and the text from all three files
>     would be concatenated and saved to a file called `alkanes.pdb`.
> 3.  Prints `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, and `pentane.pdb`,
>     and the text from `propane.pdb` will be saved to a file called `alkanes.pdb`.
> 4.  None of the above.
>
> > <solution-title></solution-title>
> > 1 is correct. The text from each file in turn gets written to the `alkanes.pdb` file.
> > However, the file gets overwritten on each loop iteration, so the final content of `alkanes.pdb`
> > is the text from the `propane.pdb` file.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Saving to a File in a Loop - Part Two</question-title>
>
> Also in the `shell-lesson-data/molecules` directory,
> what would be the output of the following loop?
>
> ```
> for datafile in *.pdb
> do
>     cat $datafile >> all.pdb
> done
> ```
>
> 1.  All of the text from `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, and
>     `pentane.pdb` would be concatenated and saved to a file called `all.pdb`.
> 2.  The text from `ethane.pdb` will be saved to a file called `all.pdb`.
> 3.  All of the text from `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, `pentane.pdb`
>     and `propane.pdb` would be concatenated and saved to a file called `all.pdb`.
> 4.  All of the text from `cubane.pdb`, `ethane.pdb`, `methane.pdb`, `octane.pdb`, `pentane.pdb`
>     and `propane.pdb` would be printed to the screen and saved to a file called `all.pdb`.
>
> > <solution-title></solution-title>
> > 3 is the correct answer. `>>` appends to a file, rather than overwriting it with the redirected
> > output from a command.
> > Given the output from the `cat` command has been redirected, nothing is printed to the screen.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

Let's continue with our example in the `shell-lesson-data/creatures` directory.
Here's a slightly more complicated loop:

```bash
for filename in *.dat
do
    echo $filename
    head -n 100 $filename | tail -n 20
done
```

The shell starts by expanding `*.dat` to create the list of files it will process.
The **loop body**
then executes two commands for each of those files.
The first command, `echo`, prints its command-line arguments to standard output.
For example:

```bash
echo hello there
```

prints:

```
hello there
```

In this case,
since the shell expands `$filename` to be the name of a file,
`echo $filename` prints the name of the file.
Note that we can't write this as:

```bash
for filename in *.dat
do
    $filename
    head -n 100 $filename | tail -n 20
done
```

because then the first time through the loop,
when `$filename` expanded to `basilisk.dat`, the shell would try to run `basilisk.dat` as a program.
Finally,
the `head` and `tail` combination selects lines 81-100
from whatever file is being processed
(assuming the file has at least 100 lines).

> <tip-title>Spaces in Names</tip-title>
>
> Spaces are used to separate the elements of the list
> that we are going to loop over. If one of those elements
> contains a space character, we need to surround it with
> quotes, and do the same thing to our loop variable.
> Suppose our data files are named:
>
> ```
> red dragon.dat
> purple unicorn.dat
> ```
>
> To loop over these files, we would need to add double quotes like so:
>
> ```
> $ for filename in "red dragon.dat" "purple unicorn.dat"
> > do
> >     head -n 100 "$filename" | tail -n 20
> > done
> ```
>
> It is simpler to avoid using spaces (or other special characters) in filenames.
>
> The files above don't exist, so if we run the above code, the `head` command will be unable
> to find them, however the error message returned will show the name of the files it is
> expecting:
>
> ```
> head: cannot open ‘red dragon.dat’ for reading: No such file or directory
> head: cannot open ‘purple unicorn.dat’ for reading: No such file or directory
> ```
>
> Try removing the quotes around `$filename` in the loop above to see the effect of the quote
> marks on spaces. Note that we get a result from the loop command for unicorn.dat
> when we run this code in the `creatures` directory:
>
> ```
> head: cannot open ‘red’ for reading: No such file or directory
> head: cannot open ‘dragon.dat’ for reading: No such file or directory
> head: cannot open ‘purple’ for reading: No such file or directory
> CGGTACCGAA
> AAGGGTCGCG
> CAAGTGTTCC
> ...
> ```
{: .tip}

We would like to modify each of the files in `shell-lesson-data/creatures`, but also save a version
of the original files, naming the copies `original-basilisk.dat` and `original-unicorn.dat`.
We can't use:

```
cp *.dat original-*.dat
```

because that would expand to:

```bash
cp basilisk.dat minotaur.dat unicorn.dat original-*.dat
```

This wouldn't back up our files, instead we get an error.

This problem arises when `cp` receives more than two inputs. When this happens, it
expects the last input to be a directory where it can copy all the files it was passed.
Since there is no directory named `original-*.dat` in the `creatures` directory we get an
error.

Instead, we can use a loop:
```bash
for filename in *.dat
do
    cp $filename original-$filename
done
```

This loop runs the `cp` command once for each filename.
The first time,
when `$filename` expands to `basilisk.dat`,
the shell executes:

```
cp basilisk.dat original-basilisk.dat
```

The second time, the command is:

```
cp minotaur.dat original-minotaur.dat
```

The third and last time, the command is:

```
cp unicorn.dat original-unicorn.dat
```

Since the `cp` command does not normally produce any output, it's hard to check
that the loop is doing the correct thing.
However, we learned earlier how to print strings using `echo`, and we can modify the loop
to use `echo` to print our commands without actually executing them.
As such we can check what commands *would be* run in the unmodified loop.

The following diagram
shows what happens when the modified loop is executed, and demonstrates how the
judicious use of `echo` is a good debugging technique.

![The for loop 'for filename in *.dat; do echo cp $filename original-$filename; done' will successively assign the names of all '*.dat' files in your current directory to the variable '$filename' and then execute the command. With the files 'basilisk.dat', 'minotaur.dat' and 'unicorn.dat' in the current directory the loop will successively call the echo command three times and print three lines: 'cp basislisk.dat original-basilisk.dat', then 'cp minotaur.dat original-minotaur.dat' and finally 'cp unicorn.dat original-unicorn.dat'](../../images/carpentries-cli/shell_script_for_loop_flow_chart.svg)

## Nelle's Pipeline: Processing Files

Nelle is now ready to process her data files using `goostats.sh` ---
a shell script written by her supervisor.
This calculates some statistics from a protein sample file, and takes two arguments:

1. an input file (containing the raw data)
2. an output file (to store the calculated statistics)

Since she's still learning how to use the shell,
she decides to build up the required commands in stages.
Her first step is to make sure that she can select the right input files --- remember,
these are ones whose names end in 'A' or 'B', rather than 'Z'.
Starting from her home directory, Nelle types:

```bash
cd ~/Desktop/shell-lesson-data/north-pacific-gyre/2012-07-03
for datafile in NENE*A.txt NENE*B.txt
do
    echo $datafile
done
```

Her next step is to decide
what to call the files that the `goostats.sh` analysis program will create.
Prefixing each input file's name with 'stats' seems simple,
so she modifies her loop to do that:

```bash
for datafile in NENE*A.txt NENE*B.txt
do
    echo $datafile stats-$datafile
done
```

She hasn't actually run `goostats.sh` yet,
but now she's sure she can select the right files and generate the right output filenames.

> <tip-title>Top Terminal Tip: Re-running previous commands</tip-title>
> Typing in commands over and over again is becoming tedious,
> though,
> and Nelle is worried about making mistakes,
> so instead of re-entering her loop,
> she presses <kbd>↑</kbd>.
> In response,
> the shell redisplays the whole loop on one line
> (using semi-colons to separate the pieces):
>
> ```
> for datafile in NENE*A.txt NENE*B.txt; do echo $datafile stats-$datafile; done
> ```
{: .tip}

Using the left arrow key,
Nelle backs up and changes the command `echo` to `bash goostats.sh`:

```
for datafile in NENE*A.txt NENE*B.txt; do bash goostats.sh $datafile stats-$datafile; done
```

When she presses <kbd>Enter</kbd>,
the shell runs the modified command.
However, nothing appears to happen --- there is no output.
After a moment, Nelle realizes that since her script doesn't print anything to the screen
any longer, she has no idea whether it is running, much less how quickly.
She kills the running command by typing <kbd>Ctrl</kbd>+<kbd>C</kbd>,
uses <kbd>↑</kbd> to repeat the command,
and edits it to read:

```
for datafile in NENE*A.txt NENE*B.txt; do echo $datafile; bash goostats.sh $datafile stats-$datafile; done
```

> <tip-title>Beginning and End</tip-title>
>
> We can move to the beginning of a line in the shell by typing <kbd>Ctrl</kbd>+<kbd>A</kbd>
> and to the end using <kbd>Ctrl</kbd>+<kbd>E</kbd>.
{: .tip}

When she runs her program now,
it produces one line of output every five seconds or so:

1518 times 5 seconds,
divided by 60,
tells her that her script will take about two hours to run.
As a final check,
she opens another terminal window,
goes into `north-pacific-gyre/2012-07-03`,
and uses `cat stats-NENE01729B.txt`
to examine one of the output files.
It looks good,
so she decides to get some coffee and catch up on her reading.

> <tip-title>Those Who Know History Can Choose to Repeat It</tip-title>
>
> Another way to repeat previous work is to use the `history` command to
> get a list of the last few hundred commands that have been executed, and
> then to use `!123` (where '123' is replaced by the command number) to
> repeat one of those commands. For example, if Nelle types this:
>
> ```
> $ history | tail -n 5
> ```
>
> ```
>   456  ls -l NENE0*.txt
>   457  rm stats-NENE01729B.txt.txt
>   458  bash goostats.sh NENE01729B.txt stats-NENE01729B.txt
>   459  ls -l NENE0*.txt
>   460  history
> ```
>
> then she can re-run `goostats.sh` on `NENE01729B.txt` simply by typing
> `!458`. This number will be different for you, you should check your history before running it!
{: .tip}

> <tip-title>Other History Commands</tip-title>
>
> There are a number of other shortcut commands for getting at the history.
>
> - <kbd>Ctrl</kbd>+<kbd>R</kbd> enters a history search mode 'reverse-i-search' and finds the
> most recent command in your history that matches the text you enter next.
> Press <kbd>Ctrl</kbd>+<kbd>R</kbd> one or more additional times to search for earlier matches.
> You can then use the left and right arrow keys to choose that line and edit
> it then hit <kbd>Return</kbd> to run the command.
> - `!!` retrieves the immediately preceding command
> (you may or may not find this more convenient than using <kbd>↑</kbd>)
> - `!$` retrieves the last word of the last command.
> That's useful more often than you might expect: after
> `bash goostats.sh NENE01729B.txt stats-NENE01729B.txt`, you can type
> `less !$` to look at the file `stats-NENE01729B.txt`, which is
> quicker than doing <kbd>↑</kbd> and editing the command-line.
{: .tip}

> <question-title>Doing a Dry Run</question-title>
>
> A loop is a way to do many things at once --- or to make many mistakes at
> once if it does the wrong thing. One way to check what a loop *would* do
> is to `echo` the commands it would run instead of actually running them.
>
> Suppose we want to preview the commands the following loop will execute
> without actually running those commands:
>
> ```
> cd ~/Desktop/shell-lesson-data/pdb/
> for datafile in *.pdb
> do
>     cat $datafile >> all.pdb
> done
> ```
>
> What is the difference between the two loops below, and which one would we
> want to run?
>
> > <code-in-title>Version 1</code-in-title>
> > ```
> > for datafile in *.pdb
> > do
> >     echo cat $datafile >> all.pdb
> > done
> > ```
> {: .code-in}
>
> > <code-in-title>Version 2</code-in-title>
> > ```
> > for datafile in *.pdb
> > do
> >     echo "cat $datafile >> all.pdb"
> > done
> > ```
> {: .code-in}
>
> > <solution-title></solution-title>
> > The second version is the one we want to run.
> > This prints to screen everything enclosed in the quote marks, expanding the
> > loop variable name because we have prefixed it with a dollar sign.
> >
> > The first version appends the output from the command `echo cat $datafile`
> > to the file, `all.pdb`. This file will just contain the list;
> > `cat cubane.pdb`, `cat ethane.pdb`, `cat methane.pdb` etc.
> >
> > Try both versions for yourself to see the output! Be sure to open the
> > `all.pdb` file to view its contents.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Nested Loops</question-title>
>
> Suppose we want to set up a directory structure to organize
> some experiments measuring reaction rate constants with different compounds
> *and* different temperatures.  What would be the
> result of the following code:
>
> ```
> for species in cubane ethane methane
> do
>     for temperature in 25 30 37 40
>     do
>         mkdir $species-$temperature
>     done
> done
> ```
>
> > <solution-title></solution-title>
> > We have a nested loop, i.e. contained within another loop, so for each species
> > in the outer loop, the inner loop (the nested loop) iterates over the list of
> > temperatures, and creates a new directory for each combination.
> >
> > Try running the code for yourself to see which directories are created!
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```


# Finding Things

In the same way that many of us now use 'Google' as a
verb meaning 'to find', Unix programmers often use the
word 'grep'.
'grep' is a contraction of 'global/regular expression/print',
a common sequence of operations in early Unix text editors.
It is also the name of a very useful command-line program.

`grep` finds and prints lines in files that match a pattern.
For our examples,
we will use a file that contains three haiku taken from a
1998 competition in *Salon* magazine. For this set of examples,
we're going to be working in the writing subdirectory:

```bash
cd
cd Desktop/shell-lesson-data/writing
cat haiku.txt
```


> <tip-title>Forever, or Five Years</tip-title>
>
> We haven't linked to the original haiku because
> they don't appear to be on *Salon*'s site any longer.
> As [Jeff Rothenberg said](https://www.clir.org/wp-content/uploads/sites/6/ensuring.pdf),
> 'Digital information lasts forever --- or five years, whichever comes first.'
> Luckily, popular content often [has backups](http://wiki.c2.com/?ComputerErrorHaiku).
{: .tip}

Let's find lines that contain the word 'not':

```bash
grep not haiku.txt
```

Here, `not` is the pattern we're searching for.
The grep command searches through the file, looking for matches to the pattern specified.
To use it type `grep`, then the pattern we're searching for and finally
the name of the file (or files) we're searching in.

The output is the three lines in the file that contain the letters 'not'.

By default, grep searches for a pattern in a case-sensitive way.
In addition, the search pattern we have selected does not have to form a complete word,
as we will see in the next example.

Let's search for the pattern: 'The'.

```bash
grep The haiku.txt
```

This time, two lines that include the letters 'The' are outputted,
one of which contained our search pattern within a larger word, 'Thesis'.

To restrict matches to lines containing the word 'The' on its own,
we can give `grep` with the `-w` option.
This will limit matches to word boundaries.

Later in this lesson, we will also see how we can change the search behavior of grep
with respect to its case sensitivity.

```bash
grep -w The haiku.txt
```

Note that a 'word boundary' includes the start and end of a line, so not
just letters surrounded by spaces.
Sometimes we don't
want to search for a single word, but a phrase. This is also easy to do with
`grep` by putting the phrase in quotes.

```bash
grep -w "is not" haiku.txt
```

We've now seen that you don't have to have quotes around single words,
but it is useful to use quotes when searching for multiple words.
It also helps to make it easier to distinguish between the search term or phrase
and the file being searched.
We will use quotes in the remaining examples.

Another useful option is `-n`, which numbers the lines that match:

```bash
grep -n "it" haiku.txt
```

Here, we can see that lines 5, 9, and 10 contain the letters 'it'.

We can combine options (i.e. flags) as we do with other Unix commands.
For example, let's find the lines that contain the word 'the'.
We can combine the option `-w` to find the lines that contain the word 'the'
and `-n` to number the lines that match:

```bash
grep -n -w "the" haiku.txt
```

Now we want to use the option `-i` to make our search case-insensitive:

```bash
grep -n -w -i "the" haiku.txt
```

Now, we want to use the option `-v` to invert our search, i.e., we want to output
the lines that do not contain the word 'the'.

```bash
grep -n -w -v "the" haiku.txt
```


If we use the `-r` (recursive) option,
`grep` can search for a pattern recursively through a set of files in subdirectories.

Let's search recursively for `Yesterday` in the `shell-lesson-data/writing` directory:

```bash
grep -r Yesterday .
```

`grep` has lots of other options. To find out what they are, we can type:

```bash
grep --help
```

> <question-title>Using `grep`</question-title>
>
> Which command would result in the following output:
>
> ```
> and the presence of absence:
> ```
>
> 1. `grep "of" haiku.txt`
> 2. `grep -E "of" haiku.txt`
> 3. `grep -w "of" haiku.txt`
> 4. `grep -i "of" haiku.txt`
>
> > <solution-title></solution-title>
> > The correct answer is 3, because the `-w` option looks only for whole-word matches.
> > The other options will also match 'of' when part of another word.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <tip-title>Wildcards</tip-title>
>
> `grep`'s real power doesn't come from its options, though; it comes from
> the fact that patterns can include wildcards. (The technical name for
> these is **regular expressions**, which
> is what the 're' in 'grep' stands for.) Regular expressions are both complex
> and powerful; if you want to do complex searches, please look at the lesson
> on [our website](http://v4.software-carpentry.org/regexp/index.html). As a taster, we can
> find lines that have an 'o' in the second position like this:
>
> ```
> $ grep -E "^.o" haiku.txt
> ```
>
> ```
> You bring fresh toner.
> Today it is not working
> Software is like that.
> ```
>
> We use the `-E` option and put the pattern in quotes to prevent the shell
> from trying to interpret it. (If the pattern contained a `*`, for
> example, the shell would try to expand it before running `grep`.) The
> `^` in the pattern anchors the match to the start of the line. The `.`
> matches a single character (just like `?` in the shell), while the `o`
> matches an actual 'o'.
{: .tip}

```bash
# Explore the possible solutions here!
```

> <question-title>Tracking a Species</question-title>
>
> Leah has several hundred
> data files saved in one directory, each of which is formatted like this:
>
> ```
> 2013-11-05,deer,5
> 2013-11-05,rabbit,22
> 2013-11-05,raccoon,7
> 2013-11-06,rabbit,19
> 2013-11-06,deer,2
> ```
>
> She wants to write a shell script that takes a species as the first command-line argument
> and a directory as the second argument. The script should return one file called `species.txt`
> containing a list of dates and the number of that species seen on each date.
> For example using the data shown above, `rabbit.txt` would contain:
>
> ```
> 2013-11-05,22
> 2013-11-06,19
> ```
>
> Put these commands and pipes in the right order to achieve this:
>
> ```
> cut -d : -f 2
> >
> |
> grep -w $1 -r $2
> |
> $1.txt
> cut -d , -f 1,3
> ```
>
> Hint: use `man grep` to look for how to grep text recursively in a directory
> and `man cut` to select more than one field in a line.
>
> An example of such a file is provided in `shell-lesson-data/data/animal-counts/animals.txt`
>
> > <solution-title></solution-title>
> >
> > ```
> > grep -w $1 -r $2 | cut -d : -f 2 | cut -d , -f 1,3 > $1.txt
> > ```
> >
> > Actually, you can swap the order of the two cut commands and it still works. At the
> > command line, try changing the order of the cut commands, and have a look at the output
> > from each step to see why this is the case.
> >
> > You would call the script above like this:
> >
> > ```
> > $ bash count-species.sh bear .
> > ```
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <question-title>Little Women</question-title>
>
> You and your friend, having just finished reading *Little Women* by
> Louisa May Alcott, are in an argument.  Of the four sisters in the
> book, Jo, Meg, Beth, and Amy, your friend thinks that Jo was the
> most mentioned.  You, however, are certain it was Amy.  Luckily, you
> have a file `LittleWomen.txt` containing the full text of the novel
> (`shell-lesson-data/writing/data/LittleWomen.txt`).
> Using a `for` loop, how would you tabulate the number of times each
> of the four sisters is mentioned?
>
> Hint: one solution might employ
> the commands `grep` and `wc` and a `|`, while another might utilize
> `grep` options.
> There is often more than one way to solve a programming task, so a
> particular solution is usually chosen based on a combination of
> yielding the correct result, elegance, readability, and speed.
>
> > <solution-title></solution-title>
> > ```
> > for sis in Jo Meg Beth Amy
> > do
> > 	echo $sis:
> >	grep -ow $sis LittleWomen.txt | wc -l
> > done
> > ```
> >
> > Alternative, slightly inferior solution:
> > ```
> > for sis in Jo Meg Beth Amy
> > do
> > 	echo $sis:
> >	grep -ocw $sis LittleWomen.txt
> > done
> > ```
> >
> > This solution is inferior because `grep -c` only reports the number of lines matched.
> > The total number of matches reported by this method will be lower if there is more
> > than one match per line.
> >
> > Perceptive observers may have noticed that character names sometimes appear in all-uppercase
> > in c


topic: data-science
tutorial: python-math
title: Python - Math
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 7684f6d5d2b61a02bce04bddd0b252ea
last_modified: 2026-02-26T21:12:14.903560
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-math/tutorial.html

content:
Here we'll learn some of the fundamentals of python and how to do basic maths in Python.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Python Fundamentals

## Variables

Any Python interpreter can be used as a calculator. We can do simple sums

```python
3 + 5 - 4
```

Multiplication

```python
5 * 4
```

Division

```python
5 / 4
```

Just like you've probably learned in maths courses, we can assign values to variables

```python
x = 10
y = 2.5
z = 3
```

From now on, whenever we use `x` or `y` or `z`, Python will substitute the value we assigned to it.
You'll notice when we assign things to a variable, nothing is printed out. If we've assigned a value to a variable we can print it by doing:

```python
print(x)
print(y, z)
```

You can print out multiple variables if you separate them with a comma `,`.


> <tip-title>Variable Names</tip-title>
> In Python, variable names:
>
> - can include letters, digits, and underscores
> - cannot start with a digit
> - are case sensitive
>
> This means that, for example:
>
> - `x` is a valid name
> - `weight0` is a valid variable name, whereas `0weight` is not
> - `weight` and `Weight` are different variables
{: .tip}

```python
x * (y + z)
```

What do you think the following will result in? The best way to find out is trying things for yourself. If it's wrong, can you correct it?

```python
xy
```

Time to check what we've learned!

> <question-title>Exercise 0: Simple Equations</question-title>
> Given the equation:
>
> $$y = x * 921 + 534$$
>
> What is the value of `y` when `x = 452`
>
> You can test out solutions in the box below this question.
>
> > <solution-title></solution-title>
> > ```
> > x = 452
> > y = x * 921 + 534
> > # Remember to print it out!
> > print(y)
> > ```
> >
> > 416826
> {: .solution}
{: .question}

```python
# Test solutions here!
# By the way, lines starting with a # are comment lines!
# You can use that to take notes, and not affect how your code runes
# People use it for documentation: explaining what the function does,
# what a variable means, why they chose this or that algorithm.
```

## Libraries

A library is a collection of files (called modules) that contains functions for use by other programs. It may also contain data values (e.g., numerical constants) and other things. A library's contents are supposed to be related, but there's no way to enforce that. The Python standard library is an extensive suite of modules that comes with Python itself. Many additional libraries are available from [PyPI (the Python Package Index)](https://pypi.org/).

### Libraries and modules

A library is a collection of modules, but the terms are often used interchangeably, especially since many libraries only consist of a single module, so don't worry if you mix them.

### A program must import a library module before using it.

You can use `import` to load a library module into a program's memory, then refer to things from the `module as module_name.thing_name`. Python uses `.` to mean “part of”. For example, using `datetime`, one of the modules in the standard library:

```python
import datetime

# it is currently
datetime.datetime.now()
```


> <tip-title>More complicated importing</tip-title>
>
> First, you can use `help` to learn about the contents of a library module. it works just like help for a function.
>
> ```
> help(datetime)
> ```
>
> You can import specific items from a library module to shorten programs. You can use `from ... import ...` to load only specific items from a library module. Then refer to them directly without library name as prefix.
>
>
> ```
> from datetime import datetime
>
> datetime.now()
> ```
>
> You can create an alias for a library module when importing it to shorten programs. Use `import ... as ...` to give a library a short alias while importing it. Then refer to items in the library using that shortened name.
>
> ```
> from datetime import datetime as dt
>
> dt.now()
> ```
{: .tip}


## Math Module

Let's import the math module:

```python
import math
```

This imports the math module. If you ever don't know what a function does, you can use the `help()` command:

```python
help(math)
```

And you'll see a list of the functions and properties available. Let's try out one of those functions now:

```python
math.sqrt(9)
```

> <solution-title>Tip: Why `math.`?</solution-title>
> When we import a module like `import math`, we need to use that as a prefix. Imagine we had two different modules, `math` and `other_math`, and both have a `sqrt` function. How would Python know which `sqrt` function we wanted? So we use `math.sqrt` to be explicit about which function we need.
{: .solution}

You might also have done powers (e.g. 2 cubed, or $$2^3$$) in the past, too:

```python
math.pow(2, 3)
```

Above we computed 2 cubed, but how did we know what to write? We might have seen `math.pow` in the `help(math)` page above, but how did we know `2, 3`? The same way of course:

```python
help(math.pow)
```

That would tell us that if we want 2 cubed, we need to write `2` and `3`. So let's do some exercises now and practice some of our maths skills.


> <question-title>Exercise 1: Basics</question-title>
> Please convert this function from an equation, into python code:
>
> $$x = 2^8$$
>
> > <solution-title></solution-title>
> > ```
> > x = math.pow(2, 8)
> > print(x)
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
x =
print(x)
```


> <question-title>Exercise 2: Averaging two values</question-title>
> See if you can find the average of these two values, using the math operations
>
> - 23484
> - 12345
>
> You can find the average by summing, and dividing by 2:
>
> > <solution-title></solution-title>
> > ```
> > (23484 + 12345) / 2
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
```

> <question-title>Exercise 3: Round-trip</question-title>
> Please convert this function from an equation, into python code. Remember, you can assign values to variables, if you want to split this into multiple steps.
>
> $$x = \sqrt{9^2}$$
>
> > <solution-title></solution-title>
> > ```
> > y = math.pow(9, 2)
> > x = math.sqrt(y)
> > ```
> >
> > Or
> >
> > ```
> > x = math.sqrt(math.pow(9, 2))
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
```

> <question-title>Exercise 4: Pythagorean Theorem</question-title>
> The formula for a 90° triangle can be expressed as:
>
> $$a^2 + b^2 = c^2$$
>
> Or, expressed in terms of c, $$c = \sqrt{a^2 + b^2}$$
>
> Please convert this to python and find `c` when
>
> ```
> a = 65
> b = 72
> ```
>
> > <solution-title></solution-title>
> > c = 97
> > ```
> > c = math.sqrt(math.pow(a, 2) + math.pow(b, 2))
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
```

> <question-title>Exercise 5: Quadratic Roots</question-title>
> Way back in algebra class, you might have been given a quadratic equation, something like:
>
> $$y = 2*x^2 + x - 1$$ and were told to find the roots of this function, using a complicated equation. So challenge time: reproduce this equation in Python:
>
> Given the following variables:
>
> a = 2
> b = 1
> c = -1
>
> Convert the following formulas to Python, and find the answers
>
> $$\dfrac{-b + \sqrt{b^2 - 4ac}}{2a}$$ and $$\dfrac{-b - \sqrt{b^2 - 4ac}}{2a}$$
>
> Since we're checking your ability to write Python, not do math, it should give `-1` and `0.5` to let you check your work. If you got those values, you got it right!
>
> Make sure you save each root as it's own variable, and then print them out.
>
> > <solution-title></solution-title>
> > ```
> > root1 = (-b + math.sqrt(math.pow(b, 2) - 4 * a * c))/(2 * a)
> > root2 = (-b - math.sqrt(math.pow(b, 2) - 4 * a * c))/(2 * a)
> > print(root1)
> > print(root2)
> > ```
> {: .solution}
{: .question}

```python
# Test solutions here!
root1 =
root2 =
print(root1)
print(root2)
```


topic: data-science
tutorial: python-testing
title: Python - Testing
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 74672dffb2b52381bf89058cd2994330
last_modified: 2026-02-26T21:12:14.904871
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-testing/tutorial.html

content:
Here we will cover the basics of testing, an important part of software development. Testing lets you know that your code is correct in many situations that matter to you.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}

Being able to demonstrate that a process generates the right results is important in any field of research, whether it's software generating those results or not. So when writing software we need to ask ourselves some key questions:

- Does the code we develop work the way it should do?
- Can we (and others) verify these assertions for themselves?
- Perhaps most importantly, to what extent are we confident of the accuracy of results that appear in publications?

If we are unable to demonstrate that our software fulfills these criteria, why would anyone use it? Having well-defined tests for our software are useful for this, but manually testing software can prove an expensive process.

Automation can help, and automation where possible is a good thing - it enables us to define a potentially complex process in a repeatable way that is far less prone to error than manual approaches. Once defined, automation can also save us a lot of effort, particularly in the long run. In this episode we'll look into techniques of automated testing to improve the predictability of a software change, make development more productive, and help us produce code that works as expected and produces desired results.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## What Is Software Testing?

For the sake of argument, if each line we write has a 99% chance of being right, then a 70-line program will be wrong more than half the time. We need to do better than that, which means we need to test our software to catch these mistakes.

We can and should extensively test our software manually, and manual testing is well-suited to testing aspects such as graphical user interfaces and reconciling visual outputs against inputs. However, even with a good test plan, manual testing is very time consuming and prone to error. Another style of testing is automated testing, where we write code that tests the functions of our software. Since computers are very good and efficient at automating repetitive tasks, we should take advantage of this wherever possible.

There are three main types of automated tests:

- **Unit tests** are tests for fairly small and specific units of functionality, e.g. determining that a particular function returns output as expected given specific inputs.
- **Functional or integration tests** work at a higher level, and test functional paths through your code, e.g. given some specific inputs, a set of interconnected functions across a number of modules (or the entire code) produce the expected result. These are particularly useful for exposing faults in how functional units interact.
- **Regression tests** make sure that your program's output hasn't changed, for example after making changes your code to add new functionality or fix a bug.
- **Property tests** are an advanced testing strategy to find corner cases.

For the purposes of this course, we'll focus on unit tests. But the principles and practices we'll talk about can be built on and applied to the other types of tests too.

## Example Codebase

```python
def rle_encode(input_string):
    count = 1
    prev = ""
    lst = []
    for character in input_string:
        if character != prev:
            if prev:
                entry = (prev, count)
                lst.append(entry)
            count = 1
            prev = character
        else:
            count += 1
    entry = (character, count)
    lst.append(entry)
    return "".join([f"{k}{v}" for k, v in lst])


def rle_decode(lst):
    q = ""
    for character, count in zip(lst[::2], lst[1::2]):
        q += character * int(count)
    return q
```

This is a simple run length encoding and decoding function. It does not handle a lot of corner cases or odd input, but it may do what we need: compressing long strings of repetitive texts. It could potentially provide good genomic data compression. (Spoilers, it generally doesn't. There are better compression algorithms.)

```python
print(rle_encode("aaabbc"))
```

Let's look at how we can test this cod.


## Writing Tests to Verify Correct Behaviour

### One Way to Do It?

One way to test our functions would be to write a series of checks or tests, each executing a function we want to test with known inputs against known valid results, and throw an error if we encounter a result that is incorrect

```python
test_input = "abba"
test_output = "a1b2a1"

assert rle_encode(test_input) == test_output
```

So we use the assert keyword - part of Python - to test that our calculated result is the same as our expected result. This function explicitly checks that the two values are the same, and throws an AssertionError if they are not.

Let's write some more test cases:

```python
test_input = "wwwwssssb"
test_output = "w5s4b1"
assert rle_encode(test_input) == test_output

test_round_trip = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
assert test_round_trip == rle_decode(rle_encode(test_round_trip))

test_input = "baabaa"
test_output = "b1a2b1a2"
assert rle_encode(test_input) == test_output
```

However, if we were to enter these in this order, we’ll find we get the following after the first test:

```
AssertionError                            Traceback (most recent call last)
<ipython-input-30-05489b2ef047> in <module>
      1 test_input = "wwwwssssb"
      2 test_output = "w5s4b1"
----> 3 assert rle_encode(test_input) == test_output

AssertionError: 
```

This tells us our function couldn't handle these inputs.

We could put these tests in a separate script to automate the running of these tests. But a Python script halts at the first failed assertion, so the second and third tests aren't run at all. It would be more helpful if we could get data from all of our tests every time they're run, since the more information we have, the faster we're likely to be able to track down bugs. It would also be helpful to have some kind of summary report: if our set of tests - known as a **test suite** - includes thirty or forty tests (as it well might for a complex function or library that's widely used), we'd like to know how many passed or failed.


Going back to our failed first test, what was the issue? As it turns out, the test itself was incorrect, and should have read:

```python
test_input = "wwwwssssb"
test_output = "w4s4b1"
assert rle_encode(test_input) == test_output
```

Which highlights an important point: as well as making sure our code is returning correct answers, we also need to ensure the tests themselves are also correct. Otherwise, we may go on to fix our code only to return an incorrect result that *appears* to be correct. So a good rule is to make tests simple enough to understand so we can reason about both the correctness of our tests as well as our code. Otherwise, our tests hold little value.

### Using a Testing Framework

Keeping these things in mind, here's a different approach that builds on the ideas we've seen so far but uses a **unit testing framework**. In such a framework we define our tests we want to run as functions, and the framework automatically runs each of these functions in turn, summarising the outputs. And unlike our previous approach, it will run every test regardless of any encountered test failures.

Most people don't enjoy writing tests, so if we want them to actually do it, it must be easy to:

- Add or change tests,
- Understand the tests that have already been written,
- Run those tests, and
- Understand those tests' results

Test results must also be reliable. If a testing tool says that code is working when it's not, or reports problems when there actually aren't any, people will lose faith in it and stop using it.

```python
from main import rle_encode, rle_decode
import unittest

class TestRunLengthEncoding(unittest.TestCase):
    def test_encode(self):
      i = "aaabbcccd"
      expected = "a3b2c3d1"
      actual = rle_encode(i)
      self.assertEqual(expected, actual)

    def test_decode(self):
        i = "a3b2c3d1"
        expected = "aaabbcccd"
        actual = rle_decode(i)
        self.assertEqual(expected, actual)

    def test_empty(self):
        test_input = "wwwwssssb"
        test_output = "w4s4b1"
        self.assertEqual(test_input, test_output)

    def test_roundtrip_short(self):
        test_input = "bbb"
        test_output = rle_decode(rle_encode(test_input))
        self.assertEqual(test_input, test_output)

    def test_roundtrip_long(self):
        test_input = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
        test_output = rle_decode(rle_encode(test_input))
        self.assertEqual(test_input, test_output)

if __name__ == '__main__':
    unittest.main()
```

So here, although we have specified our tests as separate functions in a class, they run the same assertions. Each of these test functions, in a general sense, are called **test cases** - these are a specification of:

- Inputs 
- Execution conditions - any imports we might need to do
- Testing procedure
- Expected outputs

And here, we're defining each of these things for a test case we can run independently that requires no manual intervention.

> <hands-on-title>Run this locally</hands-on-title>
> 1. Create a new folder somewhere
> 2. Save the RLE encode/decode functions as `main.py` in that folder
> 3. Save the unittests above as `test.py` in that folder
> 4. Run `python test.py`
{: .hands_on}

Going back to our list of requirements, how easy is it to run these tests? We can do this using a Python package called `unittest`. This is a built-in testing framework that allows you to write test cases using Python. You can use it to test things like Python functions, database operations, or even things like service APIs - essentially anything that has inputs and expected outputs. 

> <tip-title>What About Unit Testing in Other Languages?</tip-title>
>
> Other unit testing frameworks exist for Python, including Nose and pyunit, and the approach to unit testing can be translated to other languages as well, e.g. FRUIT for Fortran, JUnit for Java (the original unit testing framework), Catch for C++, etc.
{: .tip}


### What About Testing for Errors?

There are some cases where seeing an error is actually the correct behaviour, and Python allows us to test for exceptions. Add this test in `tests/test_models.py`:

```python
import unittest

class ExpectedFailureCase(unittest.TestCase):

    @unittest.expectedFailure
    def test_bad_input():
        from main import rle_decode

        # invalid input, we expect pairs of symbols and numbers.
        rle_decode("a1b")
```

Run all your tests as before.

> <tip-title>Why Should We Test Invalid Input Data?</tip-title>
>
> Testing the behaviour of inputs, both valid and invalid, is a really good idea and is known as *data validation*. Even if you are developing command line software that cannot be exploited by malicious data entry, testing behaviour against invalid inputs prevents generation of erroneous results that could lead to serious misinterpretation (as well as saving time and compute cycles which may be expensive for longer-running applications). It is generally best not to assume your user's inputs will always be rational.
>
{: .tip}

### Testing Frameworks

There are many, many testing frameworks for Python that solve similar problems. Look around and see what fits well with your project!

- [unittest](https://docs.python.org/3/library/unittest.html) is built into Python's stdlib.
- [pytest](https://docs.pytest.org/en/7.1.x/) is a heavily recommended framework.
- [nose](https://nose.readthedocs.io/en/latest/) is an alternative to pytest that does many of the same things.
- [hypothesis](https://hypothesis.readthedocs.io/en/latest/index.html) does property testing, the author's favourite.
- [doctest](https://docs.python.org/3/library/doctest.html) tests examples written directly in the documentation, which keeps test cases close to functions.


topic: data-science
tutorial: data-manipulation-olympics-viz-r
title: Data visualisation Olympics - Visualization in R
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa", "R"]
content_hash: e1908e840a0b1d366249ee9c9775b97f
last_modified: 2026-02-26T21:12:14.849132
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics-viz-r/tutorial.html

content:
In this tutorial, you will learn how to produce scatter plots, boxplots, and time series plots using ggplot. You will also learn how to set universal plot settings, modify the aesthetics of an existing ggplot plots (including axis labels and color), and learn how to facet in ggplot.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [Data Carpentry](https://datacarpentry.org) lesson ["Data visualization with ggplot2"](https://datacarpentry.org/R-ecology-lesson/04-visualization-ggplot2.html).
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data visualisation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.

![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"]({% link topics/introduction/tutorials/data-manipulation-olympics/images/cover.jpg %})

We will show you many of these commonly needed visualisation operations, and some examples of how to perform them in R. We also provide many exercises so that you can train your skills and become a data visualisation Olympian!

## Data Visualization with ggplot2

We start by loading the required packages. `ggplot2` is included in the `tidyverse` package.

```r
library(tidyverse)
```

# Download Data

Before we can do any visualisation, we will need some data. Let's download our table with Olympics results now.

```r
olympics <- read_tsv("{{page.zenodo_link}}/files/olympics.tsv")
```

```r
View(olympics)
```

> <question-title></question-title>
>
> 1. What is the format of the file?
> 2. How is it structured?
> 3. How many lines are in the file?
> 4. How many columns?
>
> > <solution-title></solution-title>
> > 1. When you expand the `olympics.tsv` dataset in your history (see also screenshot below), you will see `format: tabular`, this is another term for a tab-separated (`tsv`) file.
> > 2. Each row represents an athlete's participation in an event. If an athlete competes in multiple events, there is a line for each event.
> > 3. 234,522. Look at the bottom of the View or Environment panels to see this number.
> > 4. There are 17 columns in this file. See View or Environment panels.
> {: .solution}
{: .question}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.tsv` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data visualisation skills in Galaxy.

## Plotting with `ggplot2`

`ggplot2` is a plotting package that provides helpful commands to create complex plots
from data in a data frame. It provides a more programmatic interface for
specifying what variables to plot, how they are displayed, and general visual
properties. Therefore, we only need minimal changes if the underlying data
change or if we decide to change from a bar plot to a scatterplot. This helps in
creating publication quality plots with minimal amounts of adjustments and
tweaking.

`ggplot2` refers to the name of the package itself. When using the package we use the
function `ggplot()` to generate the plots, and so references to using the function will
be referred to as `ggplot()` and the package as a whole as `ggplot2`

`ggplot2` plots work best with data in the 'long' format, i.e., a column for every variable,
and a row for every observation. Well-structured data will save you lots of time
when making figures with `ggplot2`

ggplot graphics are built layer by layer by adding new elements. Adding layers in
this fashion allows for extensive flexibility and customization of plots.

To build a ggplot, we will use the following basic template that can be used for different types of plots:

```
ggplot(data = <DATA>, mapping = aes(<MAPPINGS>)) +  <GEOM_FUNCTION>()
```

use the `ggplot()` function and bind the plot to a specific data frame using the `data` argument

```r
ggplot(data = olympics)
```

- define an aesthetic mapping (using the aesthetic (`aes`) function), by
  selecting the variables to be plotted and specifying how to present them in the
  graph, e.g., as x/y positions or characteristics such as size, shape, color, etc.

```r
ggplot(data = olympics, mapping = aes(x = year, y = height))
```

- add 'geoms' – graphical representations of the data in the plot (points,
  lines, bars). `ggplot2` offers many different geoms; we will use some
  common ones today, including:
  
  - `geom_point()` for scatter plots, dot plots, etc.
  - `geom_boxplot()` for, well, boxplots!
  - `geom_line()` for trend lines, time series, etc.

To add a geom to the plot use `+` operator. Because we have two continuous
variables, let's use `geom_point()` first:

```r
ggplot(data = olympics, aes(x = year, y = height)) +
  geom_point()
```

The `+` in the `ggplot2` package is particularly useful because it allows
you to modify existing `ggplot` objects. This means you can easily set up plot
"templates" and conveniently explore different types of plots, so the above
plot can also be generated with code like this:

```r
# Assign plot to a variable
height_plot <- ggplot(data = olympics,
                       mapping = aes(x = year, y = height))

# Draw the plot
height_plot +
    geom_point()
```

```r
## Create a ggplot and draw it.
height_plot <- ggplot(data = olympics,
                       aes(x = year, y = height))

height_plot +
  geom_point()
```

### Notes

- Anything you put in the `ggplot()` function can be seen by any geom layers
  that you add (i.e., these are universal plot settings). This includes the x-
  and y-axis you set up in `aes()`.
- You can also specify aesthetics for a given geom independently of the
  aesthetics defined globally in the `ggplot()` function.
- The `+` sign used to add layers must be placed at the end of each line
  containing a layer. If, instead, the `+` sign is added in the line before the
  other layer, `ggplot2` will not add the new layer and will return an error
  message.
- You may notice that we sometimes reference 'ggplot2' and sometimes 'ggplot'.
  To clarify, 'ggplot2' is the name of the most recent version of the package.
  However, any time we call the function itself, it's just called 'ggplot'.
- The previous version of the `ggplot2` package, called `ggplot`,
  which also contained the `ggplot()` function is now unsupported and has
  been removed from CRAN in order to reduce accidental installations
  and further confusion.

```r
# This is the correct syntax for adding layers
height_plot +
  geom_point()

# This will not add the new layer and will return an error message
# height_plot
#   + geom_point()
```

> <question-title noprefix>Challenge (optional)</question-title>
> 
> Scatter plots can be useful exploratory tools for small datasets. For data
> sets with large numbers of observations, such as the `olympics` data
> set, overplotting of points can be a limitation of scatter plots. One strategy
> for handling such settings is to use hexagonal binning of observations. The
> plot space is tessellated into hexagons. Each hexagon is assigned a color
> based on the number of observations that fall within its boundaries.  To use
> hexagonal binning with `ggplot2`, first install the R package `hexbin`
> from CRAN:
> 
> ```r
> # install.packages("hexbin")
> library(hexbin)
> ```
> 
> Then use the `geom_hex()` function:
> 
> ```r
> height_plot +
>  geom_hex()
> ```
> 
> What are the relative strengths and weaknesses of a hexagonal bin plot
> compared to a scatter plot? Examine the above scatter plot and compare it
> with the hexagonal bin plot that you created.
> 
{: .question}

## Data Cleaning & Calculations

We'll calculate some new fields to enable us to answer more questions, let's do that now.

```r
olympics <- olympics %>%
    mutate(age = year - birth_year) %>%
    mutate(weight = as.integer(weight)) %>%
    filter(!is.na(weight)) %>%
    filter(!is.na(height))
```

And to speed up future plots, let's pick three countries and three sports we're
interested in to reduce the amount of data we'll need to plot:

```r
sports = c("Archery", "Judo", "Speed Skating")
# You can change this to any of:
# c("Alpine Skiing", "Archery", "Art Competitions", "Artistic Gymnastics", "Artistic Swimming", "Athletics", "Badminton", "Baseball", "Basketball", "Biathlon", "Bobsleigh", "Bowling", "Boxing", "Canoe Marathon", "Canoe Slalom", "Canoe Sprint", "Cross Country Skiing", "Cycling BMX Freestyle", "Cycling BMX Racing", "Cycling Mountain Bike", "Cycling Road", "Cycling Track", "Diving", "Dogsled Racing", "Equestrian Dressage", "Equestrian Eventing", "Equestrian Jumping", "Fencing", "Figure Skating", "Freestyle Skiing", "Golf", "Handball", "Hockey", "Judo", "Karate", "Luge", "Marathon Swimming", "Military Ski Patrol", "Modern Pentathlon", "Nordic Combined", "Rhythmic Gymnastics", "Rowing", "Rugby", "Sailing", "Shooting", "Short Track Speed Skating", "Skateboarding", "Skeleton", "Ski Jumping", "Snowboarding", "Speed Skating", "Speed Skiing", "Surfing", "Swimming", "Table Tennis", "Taekwondo", "Tennis", "Trampolining", "Triathlon", "Tug-Of-War", "Volleyball", "Water Polo", "Weightlifting", "Winter Pentathlon", "Wrestling", "Wushu")

countries = c("NED", "USA", "CHN")

olympics_small <- olympics %>% filter(sport %in% sports) %>% filter(noc %in% countries)
```

## Building your plots iteratively

Building plots with `ggplot2` is typically an iterative process. We start by
defining the dataset we'll use, lay out the axes, and choose a geom:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point()
```

Then, we start modifying this plot to extract more information from it. For
instance, we can add transparency (`alpha`) to avoid overplotting:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point(alpha = 0.1)
```

We can also add colors for all the points:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point(alpha = 0.1, color = "blue")
```

Or to color each species in the plot differently, you could use a vector as an input to the argument **color**. `ggplot2` will provide a different color corresponding to different values in the vector. Here is an example where we color with **`species_id`**:

```r
ggplot(olympics_small, aes(x=age, y=sport)) +
    geom_point(alpha = 0.1, aes(color = sex))
```


> <question-title noprefix>Challenge</question-title>
> 
> Use what you just learned to create a scatter plot of `height` over
> `sport` with the plot types showing the season in different colors.
> Is this a good way to show this type of data?
> 
> > <solution-title></solution-title>
> > 
> > ```r
> > ggplot(data = olympics,
> >        mapping = aes(x = height, y = sport)) +
> >    geom_point(aes(color = season))
> > ```
> {: .solution}
{: .question}


## Boxplot

We can use boxplots to visualize the distribution of height within each sport:

```r
ggplot(data = olympics_small, mapping = aes(x = height, y = sport)) +
    geom_boxplot()
```

But this is a bit boring with all three merged together, so let's colour by NOC.

```r
ggplot(olympics_small, aes(x = height, y = sport, fill=noc)) +
    geom_boxplot()
```

By adding points to the boxplot, we can have a better idea of the number of
measurements and of their distribution. Because the boxplot will show the outliers
by default these points will be plotted twice -- by `geom_boxplot` and
`geom_jitter`. To avoid this we must specify that no outliers should be added
to the boxplot by specifying `outlier.shape = NA`.

```r
ggplot(olympics_small, aes(x = height, y = sport, fill=noc)) +
    geom_boxplot(outlier.shape = NA) +
    geom_jitter(alpha = 0.3, color = "orange")
```

Notice how the boxplot layer is behind the jitter layer? What do you need to
change in the code to put the boxplot in front of the points such that it's not
hidden?

> <question-title noprefix>Challenge</question-title>
> 
> Boxplots are useful summaries, but hide the *shape* of the distribution. For
> example, if there is a bimodal distribution, it would not be observed with a
> boxplot. An alternative to the boxplot is the violin plot (sometimes known as
> a beanplot), where the shape (of the density of points) is drawn.
> 
> - Replace the box plot with a violin plot; see `geom_violin()`.
> 
> > <solution-title></solution-title>
> > ```
> > ggplot(olympics_small, aes(x = height, y = sport)) +
> >     geom_jitter(alpha = 0.3, color = "orange") +
> >     geom_violin()
> > ```
> {: .solution}
{: .question}

In many types of data, it is important to consider the *scale* of the
observations.  For example, it may be worth changing the scale of the axis to
better distribute the observations in the space of the plot.  Changing the scale
of the axes is done similarly to adding/modifying other components (i.e., by
incrementally adding commands). Try making these modifications:

- Represent height on the `log_10` scale; see `scale_y_log10()` and `scale_x_log10()`.

> <solution-title></solution-title>
> 
> ```r
> ggplot(olympics_small, aes(x = height, y = sport)) +
> scale_x_log10() +
> geom_jitter(alpha = 0.3, color = "orange") +
> geom_boxplot(outlier.shape = NA)
> ```
{: .solution}

So far, we've looked at the distribution of height within specific sports. Try making
a new plot to explore the distribution of another variable within each sport!

Create boxplot for `age` by sport.  Overlay the boxplot layer on a jitter
layer to show actual measurements.

> <solution-title></solution-title>
> ```r
> ggplot(olympics_small, aes(x = age, y = sport)) +
> geom_jitter(alpha = 0.3, color = "orange") +
> geom_boxplot(outlier.shape = NA)
> ```
{: .solution}

Add color to the data points on your boxplot according to the year from which
the sample was taken (`year`).

Hint: Check the class for `year`. Consider changing the class of `year`
from integer to factor. Why does this change how R makes the graph?


> <solution-title></solution-title>
> ```r
> ggplot(olympics_small, aes(x = age, y = sport)) +
>   geom_jitter(alpha = 0.3, aes(color=year)) +
>   geom_boxplot(outlier.shape = NA)
>
> # As a factor:
> ggplot(olympics_small, aes(x = age, y = sport)) +
>   geom_jitter(alpha = 0.3, aes(color=as.factor(year))) +
>   geom_boxplot(outlier.shape = NA)
> ```
{: .solution}

## Plotting time series data

Let's calculate number of participants per year for each games. First we need
to group the data and count records within each group:

```r
yearly_counts <- olympics %>% count(year, season)
```

Timelapse data can be visualized as a line plot with years on the x-axis and
counts on the y-axis:

```r
ggplot(data = yearly_counts, aes(x = year, y = n)) +
     geom_line()
```

Unfortunately, this does not work well because our data is quite sparse, datapoints only ever 2 or 4 years.
Let's instead use a box plot

```r
ggplot(data = yearly_counts, aes(x = year, y = n)) +
     geom_col()
```

We can't use `geom_box()` here, instead we should use `geom_col()` as our data is already aggregated.

We need to tell ggplot to draw a line for each season by modifying
the aesthetic function to include `group = season`:

```r
ggplot(data = yearly_counts, aes(x = year, y = n, group = season)) +
    geom_col()
```

We will be able to distinguish season in the plot if we add colors (using
`color` or `fill` also automatically groups the data:

```r
ggplot(data = yearly_counts, aes(x = year, y = n, fill = season)) +
    geom_col()
```

If you want the histograms to be side-by-side, we can do that with the "dodge" positioning:

```r
ggplot(data = yearly_counts, aes(x = year, y = n, fill = season)) +
    geom_col(position = "dodge")
```

## Integrating the pipe operator with ggplot2

In the previous lesson, we saw how to use the pipe operator `%>%` to use
different functions in a sequence and create a coherent workflow.
We can also use the pipe operator to pass the `data` argument to the
`ggplot()` function. The hard part is to remember that to build your ggplot,
you need to use `+` and not `%>%`.

```r
yearly_counts %>%
    ggplot(aes(x = year, y = n, fill = season)) +
    geom_col()
```

The pipe operator can also be used to link data manipulation with consequent data visualization.

```r
yearly_counts_graph <- olympics %>%
    count(year, season) %>%
    ggplot(mapping = aes(x = year, y = n, fill = season)) +
    geom_col()

yearly_counts_graph
```

## Faceting

`ggplot` has a special technique called *faceting* that allows the user to split
one plot into multiple plots based on a factor included in the dataset. We will
use it to make a time series plot for each season separately:

```r
yearly_counts %>%
    ggplot(aes(x = year, y = n, fill = season)) +
    geom_col() + facet_wrap(facets=vars(season))
```

Now we would like to split the line in each plot by the sex of each individual, sport, and their medal placement.
To do that we need to make counts in the data frame grouped by `year`, `season`, `sport`, `medal`, and `noc`.

```r
year_detail_counts <- olympics_small %>%
                      count(year, season, sport, medal, noc)
```

We can now make the faceted plot by splitting further by medal using `color`
(within a single plot), and per NOC:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_wrap(facets =  vars(noc))
```

We can also facet both by NOC and sport:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_grid(rows = vars(noc), cols =  vars(sport))
```

You can also organise the panels only by rows (or only by columns):

```r
# One column, facet by rows
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_grid(rows = vars(noc))
```

```r
# One row, facet by column
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal)) +
  geom_line() +
  facet_grid(cols = vars(noc))
```

> <tip-title>facet_grid/wrap with ~</tip-title>
> `ggplot2` before version 3.0.0 used formulas to specify how plots are faceted.
> If you encounter `facet_grid`/`wrap(...)` code containing `~`, please read
> [https://ggplot2.tidyverse.org/news/#tidy-evaluation](https://ggplot2.tidyverse.org/news/#tidy-evaluation).
{: .tip}

## `ggplot2` themes

Usually plots with white background look more readable when printed.
Every single component of a `ggplot` graph can be customized using the generic
`theme()` function, as we will see below. However, there are pre-loaded themes
available that change the overall appearance of the graph without much effort.

For example, we can change our previous graph to have a simpler white background
using the `theme_bw()` function:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, color = medal))  +
    geom_line() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    theme_bw()
```

In addition to `theme_bw()`, which changes the plot background to white, `ggplot2`
comes with several other themes which can be useful to quickly change the look
of your visualization. The complete list of themes is [documented in the ggthemes reference](https://ggplot2.tidyverse.org/reference/ggtheme.html). `theme_minimal()` and
`theme_light()` are popular, and `theme_void()` can be useful as a starting
point to create a new hand-crafted theme.

The
[ggthemes](https://jrnold.github.io/ggthemes/reference/index.html) package
provides a wide variety of options.


> <question-title noprefix>Challenge</question-title>
> Use what you just learned to create a plot the relationship between height and weight,
> of participants, broken down by NOC and Sport.
> > <solution-title></solution-title>
> > ```r
> > ggplot(data = olympics_small, mapping = aes(x = height, y = weight, color = medal)) +
> >     geom_point() +
> >     facet_grid(rows = vars(noc), cols = vars(sport))
> > ```
> {: .solution}
{: .question}

## Customization

Take a look at the [`ggplot2` cheat sheet](https://posit.co/wp-content/uploads/2022/10/data-visualization-1.pdf), and
think of ways you could improve the plot.

Now, let's change names of axes to something more informative than 'year'
and 'n' and add a title to the figure:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
```

The axes have more informative names, but their readability can be improved by
increasing the font size. This can be done with the generic `theme()` function:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
    theme(text=element_text(size = 16))
```

Note that it is also possible to change the fonts of your plots. If you are on
Windows, you may have to install
the [`extrafont` package](https://github.com/wch/extrafont), and follow the
instructions included in the README for this package.

After our manipulations, you may notice that the values on the x-axis are still
not properly readable. Let's change the orientation of the labels and adjust
them vertically and horizontally so they don't overlap. You can use a 90 degree
angle, or experiment to find the appropriate angle for diagonally oriented
labels. We can also modify the facet label text (`strip.text`) to italicize the genus
names:

```r
ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
    theme(axis.text.x = element_text(colour = "grey20", size = 12, angle = 90, hjust = 0.5, vjust = 0.5),
                        axis.text.y = element_text(colour = "grey20", size = 12),
                        strip.text = element_text(face = "italic"),
                        text = element_text(size = 16))
```

If you like the changes you created better than the default theme, you can save
them as an object to be able to easily apply them to other plots you may create:

```r
grey_theme <- theme(axis.text.x = element_text(colour="grey20", size = 12,
                                               angle = 90, hjust = 0.5,
                                               vjust = 0.5),
                    axis.text.y = element_text(colour = "grey20", size = 12),
                    text=element_text(size = 16))

ggplot(data = year_detail_counts, mapping = aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    grey_theme
```


> <question-title noprefix>Challenge</question-title>
>
> With all of this information in hand, please take another five minutes to either
> improve one of the plots generated in this exercise or create a beautiful graph
> of your own. Use the RStudio [`ggplot2` cheat sheet](https://posit.co/wp-content/uploads/2022/10/data-visualization-1.pdf)
> for inspiration.
> 
> Here are some ideas:
> 
> - See if you can change the plot type to another plot
> - Can you find a way to change the name of the legend? What about its labels?
> - Try using a different color palette (see [https://r-graphics.org/chapter-colors](https://r-graphics.org/chapter-colors)).
>
> > <solution-title></solution-title>
> > This optional exercise currently lacks solutions. If you have them, please feel free to contribute suggestions here :)
> {: .solution}
>
{: .question}


## Arranging plots

Faceting is a great tool for splitting one plot into multiple plots, but
sometimes you may want to produce a single figure that contains multiple plots
using different variables or even different data frames. The `patchwork`
package allows us to combine separate ggplots into a single figure while keeping
everything aligned properly. Like most R packages, we can install `patchwork`
from CRAN, the R package repository, if it isn't already available:

```r
# install.packages("patchwork")
```

After you have loaded the `patchwork` package you can use `+` to place plots
next to each other, `/` to arrange them vertically, and `plot_layout()` to
determine how much space each plot uses:

```r
library(patchwork)

plot_weight <- olympics_small %>% ggplot(aes(x=noc, y=weight)) +
  geom_boxplot() +
  labs(x = "NOC", y = expression(log[10](Weight))) +
  scale_y_log10()

plot_height <- olympics_small %>% ggplot(aes(x=noc, y=height)) +
  geom_boxplot() +
  labs(x = "NOC", y = expression(log[10](Height))) +
  scale_y_log10()

plot_weight / plot_height + plot_layout(heights = c(3, 2))
```

You can also use parentheses `()` to create more complex layouts. There are
many useful examples on the [patchwork website](https://patchwork.data-imaginist.com/)

## Exporting plots

After creating your plot, you can save it to a file in your favorite format. The
Export tab in the **Plot** pane in RStudio will save your plots at low
resolution, which will not be accepted by many journals and will not scale well
for posters. The [`ggplot2` extensions website](https://exts.ggplot2.tidyverse.org/) provides a list
of packages that extend the capabilities of `ggplot2`, including additional
themes.

Instead, use the `ggsave()` function, which allows you to easily change the
dimension and resolution of your plot by adjusting the appropriate arguments
(`width`, `height` and `dpi`):

```r
my_plot <- year_detail_counts %>% ggplot(aes(x = year, y = n, fill = medal))  +
    geom_col() +
    facet_grid(rows = vars(noc), cols =  vars(sport)) +
    labs(title = "Participants and medals over the years",
         x = "Year",
         y = "Number of individuals") +
    theme_bw()
    theme(axis.text.x = element_text(colour = "grey20", size = 12, angle = 90, hjust = 0.5, vjust = 0.5),
                        axis.text.y = element_text(colour = "grey20", size = 12),
                        strip.text = element_text(face = "italic"),
                        text = element_text(size = 16))

ggsave("name_of_file.png", my_plot, width = 15, height = 10)

## This also works for plots combined with patchwork
plot_combined <- plot_weight / plot_height + plot_layout(heights = c(3, 2))
ggsave("plot_combined.png", plot_combined, width = 10, dpi = 300)
```

Note: The parameters `width` and `height` also determine the font size in the
saved plot.

```r
### Final plotting challenge:
##  With all of this information in hand, please take another five
##  minutes to either improve one of the plots generated in this
##  exercise or create a beautiful graph of your own. Use the RStudio
##  ggplot2 cheat sheet for inspiration:
##  https://posit.co/wp-content/uploads/2022/10/data-visualization-1.pdf
```


topic: data-science
tutorial: python-basics-recap
title: Python - Introductory Graduation
difficulty: introductory
time_estimation: 1H30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: b72bb235ded6b5449a31d3e038a87636
last_modified: 2026-02-26T21:12:14.900266
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-basics-recap/tutorial.html

content:
This module provide something like a recap of everything covered by the modular Python Introductory level curriculum. This serves as something of a graduation into the Intermediate tutorials which cover more advanced topics.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Review

This recapitulates the main points from all of the previous modular tutorials

## Math

Math in python works a lot like math in real life (from algebra onwards). Variables can be assigned, and worked with in the place of numbers

```python
x = 1
y = 2
z = x * y
```

We can use familiar math operators:

Operator | Operation
--- | ---
`+` | Addition
`-` | Subtraction
`*` | Multiplication
`/` | Division (`//` for rounded, integer division)

And some familiar operations require the use of the `math` module:

```python
import math
print(math.pow(2, 8))
print(math.sqrt(9))
```

## Functions

Functions were similarly analogous to algebra and mathematics, we can express `f(x) = x * 3` in python as:

```python
def f(x, y=3):
    z = y * 2
    return x * z
```

There are a few basic parts of a function:

- `def` starts a function *definition*
- it needs a name, here it's `f`
- between some parentheses are the arguments
    - the are arguments, just the variable name (`x`)
    - and keyword arguments, where there is a variable and a value (`y=3`)
- The function body
    - With one or more lines
    - Usually ending in a `return`

And we know we can nest functions, using functional composition, just like in math. In math functional composition was written `f(g(x))` and in python it's exactly the same:

```python
print(math.sqrt(math.pow(2, 4)))
```

Here we've nested three different functions (print is a function!). To read this we start in the middle (math.pow) and move outwards (math.sqrt, print).

## Types

There are lots of different datatypes in Python! The basic types are `bool`, `int`, `float`, `str`. Then we have more complex datatypes like `list` and `dict` which can contain the basic types (as well as other lists/dicts nested.)

Data type       | Examples                  | When to use it | When **not** to use it
---             | ---                       | ---- | ----
Boolean (`bool`) | `True`, `False` | If there are only two possible states, true or false | If your data is not binary
Integer (`int`) | 1, 0, -1023, 42           | Countable, singular items. How many patients are there, how many events did you record, how many variants are there in the sequence | If doubling or halving the value would not make sense: do not use for e.g. patient IDs, or phone numbers. If these are integers you might accidentally do math on the value.
Float (`float`) | 123.49, 3.14159, -3.33334 | If you need more precision or partial values. Recording distance between places, height, mass, etc.
Strings (`str`) | 'patient_12312', 'Jane Doe', '火锅' | To store free text, identifiers, sequence IDs, etc. | If it's truly a numeric value you can do calculations with, like adding or subtracting or doing statistics.
List / Array (`list`) | `['A', 1, 3.4, ['Nested']]` | If you need to store a list of items, like sequences from a file. Especially if you're reading in a table of data from a file. | If you want to retrieve individual values, and there are clear identifiers it might be better as a dict.
Dictionary / Associative Array / map (`dict`) | `{"weight": 3.4, "age": 12, "name": "Fluffy"}` | When you have identifiers for your data, and want to look them up by that value. E.g. looking up sequences by an identifier, or data about students based on their name. Counting values. | If you just have a list of items without identifiers, it makes more sense to just use a list.

There are a couple more datatypes we didn't cover in detail: `set`s, `tuple`, `None`, `enum`, `byte`, all of which [can be read about in Python's documentation.](https://docs.python.org/3/library/datatypes.html)

## Comparators

We have a couple of comparators available to use specifically for numeric values:

- `>`: greater than
- `<`: less than
- `>=`: greater than or equal to
- `<=`: less than or equal to

And a couple that can be used with numbers and strings (or other values!)

- `==`: equal to
- `!=`: does not equal

## Iterables

In Python, there is a class of things which can be easily looped over, called "iterables". All of the following are examples of iterable items:

- `range(10)`
- `'abcd'`, a string
- `['a', 'b' , 'c' , 'd']`, a list

## Flow Control

Basic flow control looks like `if`, `elif`, `else`:

```python
blood_oxygen_saturation = 92.3
altitude = 0

if blood_oxygen_saturation > 96 and altitude == 0:
    print("Healthy individual at sea level")
elif blood_oxygen_saturation > 92 and altitude >= 1000:
    print("Healthy value above sea level")
elif blood_oxygen_saturation == 0:
    print("Monitor failure")
else:
    print("Not good")
```

We must start with an `if`, then we can have one or more `elif`s, and 0 or 1 else to end our clause. If there is no `else`, it's like nothing happens, we just check the `if` and `elif`s and if none match, nothing happens by default.

We could use `and` to check if both conditions are true, and `or` to check if one condition is true.

```python
# And
for i in (True, False):
    for j in (True, False):
        print(f"{i} AND {j} => {i and j}")
# Or
for i in (True, False):
    for j in (True, False):
        print(f"{i} OR {j} => {i or j}")
```

And if we needed, we can invert conditions with not.

```python
# Not
for i in (True, False):
    print(f"NOT {i} => {not i}")
```

All of these components (`if/elif/else`, `and`, `or`, `not`, numerical and value comparators) let us build up

## Loops

Loops let us *loop* over the contents of something iterable (a string, a list, lines in a file). We write

```
for loopVariable in myIterable:
    # Do something
    print(loopVariable)
```

Each loop has:

- `for`, a keyword to start the loop
- a loop variable, here named `loopVariable` which is set automatically every iteration of the loop
- `in`, a keyword used in a loop
- something we want to iterate over like a list or string (which is really just a list of single characters) or lines in a file.
- a loop body where all the action happens

```python
a = 0
c = 0
t = 0
g = 0

for base in 'ACTGATGCYGGCA':
    if base == 'A':
        a = a + 1
    elif base == 'C':
        c = c + 1
    elif base == 'T':
        t = t + 1
    elif base == 'G':
        g = g + 1
    else:
        print("Unexpected base!")

print(f"a={a} c={c} t={t} g={g}")
```

## Files

In python you must `open()` a file handle, using one of the three modes (read, write, or append). Normally you must also later `close()` that file, but your life can be a bit

```python
with open('out.txt', 'w') as handle:
    handle.write("Здравствуйте ")
    handle.write("世界!\n")
    handle.write("Welcome!\n")

# Can no longer handle.write(), once we've exited the with block, the file is automatically closed for us.
```

There are several basic parts

- We use `with` to indicate we want to use the context manager for file opening (this is what automatically closes the file afterwards)
- `open(path, mode)` opens a file
- `as` is a keyword
- `handle` is the name of a file handle, something that represents the file which we can write to, or read from.

Additionally if you need a newline in your file, you *must* write it yourself with a `\n`.

The above code is equivalent to this, but it is not recommended, it's a bit harder to read, and it is very very common to forget to close files which is not ideal.

```python
handle = open('out.txt', 'w')
handle.write("Здравствуйте ")
handle.write("世界!\n")
handle.write("Welcome!\n")
handle.close()

# Can no longer write.
```

You can also read from a file:

```python
# Read the entire file as one giant string
with open('out.txt', 'r') as handle:
    print(handle.read())

# Or read it as separate lines.
with open('out.txt', 'r') as handle:
    print(handle.readlines())
```

## Exceptions

Sometimes things go wrong!

- You divide by zero (`ZeroDivisionError`)
- You try and add a string and a number (`TypeError`)
- You have incorrect indentation (`IndentationError`)
- Files are unreadable
- Files have the wrong permissions

If you expect that something will go wrong you can guard against it with a `try`/`except`

```python
try:
    print(1 / 0)
except ZeroDivisionError:
    print("Nope! I expected that!")
```

Some of the most common reasons to do this are when you're processing user input data. Users often input invalid data, unfortunately.

# Exercises

## Series Approximation

Write a program that computes the sum of an alternating series where each element of the series is an expression of the form

$$4\cdot\sum_{k=1}^{N} \dfrac{(-1)^{k+1}}{2 * k-1}$$

Use that expression and calculate the sum for various values of N like `10`, `1000`, `1000000`

```python
# Write your approximation here!
def calculate_sum(N):
    # for values in the range [1, N] (inclusive!)
        # calculate the expression (the bit after the Sigma)
    # sum up all of those numbers!
    return value

print(calculate_sum(10))
```

## Monte Carlo Simulation

You can use a monte carlo simulation to calculate the value of π. The easy way to do this is to take the region `x = [0, 1], y = [0, 1]`, and fill it with random points. For each point, calculate the distance to the origin. Calculate the ratio of the inside points to the total points, and multiply the value by 4 to estimate π.

You can use the `random` module to generate random values:

```python
import random

print(random.random())
print(random.random())
print(random.random())
```

Using the `random.random()` to generate x and y coordinates, write a function that:

- generates N random points
- calculate their distance to the origin
- Calculate the number of points that are `distance<=1`
- divide that by `N`, and multiply by `4`.

![Gif of the unit square 0, 1 on the x and y axes. Points are being randomly generated and points inside the unit circle are marked in red, outside in blue. A quarter of a circle starts to appear, and the numbers at the top show as N increases, the approximation of Pi improves.](https://upload.wikimedia.org/wikipedia/commons/8/84/Pi_30K.gif "Gif from the Wikipedia Article for Monte Carlo Method")

```python
# Write your code here!
import math

# Just a suggestion: write a function to calculate the distance to origin.
#
def distance(x, y):
    # ...
    return

def generate_random_point():
    # ....
    return [x, y]

def approximate(N=1000):
    # For every point in the range [0, N]
    #   check if it's distance is great than 1

    # Find the ratio of how many are distance<=1
    # and return 4 times that ratio.
    return 4 * x

# Try it with a couple N values like 1, 100, 100000,
n = 10000
# Since we're using a random number function, the result is different every
# time we run the simulation.
print(approximate(n))
print(approximate(n))
print(approximate(n))
```

## Sixpack

Sixpack is an old EMBOSS program which takes in a DNA sequence, and then for every frame, for both strands, emits every {ORF} that it sees.

```
  G  R  G  F  W  C  L  G  G  K  A  A  K  N  Y  R  E  K  S  V  D  V  A  G  Y  D  X   F1
   G  V  A  S  G  A  W  A  V  K  R  Q  K  T  T  V  K  S  R  W  M  W  R  V  M  M     F2
    A  W  L  L  V  P  G  R  *  S  G  K  K  L  P  *  K  V  G  G  C  G  G  L  *  X    F3
1 GGGCGTGGCTTCTGGTGCCTGGGCGGTAAAGCGGCAAAAAACTACCGTGAAAAGTCGGTGGATGTGGCGGGTTATGATG 79
  ----:----|----:----|----:----|----:----|----:----|----:----|----:----|----:----
1 CCCGCACCGAAGACCACGGACCCGCCATTTCGCCGTTTTTTGATGGCACTTTTCAGCCACCTACACCGCCCAATACTAC 79
   P  R  P  K  Q  H  R  P  P  L  A  A  F  F  *  R  S  F  D  T  S  T  A  P  *  S     F6
  X  A  H  S  R  T  G  P  R  Y  L  P  L  F  S  G  H  F  T  P  P  H  P  P  N  H  H   F5
    P  T  A  E  P  A  Q  A  T  F  R  C  F  V  V  T  F  L  R  H  I  H  R  T  I  I    F4
```

Here we see a DNA sequence `GGGCGTGGCTTCTGGTGCCTGGGCGGTAAAGCGGCAAAAAACTACCGTGAAAAGTCGGTGGATGTGGCGGGTTATGATG` which you'll use as input. Above is the translation of the sequence to protein, for each of the three frames (F1-6). Below is the reverse complement of the sequence, and the three frame translation again.

What sixpack does is:

```
orfs = []

for sequence in [forward, reverse_complement(forward)]:
    for frame in [sequence, sequence[1:], sequence[2:]]:
        # Remembering
        for potential start_codon:
            # accumulate until it sees a stop codon
            # and append it to the orfs array once it does.
```

Here are some variables for your convenience:

```python
start_codons = ['TTG', 'CTG', 'ATG']
stop_codons = ['TAA', 'TAG', 'TGA']

# And some convenience functions
def is_start_codon(codon):
    return codon in start_codons

def is_stop_codon(codon):
    return codon in stop_codons
```

It's a good exercise to rewrite `sixpack` in a very simplified version without most of the features in sixpack:

```python
# Write your code here!

# Some recommendations:
def reverse_complement(sequence):
    return ...

orfs = []

for sequence in [forward, reverse_complement(forward)]:
    for frame in [sequence, sequence[1:], sequence[2:]]:
        # Remembering
        for potential start_codon:
            # accumulate until it sees a stop codon
            # and append it to the orfs array once it does.
```


topic: data-science
tutorial: python-warmup-stat-ml
title: Python - Warm-up for statistics and machine learning
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["elixir", "ai-ml"]
content_hash: 34bea2aa5e1d4cf9c000cc69cba73367
last_modified: 2026-02-26T21:12:14.905901
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-warmup-stat-ml/tutorial.html

content:
> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Basic python



```python

X = []

for i in range(10):
    X.append( i**2 )

print(X)
```

    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]



```python

for x in X:
    print(x)

```

    0
    1
    4
    9
    16
    25
    36
    49
    64
    81



```python
for x in X:
    if x%2 == 1:
        print(x,'is odd')
    else:
        print(x,'is even')
```

    0 is even
    1 is odd
    4 is even
    9 is odd
    16 is even
    25 is odd
    36 is even
    49 is odd
    64 is even
    81 is odd



```python
# list comprehension is a very fine way of compressing all this

X = [ i**2 for i in range(10) ]

Xeven = [ x for x in X if x%2 == 0 ]
Xodd = [ x for x in X if x%2 == 1 ]


print( 'X    ', X )
print( 'Xeven', Xeven )
print( 'Xodd ', Xodd )
```

    X     [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
    Xeven [0, 4, 16, 36, 64]
    Xodd  [1, 9, 25, 49, 81]


[back to the top](#top)

## numpy and vectorized operations


```python
import numpy as np

X_array = np.array(X)

print(X_array)
```

    [ 0  1  4  9 16 25 36 49 64 81]



```python
print(X_array / 2 )
```

    [ 0.   0.5  2.   4.5  8.  12.5 18.  24.5 32.  40.5]



```python
print( np.exp(X_array ) )
print( np.log(X_array ) )
```

    [1.00000000e+00 2.71828183e+00 5.45981500e+01 8.10308393e+03
     8.88611052e+06 7.20048993e+10 4.31123155e+15 1.90734657e+21
     6.23514908e+27 1.50609731e+35]
    [      -inf 0.         1.38629436 2.19722458 2.77258872 3.21887582
     3.58351894 3.8918203  4.15888308 4.39444915]


    /tmp/ipykernel_490123/2855859755.py:2: RuntimeWarning: divide by zero encountered in log
      print( np.log(X_array ) )



```python
print( 'shape' , X_array.shape )
print( 'mean ' , np.mean(X_array) )
print( 'standard deviation' , np.std(X_array) )
```

    shape (10,)
    mean  28.5
    standard deviation 26.852374196707448


### linspace and arange


```python
print( 'linspace 0,2,9 :' , np.linspace(0,2,9) , sep='\t' )
print( 'linspace -0.5,0.5,11 :' , np.linspace(-0.5,0.5,11) , sep='\t' )
print( 'linspace 10,0,11 :' , np.linspace(10,0,11) , sep='\t' )
```

    linspace 0,2,9 :	[0.   0.25 0.5  0.75 1.   1.25 1.5  1.75 2.  ]
    linspace -0.5,0.5,11 :	[-0.5 -0.4 -0.3 -0.2 -0.1  0.   0.1  0.2  0.3  0.4  0.5]
    linspace 10,0,11 :	[10.  9.  8.  7.  6.  5.  4.  3.  2.  1.  0.]



```python
print( "arange 0,2,0.1 :", np.arange(1.5,2,0.1) , sep='\t' )
print( "arange -1,1,0.125 :", np.arange(-1,1,0.125) , sep='\t' )
print( "arange 10,2 :", np.arange(10,2,1) , sep='\t' ) # reverse does not work!
```

    arange 0,2,0.1 :	[1.5 1.6 1.7 1.8 1.9]
    arange -1,1,0.125 :	[-1.    -0.875 -0.75  -0.625 -0.5   -0.375 -0.25  -0.125  0.     0.125
      0.25   0.375  0.5    0.625  0.75   0.875]
    arange 10,2 :	[]


## Basic plotting


```python
import matplotlib.pyplot as plt

plt.plot( [0,1,2,3] , [10,5,7,0.2] )
plt.show()
```



### Adding color, symbols, ...

`matplotlib` offers many options to customize the appearance of your plot.

Here are the (some) common arguments to `plot()` (which can also be applied to many other graphical representations):
 * `color` : could be given as a (red,green,blue) tuple, a [name](https://matplotlib.org/3.1.0/gallery/color/named_colors.html), a hex code, ...  (see [Something better here](https://matplotlib.org/tutorials/colors/colors.html) for all the options)
 * `marker` : symbols for the data point. `'.'` is a point, `'v'` a down triangle, ... see [Something better here](https://matplotlib.org/3.3.3/api/markers_api.html#module-matplotlib.markers) for the list of possibilities.
 * `linestyle` : style of the line. `'-'` is solid, `'--'` is dashed, `''` for no line. See [Something better here](https://matplotlib.org/3.3.3/gallery/lines_bars_and_markers/linestyles.html) for more options
 * `linewidth` : width of the lines
 * `markersize` : size of the markers

You are invited to experiment and explore these options. Here are a few examples:



```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]

# green, dashed line, with circle markers
plt.plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# blue triangle with no line
plt.plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )

# solid orange line
plt.plot(y3, color = 'orange', marker = '', linestyle = '-', linewidth = 4 )

plt.show()
```
    


Note that:
 * you can call plot several time in a row to make several lines appear (only `plt.show()` causes the figure to appear)
 * the frame of the picture automatically adjust to what it needs to show

### multiple subplots

Now would normally be when we show you how to add labels, titles and legends to figures. 

However, the way `matplotlib` is built, it is actually a bit more efficient to first learn how to create multiple subplots.


Creating multiple plots is possible with the function `plt.subplots()`.
Amon its many arguments, it takes:
 * `nrows` : number of subplot rows
 * `ncols` : number of subplot columns
 * `figsize` : tuple (width,height) of the figure

This function creates a Figure and an Axes object.
The Axes object can be either : 
 * a simple Axe is there is 1 row and 1 columns
 * a list of Axe objects if there is 1 row and multiple columns, or 1 column and multiple rows
 * a list of lists of Axes objects if there is multiple rows and multiple columns



```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]


# subplots returns a Figure and an Axes object
fig, ax = plt.subplots(nrows=1, ncols=2) # 2 columns and 1 row

# ax is a list with two objects. Each object correspond to 1 subplot

# accessing to the first column ax[0]
ax[0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# accessing to the second column ax[1]
ax[1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )
ax[1].plot( y3, color = 'orange', marker = '', linestyle = '-' )

plt.show()
```
    


Notice how we call `ax[0].plot(...)` instead of `plt.plot(...)` to specify in which subplots we want to plot.

### multiple subplots - continued

Let's see the same thing with several lines and several columns


```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]
y4 = [1,2,3,7,5]

# 2 columns and 2 rows, and we also set the figure size
fig, ax = plt.subplots(nrows=2, ncols=2 , figsize = (12,12))

# ax is a list of two lists with two objects each.

# accessing to the first row, first column : ax[0][0]
ax[0][0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# accessing to the first row, second column : ax[0][1]
ax[0][1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )

# accessing to the second row, first column : ax[1][0]
ax[1][0].plot( y3, color = 'orange', marker = 'x', linestyle = '-' )

# accessing to the first row, second column : ax[1][1]
ax[1][1].plot( y4, color = 'teal', linestyle = '-.' , linewidth=5 )

plt.show()
```
    


### setting up labels

To set the labels at the x-axis, y-axis and title, we use the method of the Axe object:
 * `.set_xlabel(...)`
 * `.set_ylabel(...)`
 * `.set_title(...) `



```python
y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]

# subplots returns a Figure and an Axes object
fig, ax = plt.subplots(nrows=1, ncols=2 , figsize=(10,5)) # 2 columns and 1 row


# accessing to the first column ax[0]
ax[0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )
ax[0].set_xlabel('x-axis label')
ax[0].set_ylabel('y-axis label')
ax[0].set_title('plot 1')


# accessing to the second column ax[1]
ax[1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )
ax[1].plot( y3, color = 'orange', marker = '', linestyle = '-' )
ax[1].set_xlabel('x-axis label')
ax[1].set_ylabel('y-axis label')
ax[1].set_title('plot 2')

plt.show()
```
    


**setting up a legend** 

Each element we add to the figure using `plot()` can be given a label using the `label` argument.
Then, a legend may be added to the figure using the `legend()` method.

This `legend()` method can take a `loc` argument that specifies where it should be plotted. 
Possible values for this argument are: `'best' , 'upper right' , 'upper left' , 'lower left' , 'lower right' , 'right' , 'center left' , 'center right' , 'lower center' , 'upper center' , 'center'` (the default is `best`).



```python

fig, ax = plt.subplots(nrows=1, ncols=1 , figsize=(10,5)) # 2 columns and 1 row

# NB : with 1 col and 1 row, ax is directly the sole subplot we have
#      so to call it we just use ax.plot , ax.set_xlabel , ...

ax.plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2 , label = 'line A' )
ax.plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize =  8 , label = 'line B' )
ax.plot( y3, color = 'orange', marker = '', linestyle = '-' , linewidth = 2 , label = 'line C' )

ax.set_xlabel('x-axis label')
ax.set_ylabel('y-axis label')
ax.set_title('plot with a legend')

#adding a legend in the upper right
ax.legend( loc='upper right')

plt.show()

```

    


### additional : writing a figure to a file

Writing a matplotlib figure to a file can be achieved simply by replacing the call to `plt.show()` to `plt.savefig(...)`.

`plt.savefig` takes a number of argument, the most commons are :
 * `fname` : name of the file to write the figure. The extension is used to determine the output format (.pdf,.png, .jpg , .svg ,  ...). Many formats are supported, you can get a list with this command : `plt.gcf().canvas.get_supported_filetypes()`
 * `dpi` : dots per inches , useful to set-up when saving to raster formats (ie., pixel-based such as png or jpeg). The actual size of the image is set using the argument `figsize` of `plt.subplots()`


> <comment-title></comment-title>
>
> in a jupyter notebook the figure will still be shown, whereas in a standard .py script it will not appear on screen.
>
{: .comment}

Here is a demonstration. Apply in on your side and verify that the file `testPlot.png` was created:


```python
import matplotlib.pyplot as plt

y1 = [1,2,3,10,5]
y2 = [10,9,7,5.5,6]
y3 = [4,3,1.5,1]


# subplots returns a Figure and an Axes object
fig, ax = plt.subplots(nrows=1, ncols=2 , figsize = (10,6) ) # 2 columns and 1 row

# ax is a list with two objects. Each object correspond to 1 subplot

# accessing to the first column ax[0]
ax[0].plot( y1, color = 'green', marker = 'o', linestyle = '--', linewidth = 2, markersize = 8 )

# accessing to the second column ax[1]
ax[1].plot( y2, color = 'blue', marker = 'v', linestyle = '' , markersize = 16 )
ax[1].plot( y3, color = 'orange', marker = '', linestyle = '-' )

plt.savefig( 'testPlot.png' , dpi = 90  )
```


## Exercise 00.01 : bringing together numpy and matplotlib

Numpy arrays can be plotted as if they were lists.

1. plot x and y, where:
    * y = 1/(1+exp(-x))
    * x varies between -5 and 5 (plotting around a 100 points should suffice).

2. **Bonus :** plot multiples lines : y = 1/(1+exp(-x*b)) , for the following values of b: 0.5 , 1 , 2 , 4.
    * x still varies between -5 and 5 (plotting around a 100 points should suffice).
    * put a legend in your plot




```python

```

You can load the solution directly in this notebook by uncommenting and running the following line:


```python
# %load  -r -8 solutions/solution_00_01.py
```

bonus question solution:


```python
# %load  -r 9- solutions/solution_00_01.py
```


## Generating random numbers


### the basics


```python
import numpy.random as rd

# random floats between 0 and 1
for i in range(4):
    print( rd.random() )

```

    0.6696103730869407
    0.7426639266737763
    0.6767219223242785
    0.8602105555191791



```python
print( rd.random(size=10) ) # draw directly 10 numbers
```

    [0.37971723 0.80354745 0.4168427  0.70867247 0.17547126 0.43760884
     0.75933345 0.06571168 0.45772397 0.67191214]


### setting the seed: pseudorandomness and reproducibility


```python
rd.seed(42) # setting the seed to 42
print( '1st draw' , rd.random(size=5) )
print( '2nd draw' , rd.random(size=5) )
rd.seed(42)
print( 'after resetting seed' , rd.random(size=5) )
```

    1st draw [0.37454012 0.95071431 0.73199394 0.59865848 0.15601864]
    2nd draw [0.15599452 0.05808361 0.86617615 0.60111501 0.70807258]
    after resetting seed [0.37454012 0.95071431 0.73199394 0.59865848 0.15601864]


### beyond the uniform distribution

numpy offers you quite a large [set of distributions you can draw from](https://docs.scipy.org/doc/numpy-1.15.0/reference/routines.random.html#distributions).

Let's look at the normal distribution:


```python

normalDraw = rd.normal(size = 1000 )

print( 'mean ' , np.mean( normalDraw ) )
print( 'stdev' , np.std( normalDraw ) )
```

    mean  0.025354699638558926
    stdev 1.0003731428167348



```python
normalDraw2 = rd.normal( loc = -2 , scale = 3 , size = 300 ) # loc chnages the location (mean), and scale changes the standard deviation

print( 'mean ' , np.mean( normalDraw2 ) )
print( 'stdev' , np.std( normalDraw2 ) )
```

    mean  -1.9773491637651965
    stdev 2.964622032924749


of course, we could want to plot these drawn numbers:


```python
plt.hist( normalDraw  , alpha = 0.5 , label='loc=0  , scale=1')
plt.hist( normalDraw2 , alpha = 0.5 , label='loc=-2 , scale=3')
plt.legend()
plt.show()
```


## Statistical testing

`numpy.random` let's you draw random numbers ;
`scipy.stats` implements the probability density functions, and Percent point function, as well as the most statistical tests.



```python
import scipy.stats as stats

# plotting the probability density function for 1 of the random draw we just made:

x = np.linspace(-10,10,1001)

normPDF = stats.norm.pdf( x , loc = -2 , scale = 3 )

plt.hist( normalDraw2 , alpha = 0.5 , label='random draw' , density = True) # don't forget density=True
plt.plot(x,normPDF , label='PDF' )
plt.legend()
plt.show()
```


We can also get the expected quantiles of a distribution:


```python
print( '95% quantile of a Chi-square distribution with 3 degrees of freedom:', stats.chi2.ppf(0.95 , df=3))
print( 'fraction of a Chi-square distribution with 3 degrees of freedom above of equal to 5' ,  
      1 - stats.chi2.cdf( 5 , df=3 ) )
```

    95% quantile of a Chi-square distribution with 3 degrees of freedom: 7.814727903251179
    fraction of a Chi-square distribution with 3 degrees of freedom above of equal to 5 0.17179714429673354


And you can apply some classical statistical tests:


```python
# t-test of independance between two random samples:
rd.seed(73)

s1 = rd.normal(size=67)
s2 = rd.normal(size=54 , loc = 0.2)

testStat , pval = stats.ttest_ind(s1,s2 , equal_var=True)  # equal variance : Student's t-test ; unequal : Welch's
#almost all of these stat functions return the same test-statistic , pvalue tuple

print('result of the t-test')
print('\tt:',testStat)
print('\tp-value:',pval)
```

    result of the t-test
        t: 0.26673986193074073
        p-value: 0.7901311339594405


### What is our conclusion for these tests results? What do you think about this?


```python

# Kolmogorov-smirnov test for a chi-square distribution

sample = rd.chisquare(df=13 , size = 43)


# kstest expect as second argument the cdf function of the reference distribution
# this is how to handle the fact that me must set an argument (degree of freedom)
refDistribution = stats.chi2(df=13).cdf

testStat , pval = stats.kstest( sample , refDistribution )
# alternative : 
# testStat , pval = stats.kstest( sample , lambda x : stats.chi2.cdf(x , df=13 ) )

print('result of the Kolmogorov-Smirnov test comparing our sample to a Chi-square distribution with 13 degrees of freedom')
print('\tK:',testStat)
print('\tp-value:',pval)

```

    result of the Kolmogorov-Smirnov test comparing our sample to a Chi-square distribution with 13 degrees of freedom
        K: 0.12249766392962913
        p-value: 0.5003109000967569


If you are interested, this [webpage](https://machinelearningmastery.com/statistical-hypothesis-tests-in-python-cheat-sheet/) references all implemented tests, with examples.

[back to the top](#top)

## Bringing together numpy, numpy.random, and matplotlib

The random generation function return a numpy array, meaning it is fairly trivial to combine it with other arrays:



```python
# combining 

x = np.sort( rd.normal(loc=170 , scale = 23 , size = 100) )

y_theoretical = 0.75 * x + 100 # simple linear relationship : y = a * x + b

measurement_noise = rd.normal(scale = 10 , size = 100) # some noise associated to the measure

y_observed = y_theoretical + measurement_noise # observed = expected + noise

fig,ax = plt.subplots(figsize=(8,8))
plt.plot( x , y_theoretical , label = 'expected' )
plt.plot( x , y_observed , marker = '.' , linestyle='' , alpha = 0.7 , label = 'observed')
plt.legend()
plt.show()
```



## The briefest intro to pandas

`pandas` is a powerful library when doing data analysis, especially in the forms of table.

Basically, it reimplements R data.frame as a DataFrame object and ties together neatly with the libraries we've just seen.



```python
 import pandas as pd
    
df = pd.read_table( 'data/beetle.csv' , sep=',' , index_col=0 ) # pandas automatically detects header.

df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dose</th>
      <th>nexp</th>
      <th>ndied</th>
      <th>prop</th>
      <th>nalive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>49.1</td>
      <td>59</td>
      <td>6</td>
      <td>0.102</td>
      <td>53</td>
    </tr>
    <tr>
      <th>2</th>
      <td>53.0</td>
      <td>60</td>
      <td>13</td>
      <td>0.217</td>
      <td>47</td>
    </tr>
    <tr>
      <th>3</th>
      <td>56.9</td>
      <td>62</td>
      <td>18</td>
      <td>0.290</td>
      <td>44</td>
    </tr>
    <tr>
      <th>4</th>
      <td>60.8</td>
      <td>56</td>
      <td>28</td>
      <td>0.500</td>
      <td>28</td>
    </tr>
    <tr>
      <th>5</th>
      <td>64.8</td>
      <td>63</td>
      <td>52</td>
      <td>0.825</td>
      <td>11</td>
    </tr>
  </tbody>
</table>
</div>




```python
Nrows, Ncols = df.shape
print( 'number of rows:',Nrows, 'number of columns:', Ncols )
print( 'column names' , df.columns )
```

    number of rows: 8 number of columns: 5
    column names Index(['dose', 'nexp', 'ndied', 'prop', 'nalive'], dtype='object')



```python
df.describe()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dose</th>
      <th>nexp</th>
      <th>ndied</th>
      <th>prop</th>
      <th>nalive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>8.000000</td>
      <td>8.000000</td>
      <td>8.000000</td>
      <td>8.000000</td>
      <td>8.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>62.800000</td>
      <td>60.125000</td>
      <td>36.375000</td>
      <td>0.602000</td>
      <td>23.750000</td>
    </tr>
    <tr>
      <th>std</th>
      <td>9.599702</td>
      <td>2.232071</td>
      <td>22.557466</td>
      <td>0.367937</td>
      <td>21.985385</td>
    </tr>
    <tr>
      <th>min</th>
      <td>49.100000</td>
      <td>56.000000</td>
      <td>6.000000</td>
      <td>0.102000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>55.925000</td>
      <td>59.000000</td>
      <td>16.750000</td>
      <td>0.271750</td>
      <td>4.750000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>62.800000</td>
      <td>60.000000</td>
      <td>40.000000</td>
      <td>0.662500</td>
      <td>19.500000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>69.675000</td>
      <td>62.000000</td>
      <td>54.750000</td>
      <td>0.919500</td>
      <td>44.750000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>76.500000</td>
      <td>63.000000</td>
      <td>61.000000</td>
      <td>1.000000</td>
      <td>53.000000</td>
    </tr>
  </tbody>
</table>
</div>




```python
# select a single column:
df['dose']
```




    1    49.1
    2    53.0
    3    56.9
    4    60.8
    5    64.8
    6    68.7
    7    72.6
    8    76.5
    Name: dose, dtype: float64




```python
df[ ['ndied','nalive'] ] # select several columns
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>ndied</th>
      <th>nalive</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>1</th>
      <td>6</td>
      <td>53</td>
    </tr>
    <tr>
      <th>2</th>
      <td>13</td>
      <td>47</td>
    </tr>
    <tr>
      <th>3</th>
      <td>18</td>
      <td>44</td>
    </tr>
    <tr>
      <th>4</th>
      <td>28</td>
      <td>28</td>
    </tr>
    <tr>
      <th>5</th>
      <td>52</td>
      <td>11</td>
    </tr>
    <tr>
      <th>6</th>
      <td>53</td>
      <td>6</td>
    </tr>
    <tr>
      <th>7</th>
      <td>61</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td>60</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>



### Plotting DataFrame Columns

Because `DataFrame` columns are iterable, they can seamlessly be given as argument to `plot()`.


```python

# plotting the column dose along the x-axis and prop along the y-axis
# I use the + marker, with a teal color.
plt.plot(df['dose'] , df['prop'] , color = 'teal' , linestyle='' , marker = '+' , markersize=10 )
plt.xlabel( 'dose' )
plt.ylabel( 'proportion of dead' )
plt.show()
```
    


DataFrame column can be manipulated like numpy array:


```python

## we can combine columns using normal operators
Odds = df['nalive'] /df['ndied'] # the odds of being alive is nalive / ndead

## adding a new column to the DataFrame is trivial:
df['Odds'] = Odds


## we can also apply numpy function to them
df['logOdds'] = np.log( df['Odds'] )


plt.plot(df['dose'] , df['logOdds'] , color = 'teal' , linestyle='' , marker = '+' , markersize=10 )
plt.xlabel( 'dose' )
plt.ylabel( 'log Odds' )
plt.show()

```



## Exercise 00.02 : tying everything together

1. Read the file `'data/kyphosis.csv'`.
2. how many columns are there ?
3. What is the maximum Age ? 
4. create a new column `Stop` , corresponding to the addition of columns `'Start'` and `'Number'`
5. plot the relationship between `'Age'` and `'Number'` (bonus point : use colors to indicate the presence or absence of kyphosis ).





```python

```

Solutions:


```python
# %load  -r -7 solutions/solution_00_02.py
```


```python
# %load  -r 8-9 solutions/solution_00_02.py
```


```python
# %load  -r 11-12 solutions/solution_00_02.py
```


```python
# %load  -r 14-15 solutions/solution_00_02.py
```


```python
# %load  -r 17-22 solutions/solution_00_02.py
```


```python
# %load  -r 24- solutions/solution_00_02.py
```


topic: data-science
tutorial: data-manipulation-olympics
title: Data Manipulation Olympics
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa"]
content_hash: b357f52b15e5beb66ff6429e8e18faca
last_modified: 2026-02-26T21:12:17.029292
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->


<!-- set up some variables to easily update tool versions throughout tutorial, since most tools are used many times %} -->
{% assign version_cat="cat1" %}
{% assign version_compute="toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.0" %}
{% assign version_count="Count1" %}
{% assign version_cut_columns="Cut1" %}
{% assign version_cut_advanced="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0" %}
{% assign version_datamash="toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0+galaxy2" %}
{% assign version_filter="Filter1" %}
{% assign version_join="join1" %}

{% assign version_remove_beginning="Remove beginning1" %}
{% assign version_remove_columns_by_header="toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/0.0.1" %}
{% assign version_replace_text_column="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1" %}
{% assign version_replace_text_line="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1" %}
{% assign version_paste="Paste1" %}

{% assign version_select_first="Show beginning1" %}
{% assign version_sort="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1" %}
{% assign version_split="toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.6" %}
{% assign version_unique="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0" %}
{% assign version_wc="wc_gnu" %}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

If you've opened this tutorial via the {% icon level %} icon in Galaxy (top menu bar), you can click on the tool names in the last column to quickly open them in Galaxy and start using them on your own!

<!-- GTN:IGNORE:006 -->

| Operation              | Description                        | Galaxy Tool    |
|------------------------|------------------------------------|----------------|
| Convert format         | Change the file format             | {% icon galaxy-pencil%} Edit attributes |
| Word count             | Count the number of lines, words and characters in a file | {% tool [Line/Word/Character count]({{version_wc}}) %} |
| Sort on a column       | Change the order of the rows based on values in one or more columns | {% tool [Sort]({{version_sort}}) %} |
| Filter                 | Remove rows based on values in one or more columns | {% tool [Filter]({{version_filter}}) %}|
| Counting               | Count occurrences of values in a column   | {% tool [Count]({{version_count}}) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Group on a column      | And perform simple operations (count, mean, min, max etc) | {% tool [Group](Grouping1) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Compute on rows        | to derive new column values from existing ones | {% tool [Compute]({{version_compute}}) %} |
| Find and Replace       | in a specific column               | {% tool [Column Regex Find and Replace]({{version_replace_text_column}}) %}|
| Find and Replace       | on every line                      | {% tool [Regex Find and Replace]({{version_replace_text_line}}) %}|
| Join two Datasets      | side by side on a specified field  | {% tool [Join two Datasets]({{version_join}}) %} |
| Concatenate datasets   | one after the other                | {% tool [Concatenate datasets]({{version_cat}}) %} |
| Remove Beginning       | Good for removing header lines     | {% tool [Remove beginning of a file]({{version_remove_beginning}}) %} |
| Select First lines     | Good for finding top 10s or saving header lines | {% tool [Select first lines]({{version_select_first}}) %} |
| Cut Columns            | By header name                     | {% tool [Remove columns by heading]({{version_remove_columns_by_header}}) %}|
| Cut Columns            | By column number                   | {% tool [Cut columns from a table]({{version_cut_columns}}) %}, {% tool [Advanced Cut columns from a table]({{version_cut_advanced}}) %}|
| Paste                  | Two files side by side             | {% tool [Paste]({{version_paste}}) %} |
| Split file             | Based on values of a column        | {% tool [Split]({{version_split}}) %} |
| Unique                 | Remove duplicate rows              | {% tool [Unique]({{version_unique}}) %} |

**TIP: Adding tools to your Favourites:** If you find yourself frequently using the same tool often but struggle to find it back in the long list of tools, you can **star** your favourite tools in Galaxy!

{% snippet faqs/galaxy/tools_favorite.md %}


In this tutorial, these tools are explained in more detail, and we provide some exercises for you to practice.


# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"](./images/cover.jpg)


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!


# Upload Data

Before we can do any manipulation, we will need some data. Let's upload our table with Olympics results now.

> <hands-on-title>Get data</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `olympics.tsv` via link
>
>    ```
>    {{page.zenodo_link}}/files/olympics.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. **Expand** on the item in your history to see some metadata and a short preview of the contents.
>
> 3. **View** {% icon galaxy-eye %} the dataset by clicking on the eye icon.
>
>    > <question-title></question-title>
>    >
>    > 1. What is the format of the file?
>    > 2. What does each row represent?
>    > 3. How many lines are in the file? (Hint: {% tool [Line/Word/Character count](wc_gnu) %})
>    > 4. How many columns?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. When you expand the `olympics.tsv` dataset in your history (see also screenshot below), you will see `format: tabular`, this is another term for a tab-separated (`tsv`) file.
>    > > 2. Each row represents an athlete's participation in an event. If an athlete competes in multiple events, there is a line for each event.
>    > > 3. 234,523. Look at the expanded view in your history, this tells us there are ~250,000 rows in the dataset. We can get the exact number using the  {% tool [Line/Word/Character count](wc_gnu) %} tool.
>    > > 4. There are 17 columns in this file. There are multiple ways to find this answer:
>    > >    - Count the columns manually (only doable for small files)
>    > >    - In the expanded view, scroll sideways on the dataset preview, at the top the columns are numbered
>    > >    - Click on the {% icon galaxy-info %} i icon on the dataset; here you will find more detailed information about the file and the job that created it.
>    > >      At the bottom is also a preview (peek) of the dataset and numbered columns
>    > >
>    > > ![a screenshot of the expanded view of the dataset in the history, it shows the datatype, number of lines in the file, and a preview
>    > > of the dataset with numbered columns](./images/columns-number.png)
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.tsv` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.tsv` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.




# File Format Conversion

The file we uploaded is a `.tsv` file. This stands for *tab-separated values*. This means that this is a file containing rows and columns, where a TAB character is used to signify a column ends and a new one begins. Galaxy is great at understanding tab-separated files, and most of the data manipulation tools are designed to work with such files.

A similar format you may come across a lot in data science, is the `.csv` file, or *comma-separated values* file. This is the same as `.tsv`, but uses comma (`,`) characters to indicate new columns, instead of TAB (`\t`) characters.

Galaxy can convert these two formats into each other.



> <hands-on-title>Convert Tabular to CSV</hands-on-title>
>
> 1. Convert the `olympics.tsv` file into a CSV file
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="csv (using 'Convert tabular to CSV')" %}
>
> 2. {% icon galaxy-eye %} **View the converted dataset**
>
>    > <question-title></question-title>
>    >
>    > 1. What do you notice?
>    > 2. Why are some values in quotes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The display in the center panel should be indistinguishable from before the conversion; however, if you look at the "quick view" of the data in the history step, you will see the commas separating columns.
>    > > 2. If the data in a column contains a comma (e.g. in this file we have events such as `swimming 5,000 meters`), we put the value in quotes to signify that that comma is part of the data, not a column delimiter.
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


## Converting vs Changing the datatype

The **file format conversion** step changed the dataset itself (by changing TABs to commas), and therefore created another dataset in your history. It is also possible to **change the datatype**; this does not change the underlying file/data, but just tells Galaxy what type of file it is. Galaxy uses this information in a variety of ways, such as:
  - How to display the data when you click the {% icon galaxy-eye %} **View** button
  - Which visualization options to offer
  - Filtering the list of possible inputs for tools (to only allow those of the correct datatype and protect you from mistakes)
  - Which file format conversions to make available
  - etc..


When you upload a file to Galaxy, by default it will attempt to auto-detect the file format. This is very often correct, but some times you may need to manually set the datatype to the correct value.


> <hands-on-title>Change datatype from Tabular to TXT</hands-on-title>
>
> 1. **Change** the datatype of the `olympics.tsv` file to `txt`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="txt" %}
>
> 2. {% icon galaxy-eye %} **View the dataset** again
>
>    > <question-title></question-title>
>    >
>    > 1. What do you see?
>    > 2. Why didn't this step create a new item in your history?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Since Galaxy now no longer know that this is a tabular file with rows and columns,
>    > > it displays the data as-is, no longer arranging the view into neat columns.
>    > >
>    > > 2. The file itself did not change, only the metadata. We simply told Galaxy to treat this file as a plain text file, instead of a tabular file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Change** the file back from `txt` to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}



# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the year?
>    > 2. Do we want ascending or descending order if we want the oldest games at the top?
>    > 3. What should we do with the very first row (the one with the header names?)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 12
>    > > 2. The file should be sorted in ascending (increasing) order
>    > > 3. The header line should always stay on top, so we want to ignore it when we sort the file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} tool
>    - Read the help text at the bottom of the tool form
>    - Which settings do you think we need to use?
>
> 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
>    - {% icon param-text %} *"Number of header lines"*: `1``
>    - Column selections
>      - {% icon param-select %} *"on column"*: `Column 12`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
>
> 4. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top of the file now?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful (e.g. `Olympics in chronological order`)
>    - It can be easy to lose track of what steps you have performed and what each output contains, especially if you are doing many
>      data manipulation steps. Do future-you a favour and give datasets good names!
>
{: .hands_on}

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, and then within each year, we sort again alphabetically by name. The sort tool can do this!


> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains athlete names?
>    > 2. Do we want ascending or descending order if we want to sort alphabetically with A first?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 2
>    > > 2. The file should be sorted in ascending (increasing) order
>    > >
>    > {: .solution}
>    {: .question}
>
>
> 2. {% icon galaxy-refresh %} **Rerun** the sort tool with the following parameters:
>    - All parameter from the first step should already be set for you, and should remain the same
>    - {% icon param-repeat %} Insert Column selections
>      - {% icon param-select %} *"on column"*: `Column 2`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Alphabetical sort`
>
> 3. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful
>
{: .hands_on}

## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> > <br>
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Height is listed in column 7. Since this is a simple number (no scientific notation), we can use *fast numerical sort*,
> >    and because we want the tallest on top, we will need to sort in *descending* (decreasing) order.
> > 2. Rerun the tool for step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Here we don't get a clear answer. That is because not all athletes have height data available, and blank fields (`NA`) are being sorted to the top. We can filter out rows with missing data to get our answer (see [Filter](#filtering) section to learn how to do this). For now we cannot answer this question with just the sort tool. Some times multiple tools are required to perform such tasks. The [exercise section at the end of this tutorial](#exercises-putting-it-all-together) has many exercises that require a combination of different tools.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> > 1. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Descending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 2. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Ascending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 12` (year column)
> >     - {% icon param-select %} *"in"*: `Descending order` (most recent on top)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >   - {% icon param-repeat %} Insert Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7` (height column)
> >     - {% icon param-select %} *"in"*: `Descending order` (tallest first)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).

## Filtering based on column contents

> <hands-on-title>Filter table based on a column</hands-on-title>
>
> We will filter the file to show only winter Olympics
>
> 1. Look at the `olympics.tsv` file and answer the following questions
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains this information?
>    > 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 13, the column with the *season* header
>    > > 2. The values can be `Summer` or `Winter`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the tool {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %}
>    - read the help text at the bottom of the tool carefully
>    - get familiar with how to indicate columns and write expressions for filtering
>
>    > <question-title></question-title>
>    >
>    > 1. How would you write the expressions for the following conditions:
>    >    1. column 6 must be 'Yes'
>    >    2. column 13 must be smaller than 75
>    >    3. column 7 cannot be 'NA'
>    >    4. column 2 cannot be empty
>    >
>    > 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    >    How would you write expressions for the following filtering conditions:
>    >    1. column 5 is larger than 2 or smaller than -2
>    >    2. column 5 is larger than 2 and smaller than 10
>    >    3. the sum of columns 4 and 6 is greater than or equal to 25,000
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The answers are:
>    > >    1. `c6=='Yes'`
>    > >    2. `c13<75`
>    > >    3. `c7!='NA'`
>    > >    4. `c2!=''`
>    > >
>    > > 2. The answers are:
>    > >    1. `c5>2 or c5<-2`
>    > >    2. `c5>2 and c5<10`
>    > >    3. `c4+c6 >= 25000`
>    > >
>    > {: .solution}
>    {: .question}
>
>    Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics:
>
> 3. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c13=='Winter'`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 4. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > How many lines are in this file? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 44,681 (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Repeat** the step for the Summer Olympics
>
>    > <question-title></question-title>
>    >
>    > 1. How many lines do you expect in this file?
>    > 2. How many lines are in this file? Were you right?
>    >
>    > > <solution-title noprefix>Hints</solution-title>
>    > >
>    > > 1. Use the {% tool [Line/Word/Character count]({{version_wc}}) %} to find the number of lines in the `olympics.tsv` file and subtract the number of rows in the Winter Olympics file
>    > > 2. Be careful to consider whether these counts include the header line of the file or not
>    > >
>    > {: .solution}
>    >
>    > > <solution-title noprefix>Answers</solution-title>
>    > >
>    > > 1. The original file has 234,523 lines, and the Winter Olympics had 44,681 lines. So we would expect 234,523 - 44,681 = 189,842 rows of data. Since we have subtracted the header line in this equation as well, we expect the Summer Olympics file to have 1 more line that this, so 189,843 total lines.
>    > > 2. 189,843. If you were off by one or two lines, it may have been that you counted the header lines double.
>    > > <br>
>    > > It is always useful to take a moment to think about the expected outcome; this makes it easier to spot mistakes and will save you time in the long run.
>    > >
>    > {: .solution}
>    {: .question}
>
> 6. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive (e.g. `Filter: Winter Olympics only` etc)
>    - This is primarily to get you into the habit. If this is a file you might want to view or use again later, it will be hard to find back again unless you name it well.
>
{: .hands_on}

## Advanced Filtering with Python Syntax

In addition to basic filtering, the Galaxy filtering tool allows for more advanced filtering using Python syntax. This can be particularly useful when you need to apply more complex conditions or manipulate the data in specific ways.

> <comment-title> Understanding Python Syntax for Filtering </comment-title>
>
>    Before we dive into specific examples, let's briefly discuss some basic Python concepts that will help you understand how to write these filtering expressions:
>
> - **Methods**: In Python, methods are functions that are associated with an object. For example, in the expression `c2.startswith('Liu')`, `startswith` is a method that checks if the value in column 2 (`c2`) starts with the substring 'Liu'.
>
> - **Boolean Expressions**: These are expressions that evaluate to either `True` or `False`. For example, `c2.startswith('Liu')` will return `True` if the value in column 2 starts with 'Liu', and `False` otherwise.
>
> - **Comparison Operators**: These include `==` (equal to), `!=` (not equal to), `<` (less than), `>` (greater than), `<=` (less than or equal to), and `>=` (greater than or equal to). These operators are used to compare values and are commonly used in filtering.
>
> - **String Methods**: Strings in Python have several built-in methods that can be used for filtering. Some commonly used ones include:
>   - `startswith(substring)`: Checks if a string starts with a specified substring.
>   - `endswith(substring)`: Checks if a string ends with a specified substring.
>   - `find(substring)`: Returns the lowest index of the substring if it is found in the string; otherwise, it returns `-1`.
>   - `count(substring)`: Returns the number of non-overlapping occurrences of the substring in the string.
>
{: .comment}

These concepts will be used in the filtering expressions we write in Galaxy. Let's now explore some specific scenarios where you might use these expressions.


### Filtering Based on Substring Occurrence

We want to filter rows where a particular column contains a specific substring using the `cX.find()` method, where `X` is the column number.

For example, while you can easily filter Summer or Winter Olympics using `c13=="Summer"` or `c13=="Winter"` if the data is stored in a dedicated column, sometimes the data may not be that straightforward. The relevant information might be embedded in another column, like in `c11` with entries such as "1992 Summer Olympics".

In such cases, you can use substring filtering:

> <question-title></question-title>
>
> 1. How would you filter the rows where the 11th column contains "Summer"?
> 2. How would you filter the rows where the 11th column does **not** contain "Summer"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c11.find('Summer') != -1`
> > 2. `c11.find('Summer') == -1`
> >
> {: .solution}
{: .question}

> <hands-on-title>Filtering Based on Substring Occurrence</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file for rows where column 11 contains "Summer":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c11.find('Summer') != -1`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string "Summer" in column 11? (Hint: use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `189843`(this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on }


### Filtering Rows that Start or End with a Specific Value

Filter rows based on whether a column starts or ends with a particular value using `cX.startswith()` or `cX.endswith()`.

For example, if we want to:
 - Find all athletes whose names start with "Liu".
 - Find all teams whose names end with "China".


> <question-title></question-title>
>
> 1. How would you filter rows where the 2nd column starts with "Liu"?
> 2. How would you filter rows where the 9th column ends with "China"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c2.startswith('Liu')`
> > 2. `c9.endswith('China')`
> >
> {: .solution}
{: .question}


> <hands-on-title>Filtering Rows that Start or End with a Specific Value</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file to find all athletes whose names start with "Liu":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c2.startswith('Liu')`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string start with "Liu" in column 2? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `324` (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on}


### Filtering Based on the Count of Substring Occurrences

Sometimes, you may need to filter rows based on **how many times** a certain substring appears in a column. This can be done using the `cX.count()` method.

For example, if we want to find athletes with three-part names by counting spaces in the name column.

> <hands-on-title>Filtering Based on the Count of Substring Occurrences</hands-on-title>
>
> We want to filter the dataset to find athletes with three-part names.
>
> 1. Have a look at the `olympics.tsv` file
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the names of athletes?
>    > 2. For athletes with 3-part names, how many spaces do we expect in that column?
>    > 3. What expression would you use to find these cases?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Column 2 contains the names
>    > > 2. For 3-part names we expect 2 space characters
>    > > 3. `c2.count(' ') == 2`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>     - {% icon param-file %} *"Filter"*: `olympics.tsv`
>     - {% icon param-text %} *"With the following condition"*: `c2.count(' ') == 2`
>     - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 3. {% icon galaxy-eye %} **View** the filtered file.
>
>     > <question-title></question-title>
>     >
>     > 1. How many rows contain three-part names (i.e., two spaces) in column 2?
>     >
>     > > <solution-title>Answers</solution-title>
>     > >
>     > > 1. `11259` (this is including the header line)
>     > >
>     > {: .solution}
>     {: .question}
>
> 4. **Exercises.** Now let's take it a step further and filter athletes with even longer names.
>
>    > <question-title></question-title>
>    >
>    > 1. How would you filter rows where the 2nd column contains **three spaces** (i.e., four-part names)?
>    > 2. How many medals were won by athletes with five-part names **or more**?
>    > 3. How many parts does the longest name have? Give an example of such a name.
>    > 4. Bonus: **How many** athletes have 6-part names won a medal?
>    >
>    > > <solution-title>Hints</solution-title>
>    > >
>    > > 1. None ;)
>    > > 2. You can also use greater than or less than symbols in your expression. You can filter on two lines using  logical operators like `and`.
>    > > 3. Keep increasing the count until no results are returned
>    > > 4. The number of lines returned indicates the number of medals won by long-named athletes, to
>    > >    find out how many different athletes have such a name, we can use the {% tool [Count - occurrences of each record](Count1) %}
>    > >    tool on Column 2.
>    > {: .solution}
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `c2.count(' ') == 3`
>    > > 2. `c17 != 'NA' and c2.count(' ') >= 4` returns 27 lines, meaning 26 medals won accounting for the header line excluding the header line. Appart from the name you need to additionally filter out athletes that did not won a medal.
>    > > 3. `c2.count(' ') == 5` is the maximum value to still return results.
>    > >    Five spaces means a 6-part name. For example Patricia Galvin de la Tour d'Auvergne
>    > > 4. `c17 != 'NA' and c2.count(' ') == 5` returns 5 records. So 5 medals were won by these long-named athletes.
>    > >     Using the {% tool [Count - occurrences of each record](Count1) %} to count occurrences in column 2, we
>    > >     discover that there are **3 different athletes with 6-part names**.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Exercises

Ok, time to train! let's see if you can use the {% tool [Filter]({{version_filter}}) %} tool to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm?
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Expand the output or use the tool {% tool [Line/Word/Character count]({{version_wc}}) %} to see the number of lines in the file
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> >
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `c17=='Gold'`)
> >  2. 24,633  (Expression: `c17=='Gold' or c17=='Silver' or c17=='Bronze'`, or `c17!='NA'`)
> >  3. 131     (Expression: `c17=='Gold' and c12==2018` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `c17!='NA' and c7>=170 and c7<=180`)
> >  5. 812     (Expression: `c17=='Gold' and (c7<160 or c7>190)` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count. The tool {% tool [**Count** occurrences of each record]({{version_count}}) %} does exactly this.


> <hands-on-title>Count occurrences of values in columns</hands-on-title>
>
> Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)
>
> 1. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11`
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How many different Olympic games are in our file?
>    > 2. Which Olympic games had the most participations? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 52; there are 53 lines in the resulting file, with one line containing the value of the column header (`games`).
>    > >
>    > >    The resulting file looks something like:
>    > >
>    > >     ```
>    > >     615	1896 Summer Olympics
>    > >     2503	1900 Summer Olympics
>    > >     2643	1904 Summer Olympics
>    > >     3213	1908 Summer Olympics
>    > >     4610	1912 Summer Olympics
>    > >     3448	1920 Summer Olympics
>    > >     5242	1924 Summer Olympics
>    > >     358	1924 Winter Olympics
>    > >     4493	1928 Summer Olympics
>    > >     ...
>    > >     ```
>    > >
>    > > 2. 1996 Summer Olympics. (10501 participations)
>    > >
>    > {: .solution}
>    {: .question}
>
>    You may have noticed that we could have selected multiple columns in this tool. This lets us count on combinations of columns.
>    Let's try counting the number of men and women in each olympic games.
>
> 3. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11, Column 3`
>
> 4. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>    >
>    > 1. How many women were in the first Olympic games?
>    >
>    > 2. Which Olympic games had the most women participants? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice).
>    > >    The file looks something like this:
>    > >    ```
>    > >    2	F	1896 Summer Olympics
>    > >    43	F	1900 Summer Olympics
>    > >    17	F	1904 Summer Olympics
>    > >    55	F	1908 Summer Olympics
>    > >    97	F	1912 Summer Olympics
>    > >    132	F	1920 Summer Olympics
>    > >    269	F	1924 Summer Olympics
>    > >    ```
>    > >
>    > > 2. 2020 Summer Olympics (4652)
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting tool above, we would get a results file for each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

To answer these types of questions we can use a slightly more advanced tool, called {% tool [Datamash]({{version_datamash}}) %}. This tool can do a lot more than counting, but here we will show you how to use it to count in this way.


> <hands-on-title>Counting number of different sports per Olympic Games</hands-on-title>
>
> We will now determine how many different sport there were in each of the different Olympics
>
> 1. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
>    - {% icon param-toggle %} *"Input file has a header line"*: `Yes`
>    - {% icon param-toggle %} *"Sort Input"*: `Yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Count Unique values`
>      - {% icon param-select %} *"On Column"*: `Column: 15` (the `sport` column)
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. Why did we tell the tool to sort the file?
>    > 2. How many sport were in the first Olympics? How many in the latest?
>    > 3. Which Olympics had the most different sports?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. This is for technical reasons; the tool assumes that your file is sorted (on the column that you supplied to the "Group by fields" parameter).
>    > >    This assumption can greatly speed up the calculation, but will give incorrect results if the file was not actually sorted.
>    > >    If you know for sure that your data is sorted in the right way, you can skip this step. If you are unsure, enable this parameter to be safe, and the
>    > >    tool will perform the sorting step before its calculation.
>    > > 2. 10 and 38.
>    > > 3. The 2020 Summer Olympics had the most different sports (38)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} the output to something descriptive
>
{: .hands_on}


Note: Datamash can do a lot more than counting,  and we will showcase some of these other operatins in the [Grouping](#grouping) section.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting participations (rows), we can use the simple {% tool [Count]({{version_count}})%} tool
> > 2. Since we are counting a bit more complex question, we need the {% tool [Datamash]({{version_datamash}}) %} tool
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations
> >  2. 15 and 250.
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> >  1. {% tool [Count]({{version_count}}) %} with the following parameters:
> >     - {% icon param-file %} *"from dataset"*: `olympics.tsv`
> >     - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 9` (the `team` column)
> >     - {% icon param-select %} *"How should the results be sorted?"*: `With the most common values first`
> >
> >     This gives an output like:
> >     ```
> >     17286	United States
> >     11700	France
> >     10230	Great Britain
> >     8898	Italy
> >     7988	Canada
> >     ```
> >
> >  2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
> >   - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
> >   - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
> >   - {% icon param-toggle %} *"Input file has a header line"*: `yes`
> >   - {% icon param-toggle %} *"Sort Input"*: `yes`
> >   - *"Operation to perform on each group"*:
> >       - {% icon param-select %} *"Type"*: `Count Unique values`
> >       - {% icon param-select %} *"On Column"*: `Column: 9` (the `team` column)
> >
> >   This gives an output like:
> >   ```
> >   1896 Summer Olympics	15
> >   1900 Summer Olympics	29
> >   1904 Summer Olympics	13
> >   ...
> >   2016 Summer Olympics	280
> >   2018 Winter Olympics	108
> >   2020 Summer Olympics	250
> >   ```
> >
> {: .solution}
{: .question}








# Grouping

Often we may want to group rows based on a value in a column, and perform some operation on the resulting rows. For example we would like to group the olympics data by one value (e.g. year, country, sport), and determine some value for each group (e.g. number of medals won, average age of athletes, etc).

In the [counting](#counting) section of this tutorial we show how to get answers that require a count (e.g. number of medals won), but sometimes we want to do something more complex, like calculating the average height of athletes in a group, say per country or per sport. This section will show some example of these types of questions.

We can use the {% tool [Datamash]({{version_datamash}}) %} tool for this purpose.

> <hands-on-title>Tallest athlete per sport</hands-on-title>
>
> We would like to answer the following question: *How tall was the tallest athlete of each sport?*
>
> 1. Open the {% tool [Datamash]({{version_datamash}}) %} tool and read the help section at the bottom
>
>    > <question-title></question-title>
>    >
>    > Which settings do you think we need to provide to answer our question?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - *"Group by fields"*: We want to group by sport (Column 15).
>    > > - *"Sort"*: `Yes`. This may not be obvious, but because our file is currently not sorted by our chosen group (sport), we need to tell the tool to do this.
>    > > - *"Skip NA or NaN values"*: since we do have NA values for athletes for whom height data is unknown, we should set this to `Yes`
>    > > - Our file has a header line, so we should indicate this as well
>    > > - *"Operation"*: we want to determine the **Maximum** height (Column 7)
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `15`
>    - {% icon param-toggle %} *"Sort input"*: `yes`
>    - {% icon param-toggle %} *"Input file has a header line"*: `yes`
>    - {% icon param-toggle %} *"Skip NA or NaN values"*: `yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Maximum`
>      - {% icon param-select %} *"On Column"*: `Column: 7`
>
> 3. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How tall was the tallest athlete in basketball? And what about karate?
>    > 2. Why do some sports have a value of `inf`?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Basketball's tallest athlete was 192cm. For Karate it is 163.
>    > > 2. Our dataset had quite a number of `NA` (unknown) values in the height column, especially for the earlier Olympics. For sports that had only NA values, there is no maximum so the tool outputs `inf` instead.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Grouping on multiple columns

You may have noticed that we could also provide multiple columns to group on. If we do this, we can compute values for combinations of groups, such as sex and sport, to find e.g. the tallest woman in basketball or the shortest man per Olympics. There are also many more options for the computation we perform, so perhaps we are more interested not in the tallest athlete, but the average height. Let's perform some of these slightly more a


topic: data-science
tutorial: r-advanced
title: Advanced R in Galaxy
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: ["R"]
content_hash: 8e524c51ea0c62122ff69b87d3d50b0c
last_modified: 2026-02-26T21:12:14.906794
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-advanced/tutorial.html

content:
With HTS-Seq data analysis, we generated tables containing list of DE genes, their expression, some statistics, etc. We can manipulate these tables using Galaxy, as we saw in some tutorials, e.g. ["Reference-based RNA-Seq data analysis"]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), and create some visualisations.

Sometimes we want to have some customizations on visualization, some complex table manipulations or some statistical analysis. If we can not find a Galaxy tools for that or the right parameters, we may need to use programming languages as R or Python.

<!-- Keeping this here because people may be familiar enough with R, that they go directly to this tutorial, and it is still useful information. -->

{% include topics/data-science/tutorials/r-basics/r_introduction.md %}
{% include topics/data-science/tutorials/r-basics/disclaimer.md %}

In this tutorial, we will take the list of DE genes extracted from DESEq2's output that we generated in the ["Reference-based RNA-Seq data analysis" tutorial]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), manipulate it and create some visualizations.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Get familiar with the annotated DE genes table in R

A substantial amount of the data we work with in science is tabular data, i.e. data arranged in rows and columns - also known as spreadsheets.

> <comment-title>Few principles when working with data</comment-title>
> We want to remind you of a few principles before we work with our first set of example data:
>
> 1. **Keep raw data separate from analyzed data**
>
>    This is principle number one, because if you can't tell which files are the original raw data, you risk making some serious mistakes (e.g. drawing conclusion from data which have been manipulated in some unknown way).
>
>    When you work with data in R, you are not changing the original file which you loaded the data from. This is different than (for example) working with a spreadsheet program where changing the value of the cell leaves you one "save"-click away from overwriting the original file. You have to purposely use a writing function (e.g. `write.csv()`) to save data loaded into R. In that case, be sure to save the manipulated data into a new file. More on this later in the lesson.
>
> 2. **Keep spreadsheet data Tidy**
>
>    The simplest principle of *Tidy* data is that we have one row in our spreadsheet for each observation or sample, and one column for every variable that we measure or report on. As simple as this sounds, it’s very easily violated. Most data scientists agree that significant amounts of their time is spent tidying data for analysis.
>
> 3. **Trust but verify**
>
>    Finally, while you don't need to be paranoid about data, you should have a plan for how you will prepare it for analysis. **This a focus of this lesson**. You probably already have a lot of intuition, expectations, assumptions about your data - the range of values you expect, how many values should have been recorded, etc. Of course, as the data get larger our human ability to keep track will start to fail (and yes, it can fail for small data sets too). R will help you to examine your data so that you can have greater confidence in your analysis, and its reproducibility.
>
{: .comment}

## Import tabular data into R

As in the "R Basics" tutorial, we will continue in {% tool [RStudio](interactive_tool_rstudio) %}.

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

There are several ways to import data into R. For our purpose here, we will focus on using the tools every R installation comes with (so called "base" R) to import a comma-delimited file containing the results of our variant calling workflow. We will need to load the sheet using a function called `read.csv()`.

> <question-title>Review the arguments of the `read.csv()` function</question-title>
>
> Before using the `read.csv()` function, use R's help feature to answer the following questions.
>
> > <tip-title>Getting help in R</tip-title>
> > Entering `?` before the function name and then running that line will bring up the help documentation. Also, when reading this particular help be careful to pay attention to the `read.csv` expression under the **Usage** heading. Other answers will be in the **Arguments** heading.
> {: .tip}
>
> 1. What is the default parameter for `header` in the `read.csv()` function?
> 2. What argument would you have to change to read a file that was delimited by semicolons (`;`) rather than commas?
> 3. What argument would you have to change to read file in which numbers used commas for decimal separation (i.e. 1,00)?
> 4. What argument would you have to change to read in only the first 10,000 rows of a very large file?
>
> > <solution-title></solution-title>
> >
> > 1. The `read.csv() `function has the argument `header` set to `TRUE` by default, this means the function always assumes the first row is header information, (i.e. column names)
> > 2. The `read.csv()` function has the argument 'sep' set to ",". This means the function assumes commas are used as delimiters, as you would expect. Changing this parameter (e.g. `sep=";"`) would now interpret semicolons as delimiters.
> > 3. Although it is not listed in the `read.csv()` usage, `read.csv()` is a "version" of the function `read.table()` and accepts all its arguments. If you set `dec=","` you could change the decimal operator. We'd probably assume the delimiter is some other character.
> > 4. You can set `nrow` to a numeric value (e.g. `nrow=10000`) to choose how many rows of a file you read in. This may be useful for very large files where not all the data is needed to test some data cleaning steps you are applying.
> >
> > Hopefully, this exercise gets you thinking about using the provided help documentation in R. There are many arguments that exist, but which we wont have time to cover. Look here to get familiar with functions you use frequently, you may be surprised at what you find they can do.
> >
> {: .solution}
{: .question}

Now, let's read the file with the annotated differentially expressed genes that was produced in the "[Reference-based RNA-Seq data analysis]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %})" tutorial.

<!--
> <hands-on-title>Import dataset into Galaxy</hands-on-title>
> 1. In Galaxy, upload the following file from Zenodo:
>    ```
>    https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular
>    ```
> 2. Remember the dataset number (e.g. `2` in the example below)
>    ![todo](../../images/rna-seq-counts-to-viz-in-r/dataset_number.png)
>
{: .hands_on}
-->

> <hands-on-title>Read the annotated differentially expressed genes</hands-on-title>
> 1. Create a new script (if needed)
> 2. Read the tabular file in an object called `annotatedDEgenes`. We can import directly by URL:
>
>    ```R
>    ## read in a CSV file and save it as 'annotatedDEgenes'
>    annotatedDEgenes <- read.csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular")
>    ```
>
>    The first argument to pass to our `read.csv()` function is the file path for our data. The file path must be in quotes and now is a good time to remember to use tab autocompletion. If you use tab autocompletion you avoid typos and errors in file paths. Use it!
>
> 3. Inspect the **Environment** panel
>
>  > <tip-title>Upload files from a Galaxy history to RStudio</tip-title>
>  > Rstudio in Galaxy provides some special functions to import and export from your history.
>  >
>  > ```R
>  > gx_get(2) # will import dataset number 2 from your history
>  > ```
>  {: .tip}
{: .hands_on}

In the **Environment** panel, you should have the `annotatedDEgenes` object, listed as 130 obs. (observations/rows) of 1 variable (column) - so the command worked (sort of)!

> <hands-on-title>Inspect the annotatedDEgenes file</hands-on-title>
> 1. Double-click on the name of the object in the **Environment** panel
>
{: .hands_on}

It will open a view of the data in a new tab on the top-left panel.

As you can see, there is a problem with how the data has been loaded. The table should contain 130 observations of 13 variables.

> <question-title>Identify the issue with data</question-title>
>
> 1. What is wrong?
> 2. How should you adjust the parameters for `read.csv()` in order to produce the intended output?
>
> > <solution-title></solution-title>
> >
> > 1. The data file was not delimited by commas (`,`) which is the default expected delimiter for `read.csv()`. Instead it seems like it's delimited by "white space", i.e. spaces and/or tabs.
> > 2. We need to set the correct delimiter using the parameter `sep`. Given that we are not sure what the actual delimiter is, we could try both options, i.e. `sep=" "` (*space*) and `sep="\t"` (*tab*).
> >
> {: .solution}
{: .question}

> <hands-on-title>Read correctly tabular file</hands-on-title>
> 1. Read the file given it's a tabular file
>
>    ```R
>    annotatedDEgenes <- read.csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t")
>    ```
>
> 2. Inspect the object in the **Environment** panel
> 3. Open the object
>
>    You should have now a nice table
>
> 3. Check the column names
>
>    ```R
>    > colnames(annotatedDEgenes)
>    [1] "GeneID"     "Base.mean"  "log2.FC."   "StdErr"     "Wald.Stats"
>    [6] "P.value"    "P.adj"      "Chromosome" "Start"      "End"
>    [11] "Strand"     "Feature"    "Gene.name"
>    ```
>
{: .hands_on}

Congratulations! You've successfully loaded your data into RStudio!

## Summarize and determine the structure of a data frame

Tabular data are stored in R using **data frame**. A data frame could also be thought of as a collection of vectors, all of which have the same length.

> <hands-on-title>Learn more about our data frame</hands-on-title>
> 1. Get summary statistics of `annotatedDEgenes` using `summary` function
>
>    ```R
>    ## get summary statistics on a data frame
>    > summary(annotatedDEgenes)
>    GeneID            Base.mean          log2.FC.          StdErr            Wald.Stats
>    FBgn0000071:  1   Min.   :   19.15   Min.   :-4.1485   Min.   :0.08433   Min.   :-30.741
>    FBgn0000079:  1   1st Qu.:  100.29   1st Qu.:-1.3363   1st Qu.:0.12849   1st Qu.:-10.005
>    FBgn0000116:  1   Median :  237.99   Median :-1.0272   Median :0.16370   Median : -4.982
>    FBgn0000406:  1   Mean   : 1911.27   Mean   :-0.2074   Mean   :0.16432   Mean   : -1.901
>    FBgn0000567:  1   3rd Qu.:  948.66   3rd Qu.: 1.2203   3rd Qu.:0.19866   3rd Qu.:  7.692
>    FBgn0001137:  1   Max.   :65114.84   Max.   : 2.6999   Max.   :0.23292   Max.   : 27.566
>    (Other)    :124
>    P.value             P.adj               Seqname    Start              End                Strand
>    Min.   :0.000e+00   Min.   :0.000e+00   chr2L:24   Min.   :  127448   Min.   :  140340   -:58
>    1st Qu.:0.000e+00   1st Qu.:0.000e+00   chr2R:31   1st Qu.: 7277516   1st Qu.: 7279063   +:72
>    Median :0.000e+00   Median :0.000e+00   chr3L:27   Median :13161546   Median :13166253
>    Mean   :4.201e-07   Mean   :9.321e-06   chr3R:32   Mean   :13436843   Mean   :13446444
>    3rd Qu.:7.000e-12   3rd Qu.:3.700e-10   chrX :16   3rd Qu.:19250429   3rd Qu.:19284637
>    Max.   :9.418e-06   Max.   :1.951e-04              Max.   :31196915   Max.   :31203722
>
>    Feature              Gene.Name
>    lincRNA       :  3   Ama        :  1
>    protein_coding:126   Amy-p      :  1
>    pseudogene    :  1   Ant2       :  1
>                         Argk       :  1
>                         BM-40-SPARC:  1
>                         bou        :  1
>                         (Other)    :124
>    ```
>
> 2. Get the structure of `annotatedDEgenes` using `str` function
>
>    ```R
>    ## get the structure of a data frame
>    > str(annotatedDEgenes)
>    'data.frame':	130 obs. of  13 variables:
>     $ GeneID    : Factor w/ 130 levels "FBgn0000071",..: 87 12 28 26 31 96 65 62 125 1 ...
>     $ Base.mean : num  1087 6410 65115 2192 5430 ...
>     $ log2.FC.  : num  -4.15 -3 -2.38 2.7 -2.11 ...
>     $ StdErr    : num  0.1349 0.1043 0.0843 0.0979 0.0925 ...
>     $ Wald.Stats: num  -30.7 -28.7 -28.2 27.6 -22.7 ...
>     $ P.value   : num  1.62e-207 9.42e-182 2.85e-175 2.85e-167 1.57e-114 ...
>     $ P.adj     : num  1.39e-203 4.04e-178 8.15e-172 6.10e-164 2.70e-111 ...
>     $ Seqname   : Factor w/ 5 levels "chr2L","chr2R",..: 4 5 4 5 3 4 2 2 3 4 ...
>     $ Start     : int  24141394 10780892 26869237 10778953 13846053 31196915 24945138 22550093 820758 >    6762592 ...
>     $ End       : int  24147490 10786958 26871995 10786907 13860001 31203722 24946636 22552113 821512 >    6765261 ...
>     $ Strand    : Factor w/ 2 levels "-","+": 2 1 1 1 2 2 2 2 2 2 ...
>     $ Feature   : Factor w/ 3 levels "lincRNA","protein_coding",..: 2 2 2 2 2 2 2 2 1 2 ...
>     $ Gene.Name : Factor w/ 130 levels "Ama","Amy-p",..: 88 113 5 3 84 26 41 53 69 1 ...
>    ```
{: .hands_on}

Ok, thats a lot to unpack! Some things to notice:

- the object type `data.frame` is displayed in the first row along with its dimensions, in this case 130 observations (rows) and 13 variables (columns)
- Each variable (column) has a name (e.g. `GeneID`). This is followed by the object mode (e.g. `factor`, `int`, `num`, etc.). Notice that before each variable name there is a `$` - this will be important later.

So from both `summary` and `str`, we know that our data frame had 13 variables that summarize the data.

- `Base.mean`, `log2.FC.` and `P.value` variables (and several others) are numerical data

    We get summary statistics on the min and max values for these columns, as well as mean, median, and interquartile ranges

- Many of the other variables (e.g. `Strand`) are treated as categorical data

    This type has special treatment in R - more on this in a bit. The most frequent 6 different categories and the number of times they appear (e.g. the `Strand` called `-` appeared 58 times) are displayed. Another example is the `protein_coding` value for `Feature` which appeared in 126 observations.

## Factors

Factors, the final major data structure we will introduce, can be thought of as vectors which are specialized for categorical data. Given R's specialization for statistics, this make sense since categorial and continuous variables usually have different treatments. Sometimes you may want to have data treated as a factor, but in other cases, this may be undesirable.

Since some of the data in our data frame are factors, lets see how factors work.

> <hands-on-title>Inspect a factor object</hands-on-title>
> 1. Extract the `feature` column to a new object called `feature`
>
>    ```R
>    ## extract the "feature" column to a new object
>    feature <- annotatedDEgenes$Feature
>    ```
>
> 2. Inspect the first few items in our  using `head` function
>
>    ```R
>    > head(feature)
>    [1] protein_coding protein_coding protein_coding protein_coding
>    [5] protein_coding protein_coding
>    Levels: lincRNA protein_coding pseudogene
>    ```
{: .hands_on}

What we get back are the items in our factor, and also something called "Levels". **Levels are the different categories contained in a factor**. By default, R will organize the levels in a factor in alphabetical order. So the first level in this factor is "lincRNA".

> <hands-on-title>Get the structure of a factor object</hands-on-title>
> 1. Get the structure of the `feature` object
>
>    ```R
>    > str(feature)
>    Factor w/ 3 levels "lincRNA","protein_coding",..: 2 2 2 2 2 2 2 2 1 2 ...
>    ```
{: .hands_on}

For the sake of efficiency, R stores the content of a factor as a vector of integers, which an integer is assigned to each of the possible levels. Recall levels are assigned in alphabetical order. In this case, the first item in our "Feature" object is "protein_coding", which happens to be the 2nd level of our factor, ordered alphabeticaly. The 9th item in the list is "lincRNA", which is the 1st level of our factor.

One of the most common uses for factors is to plot categorical values. For example, suppose we want to know how many of our differentially expressed genes correspond to different features.

> <hands-on-title>Plot a factor object</hands-on-title>
> 1. Plot the number of levels in the `feature` object using `plot` function
>
>    ```R
>    plot(feature)
>    ```
{: .hands_on}

![feature plot](../../../introduction/images/rna-seq-counts-to-viz-in-r/featurePlot.png)

This isn't a particularly pretty example of a plot. We'll be learning much more about creating nice, publication-quality graphics later.

## Subset a data frame

Now we would like to know how to get specific values from data frames, and where necessary, change the mode of a column of values.

The first thing to remember is that a data frame is two-dimensional (rows and columns). Therefore, to select a specific value we will once again use `[]` (bracket) notation, but we will specify more than one value (except in some cases where we are taking a range).

> <question-title>Subsetting a data frame</question-title>
>
> Run the following indices and functions to figure out what they return
>
> 1. `annotatedDEgenes[1, 1]`
> 2. `annotatedDEgenes[2, 4]`
> 3. `annotatedDEgenes[130, 13]`
> 4. `annotatedDEgenes[2, ]`
> 5. `annotatedDEgenes[-1, ]`
> 6. `annotatedDEgenes[1:4, 1]`
> 7. `annotatedDEgenes[1:10, c("Feature", "Gene.Name")]`
> 8. `annotatedDEgenes[, c("Gene.Name")]`
> 9. `head(annotatedDEgenes)`
> 10. `tail(annotatedDEgenes)`
> 11. `annotatedDEgenes$GeneID`
> 12. `annotatedDEgenes[annotatedDEgenes$Feature == "pseudogene", ]`
>
> > <solution-title></solution-title>
> >
> > 1. `annotatedDEgenes[1, 1]`
> >
> >     ```R
> >     > annotatedDEgenes[1,1]
> >     [1] FBgn0039155
> >     130 Levels: FBgn0000071 FBgn0000079 FBgn0000116 FBgn0000406 FBgn0000567 FBgn0001137 ... FBgn0267635
> >     ```
> >
> > 2. `annotatedDEgenes[2, 4]`
> >
> >     ```R
> >     > annotatedDEgenes[2, 4]
> >     [1] 0.1043451
> >     ```
> >
> > 3. `annotatedDEgenes[130, 13]`
> >
> >     ```R
> >     > annotatedDEgenes[130,13]
> >     [1] CG10440
> >     130 Levels: Ama Amy-p Ant2 Argk BM-40-SPARC bou bves CAHbeta cbt CG10063 CG10365 CG10440 ... zye
> >     ```
> >
> > 4. `annotatedDEgenes[2, ]`
> >
> >     ```R
> >     > annotatedDEgenes[2, ]
> >            GeneID Base.mean   log2.FC.   StdErr Wald.Stats       P.value         P.adj Seqname    Start
> >     2 FBgn0003360  6409.577 -2.999777 0.1043451  -28.74863 9.422382e-182 4.040788e-178    chrX 10780892
> >            End Strand        Feature Gene.Name
> >     2 10786958      - protein_coding      sesB
> >     ```
> >
> > 5. `annotatedDEgenes[-1, ]`
> >
> >     ```R
> >     > annotatedDEgenes[-1, ]
> >             GeneID   Base.mean  log2.FC.     StdErr Wald.Stats       P.value         P.adj Seqname
> >     2  FBgn0003360  6409.57713 -2.999777 0.10434506 -28.748628 9.422382e-182 4.040788e-178    chrX
> >     3  FBgn0026562 65114.84056 -2.380164 0.08432692 -28.225437 2.850473e-175 8.149503e-172   chr3R
> >     4  FBgn0025111  2192.32237  2.699939 0.09794457  27.565988 2.846764e-167 6.104174e-164    chrX
> >     5  FBgn0029167  5430.06728 -2.105062 0.09254660 -22.745964 1.573283e-114 2.698810e-111   chr3L
> >     6  FBgn0039827   390.90178 -3.503014 0.16002962 -21.889786 3.250384e-106 4.646424e-103   chr3R
> >     7  FBgn0035085   928.26381 -2.414074 0.11518516 -20.958204  1.579343e-97  1.935147e-94   chr2R
> >     8  FBgn0034736   330.38302 -3.018179 0.15815418 -19.083774  3.444661e-81  3.693107e-78   chr2R
> >     9  FBgn0264475   955.45445 -2.334486 0.12423003 -18.791643  8.840041e-79  8.424559e-76   chr3L
> >     10 FBgn0000071   468.05793  2.360017 0.13564397  17.398615  8.452137e-68  7.249398e-65   chr3R
> >     # ... with 120 more rows
> >
> >           Start      End Strand        Feature   Gene.Name
> >     2  10780892 10786958      - protein_coding        sesB
> >     3  26869237 26871995      - protein_coding BM-40-SPARC
> >     4  10778953 10786907      - protein_coding        Ant2
> >     5  13846053 13860001      + protein_coding         Hml
> >     6  31196915 31203722      + protein_coding      CG1544
> >     7  24945138 24946636      + protein_coding      CG3770
> >     8  22550093 22552113      + protein_coding      CG6018
> >     9    820758   821512      +        lincRNA     CR43883
> >     10  6762592  6765261      + protein_coding         Ama
> >     # ... with 120 more rows
> >     ```
> >
> > 6. `annotatedDEgenes[1:4,1]`
> >
> >     ```R
> >     > annotatedDEgenes[1:4,1]
> >     [1] FBgn0039155 FBgn0003360 FBgn0026562 FBgn0025111
> >     130 Levels: FBgn0000071 FBgn0000079 FBgn0000116 FBgn0000406 FBgn0000567 FBgn0001137 ... FBgn0267635
> >     ```
> >
> > 7. annotatedDEgenes[1:10,c("Feature","Gene.Name")]
> >
> >     ```R
> >     > annotatedDEgenes[1:10,c("Feature","Gene.Name")]
> >               Feature   Gene.Name
> >     1  protein_coding        Kal1
> >     2  protein_coding        sesB
> >     3  protein_coding BM-40-SPARC
> >     4  protein_coding        Ant2
> >     5  protein_coding         Hml
> >     6  protein_coding      CG1544
> >     7  protein_coding      CG3770
> >     8  protein_coding      CG6018
> >     9         lincRNA     CR43883
> >     10 protein_coding         Ama
> >     ```
> >
> > 8. `annotatedDEgenes[,c("Gene.Name")]`
> >
> >     ```R
> >     > annotatedDEgenes[,c("Gene.Name")]
> >     [1] Kal1         sesB         BM-40-SPARC  Ant2         Hml          CG1544       CG3770
> >     [8] CG6018       CR43883      Ama          CG3168       CG15695      CG8500       Rgk1
> >     [15] Sesn         CG9119       Sema-2a      l(1)G0196    lectin-28C   Sox100B      Hsp27
> >     [22] CG34330      Ugt58Fa      Src64B       Tequila      CG32407      LpR2         CG12290
> >     [29] ps           CG8157       nemy         bves         CG8468       CR43855      CG13315
> >     [36] CG1124       Reg-2        CG5261       Treh         Hsp23        Gale         PPO1
> >     [43] Picot        RpS24        CG14946      CG10365      NimC4        CG42694      CG33307
> >     [50] CG6006       grk          CG42369      GILT1        CG5001       Hsp26        MtnA
> >     [57] CG6330       l(3)neo38    Cyt-b5-r     CG12116      PPO2         zye          tnc
> >     [64] CG14257      GstE2        olf413       CG31642      Nxf3         CG8292       CG5397
> >     [71] Mctp         CG7777       CG4019       ChLD3        Argk         bou          Mct1
> >     [78] CG9416       CG12512      CG32625      Npc2b        CG30463      Eip74EF      CAHbeta
> >     [85] CG14326      Cyp6w1       CG3457       wdp          cbt          CrebA        Cyp4d20
> >     [92] CR45973      Sr-CIV       SPH93        Cyp9h1       sug          CG14516      Ggamma30A
> >     [99] CG4297       CG9503       CG10063      CG8501       CG43336      stet         CG8852
> >     [106] CG15080      Amy-p        sn           CG9989       mlt          CG2444       CG17189
> >     [113] RhoGAP1A     CG13743      TwdlU        Galphaf      CG32335      E(spl)m3-HLH CG16749
> >     [120] CG34434      CG43799      Pde6         Sr-CIII      NimC2        CG5895       SP1173
> >     [127] CG1311       CG14856      CG6356       CG10440
> >     130 Levels: Ama Amy-p Ant2 Argk BM-40-SPARC bou bves CAHbeta cbt CG10063 CG10365 CG10440 ... zye
> >     ```
> >
> > 9. `head(annotatedDEgenes)`
> >
> >     ```R
> >     > head(annotatedDEgenes)
> >            GeneID  Base.mean  log2.FC.      StdErr Wald.Stats       P.value         P.adj Seqname    Start
> >     1 FBgn0039155  1086.9743 -4.148450 0.13494887  -30.74090 1.617921e-207 1.387691e-203   chr3R 24141394
> >     2 FBgn0003360  6409.5771 -2.999777 0.10434506  -28.74863 9.422382e-182 4.040788e-178    chrX 10780892
> >     3 FBgn0026562 65114.8406 -2.380164 0.08432692  -28.22544 2.850473e-175 8.149503e-172   chr3R 26869237
> >     4 FBgn0025111  2192.3224  2.699939 0.09794457   27.56599 2.846764e-167 6.104174e-164    chrX 10778953
> >     5 FBgn0029167  5430.0673 -2.105062 0.09254660  -22.74596 1.573283e-114 2.698810e-111   chr3L 13846053
> >     6 FBgn0039827   390.9018 -3.503014 0.16002962  -21.88979 3.250384e-106 4.646424e-103   chr3R 31196915
> >
> >     End Strand        Feature   Gene.Name
> >     1 24147490      + protein_coding        Kal1
> >     2 10786958      - protein_coding        sesB
> >     3 26871995      - protein_coding BM-40-SPARC
> >     4 10786907      - protein_coding        Ant2
> >     5 13860001      + protein_coding         Hml
> >     6 31203722      + protein_coding      CG1544
> >     ```
> >
> > 10. `tail(annotatedDEgenes)`
> >
> >      ```R
> >      > tail(annotatedDEgenes)
> >               GeneID Base.mean  log2.FC.    StdErr Wald.Stats      P.value        P.adj Seqname    Start
> >      125 FBgn0036560  27.71424  1.089000 0.2327527   4.678786 2.885785e-06 6.671531e-05   chr3L 16035484
> >      126 FBgn0035710  26.16177  1.048979 0.2329221   4.503559 6.682472e-06 1.436480e-04   chr3L  6689326
> >      127 FBgn0035523  70.99820  1.004819 0.2237625   4.490560 7.103600e-06 1.523189e-04   chr3L  4277961
> >      128 FBgn0038261  44.27058  1.006264 0.2241243   4.489756 7.130482e-06 1.525141e-04   chr3R 14798985
> >      129 FBgn0039178  23.55006  1.040917 0.2326264   4.474631 7.654326e-06 1.629061e-04   chr3R 24283954
> >      130 FBgn0034636  24.77052 -1.028531 0.2321678  -4.430119 9.418101e-06 1.951185e-04   chr2R 21560245
> >
> >      End Strand        Feature Gene.Name
> >      125 16037227      + protein_coding    CG5895
> >      126  6703521      - protein_coding    SP1173
> >      127  4281585      + protein_coding    CG1311
> >      128 14801163      + protein_coding   CG14856
> >      129 24288617      + protein_coding    CG6356
> >      130 21576035      - protein_coding   CG10440
> >      ```
> >
> > 11. `annotatedDEgenes$GeneID`
> >
> >      ```R
> >      > annotatedDEgenes$GeneID
> >      [1] FBgn0039155 FBgn0003360 FBgn0026562 FBgn0025111 FBgn0029167 FBgn0039827 FBgn0035085 FBgn0034736
> >      [9] FBgn0264475 FBgn0000071 FBgn0029896 FBgn0038832 FBgn0037754 FBgn0264753 FBgn0034897 FBgn0035189
> >      [17] FBgn0011260 FBgn0027279 FBgn0040099 FBgn0024288 FBgn0001226 FBgn0085359 FBgn0040091 FBgn0262733
> >      [25] FBgn0023479 FBgn0052407 FBgn0051092 FBgn0039419 FBgn0261552 FBgn0034010 FBgn0261673 FBgn0031150
> >      [33] FBgn0033913 FBgn0264437 FBgn0040827 FBgn0037290 FBgn0016715 FBgn0031912 FBgn0003748 FBgn0001224
> >      [41] FBgn0035147 FBgn0261362 FBgn0024315 FBgn0261596 FBgn0032405 FBgn0039109 FBgn0260011 FBgn0261584
> >      [49] FBgn0053307 FBgn0063649 FBgn0001137 FBgn0259715 FBgn0038149 FBgn0031322 FBgn0001225 FBgn0002868
> >      [57] FBgn0039464 FBgn0265276 FBgn0000406 FBgn0030041 FBgn0033367 FBgn0036985 FBgn0039257 FBgn0039479
> >      [65] FBgn0063498 FBgn0037153 FBgn0051642 FBgn0263232 FBgn0032004 FBgn0031327 FBgn0034389 FBgn0033635
> >      [73] FBgn0034885 FBgn0032598 FBgn0000116 FBgn0261284 FBgn0023549 FBgn0034438 FBgn0031703 FBgn0052625
> >      [81] FBgn0038198 FBgn0050463 FBgn0000567 FBgn0037646 FBgn0038528 FBgn0033065 FBgn0024984 FBgn0034718
> >      [89] FBgn0043364 FBgn0004396 FBgn0035344 FBgn0267635 FBgn0031547 FBgn0032638 FBgn0033775 FBgn0033782
> >      [97] FBgn0039640 FBgn0267252 FBgn0031258 FBgn0030598 FBgn0035727 FBgn0033724 FBgn0263041 FBgn0020248
> >      [105] FBgn0031548 FBgn0034391 FBgn0000079 FBgn0003447 FBgn0039593 FBgn0265512 FBgn0030326 FBgn0039485
> >      [113] FBgn0025836 FBgn0033368 FBgn0037223 FBgn0010223 FBgn0063667 FBgn0002609 FBgn0037678 FBgn0250904
> >      [121] FBgn0264343 FBgn0038237 FBgn0020376 FBgn0028939 FBgn0036560 FBgn0035710 FBgn0035523 FBgn0038261
> >      [129] FBgn0039178 FBgn0034636
> >      130 Levels: FBgn0000071 FBgn0000079 FBgn0000116 FBgn0000406 FBgn0000567 FBgn0001137 ... FBgn0267635
> >      ```
> >
> > 12. `annotatedDEgenes[annotatedDEgenes$Feature == "pseudogene",]`
> >
> >     ```R
> >     > annotatedDEgenes[annotatedDEgenes$Feature == "pseudogene",]
> >     GeneID Base.mean   log2.FC.    StdErr Wald.Stats      P.value        P.adj Seqname   Start
> >     115 FBgn0037223  37.00783 -1.161706 0.2297393  -5.056627 4.267378e-07 1.192225e-05   chr3R 4243605
> >     End Strand    Feature Gene.Name
> >     115 4245511      + pseudogene     TwdlU
> >     ```
> >
> {: .solution}
{: .question}

The subsetting notation is very similar to what we learned for vectors. The key differences include:
- 2 values, separated by commas: `data.frame[row, column]`
- Use of a colon between the numbers (`start:stop, inclusive`) for a continuous range of numbers
- Passing a vector using `c()` for a non continuous set of numbers, pass a vector
- Index using the name of a column(s) by passing them as vectors using `c()`

> <hands-on-title>Create a new data frame by subsetting an existing one</hands-on-title>
> 1. Create a new data frame containing only observations from the "+" strand
>
>    ```R
>    # create a new data frame containing only observations from the "+" strand
>    annotatedDEgenes_plusStrand <- annotatedDEgenes[annotatedDEgenes$Strand == "+",]
>    ```
>
> 2. Check the dimension of the new data frame
>
>    ```R
>    # check the dimension of the data frame
>    > dim(annotatedDEgenes_plusStrand)
>    [1] 72 13
>    ```
>
> 3. Get a summary of the data frame
>
>    ```R
>    # get a summary of the data frame
>    > summary(annotatedDEgenes_plusStrand)
>    GeneID           Base.mean           log2.FC.         StdErr            Wald.Stats
>    FBgn0000071: 1   Min.   :   23.55   Min.   :-4.1485   Min.   :0.08603   Min.   :-30.741
>    FBgn0001224: 1   1st Qu.:   96.04   1st Qu.:-1.3664   1st Qu.:0.12523   1st Qu.:-11.223
>    FBgn0001226: 1   Median :  275.91   Median :-1.0739   Median :0.15648   Median : -5.290
>    FBgn0002609: 1   Mean   : 1365.34   Mean   :-0.3133   Mean   :0.16087   Mean   : -2.729
>    FBgn0003447: 1   3rd Qu.:  988.33   3rd Qu.: 1.1856   3rd Qu.:0.19743   3rd Qu.:  7.592
>    FBgn0003748: 1   Max.   :28915.90   Max.   : 2.3600   Max.   :0.23275   Max.   : 17.399
>    (Other)    :66
>
>    P.value             P.adj              Seqname    Start              End               Strand
>    Min.   :0.000e+00   Min.   :0.000e+00   chr2L:12   Min.   :  127448   Min.   :  140340   -: 0
>    1st Qu.:0.000e+00   1st Qu.:0.000e+00   chr2R:14   1st Qu.: 7377082   1st Qu.: 7379907   +:72
>    Median :0.000e+00   Median :0.000e+00   chr3L:15   Median :14068548   Median :14070644
>    Mean   :4.556e-07   Mean   :1.013e-05   chr3R:24   Mean   :14422245   Mean   :14431849
>    3rd Qu.:6.000e-12   3rd Qu.:3.200e-10   chrX : 7   3rd Qu.:21339535   3rd Qu.:21378634
>    Max.   :7.654e-06   Max.   :1.629e-04              Max.   :31196915   Max.   :31203722
>
>    Feature             Gene.Name
>    lincRNA       : 2   Ama    : 1
>    protein_coding:69   bou    : 1
>    pseudogene    : 1   bves   : 1
>                        CAHbeta: 1
>                        CG10365: 1
>                        CG1124 : 1
>                        (Other):66
>    ```
{: .hands_on}

## Change type of columns

Sometimes, it is possible that R will misinterpret the type of data represented in a data frame, or store that data in a mode which prevents you from operating on the data the way you wish. For example, a long list of gene names isn't usually thought of as a categorical variable, the way that your experimental condition (e.g. control, treatment) might be. More importantly, some R packages you use to analyze your data may expect characters as input, not factors. At other times (such as plotting or some statistical analyses) a factor may be more appropriate. Ultimately, you should know how to change the mode of an object.

First, its very important to recognize that coercion happens in R all the time. This can be a good thing when R gets it right, or a bad thing when the result is not what you expect.

> <hands-on-title>Format gene names</hands-on-title>
> 1. Check the structure of the `Gene.name` column
>
>    ```R
>    > str(annotatedDEgenes$Gene.name)
>    Factor w/ 130 levels "Ama","Amy-p",..: 88 113 5 3 84 26 41 53 69 1 ...
>    ```
>
> 2. Try to coerce the column into numeric
>
>    ```R
>    > as.numeric(annotatedDEgenes$Gene.name)
>    [1]  88 113   5   3  84  26  41  53  69   1  33  27  60 109 114  63 112
>    [18]  89  91 116  87  38 128 121 124  35  92  15 107  57  97   7  59  68
>    [35]  18  13 108  49 126  85  78 105 104 111  24  11  99  44  37  52  82
>    ...
>    ```
>
>    Strangely, it works! Almost. Instead of giving an error message, R returns numeric values, which in this case are the integers assigned to the levels in this factor. This kind of behavior can lead to hard-to-find bugs, for example when we do have numbers in a factor, and we get numbers from a coercion. If we don’t look carefully, we may not notice a problem.
>
> 3. Coerce the column into character (which makes sense for gene names)
>
>    ```R
>    annotatedDEgenes$Gene.name = as.character(annotatedDEgenes$Gene.name)
>    ```
>
{: .hands_on}

R will sometimes coerce without you asking for it (implicit coercion). You can explicitly change it (explicit coercion).

> <tip-title>Avoiding needless coercion</tip-title>
> One way to avoid needless coercion when importing a data frame using functions such as `read.csv()` is to set the argument `StringsAsFactors` to `FALSE`. By default, this argument is `TRUE`. Setting it to `FALSE` will treat any non-numeric column to a character type. Going through the `read.csv() `documentation, you will also see you can explicitly type your columns using the `colClasses` argument. Other R packages (such as the Tidyverse `readr`) don't have this particular conversion issue, but many packages will still try to guess a data type.
{: .tip}

## Inspect a numerical column using math, sorting, renaming

Here are a few operations that don't need much explanation, but which are good to know. There are lots of arithmetic functions you may want to apply to your data frame. For example, you can use functions like `mean()`, `min()`, `max()` on an individual column.

Let's look at the `Base.mean` column, i.e. the mean of normalized counts of all samples, normalizing for sequencing depth and library composition.

> <hands-on-title>Inspect the `Base.mean` column and sort</hands-on-title>
> 1. Get the maximum value of `Base.mean` column
>
>    ```R
>    max(annotatedDEgenes$Base.mean)
>    [1] 65114.84
>    ```
>
> 2. Sort the data frame by values in the `Base.mean` column
>
>    ```R
>    > sorted_by_BaseMean <- annotatedDEgenes[order(annotatedDEgenes$Base.mean), ]
>    > head(sorted_by_BaseMean$Base.mean)
>    [1] 19.15076 23.55006 24.77052 26.16177 27.71424 31.58769
>    ```
>
>    > <question-title>Changing the order</question-title>
>    > The `order()` function lists values in increasing order by default. Look at the documentation for this function and change `sorted_by_BaseMean` to start with differentially expressed means with the greatest base mean.
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```R
>    > > > sorted_by_BaseMean <- annotatedDEgenes[order(annotatedDEgenes$Base.mean, decreasing = TRUE), ]
>    > > > head(sorted_by_BaseMean$Base.mean)
>    > > [1] 65114.841 28915.903 24584.306 22435.886  8903.630  8573.815
>    > > ```
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Coerce the column into character (which makes sense for gene names)
>
>    ```R
>    annotatedDEgenes$Gene.name = as.character(annotatedDEgenes$Gene.name)
>    ```
>
{: .hands_on}

## Save the data frame to a file

We can save data frame as table in our Galaxy history to use it as input for other tools.

> <hands-on-title>Save data frame in Galaxy history</hands-on-title>
>
> 1. Save the `annotatedDEgenes_plusStrand` object to a `.csv` file using the `write.csv()` function:
>
>    ```R
>    > write.csv(annotatedDEgenes_plusStrand, file = "annotatedDEgenes_plusStrand.csv")
>    ```
>
> 2. ***TODO***: *Description on how to save a file from RStudio to Galaxy*
{: .hands_on}

The `write.csv()` function has some additional arguments listed in the help, but at a minimum you need to tell it what data frame to write to file, and give a path to a file name in quotes (if you only provide a file name, the file will be written in the current working directory).

# Aggregate and Analyze with dplyr

Selecting columns and rows using bracket subsetting is handy, but it can be cumbersome and difficult to read, especially for complicated operations.

The functions we've been using so far, like `str()`, come built into R. By using packages, we can access more functions. Here, we would like to use [`dplyr`](https://cran.r-project.org/package=dplyr) package provides a number of very useful functions for manipulating data frames in a way that will reduce repetition, reduce the probability of making errors, and probably even save you some typing. As an added bonus, you might even find the `dplyr` grammar easier to read.

> <comment-title>What is dplyr?</comment-title>
>
> The package `dplyr` is a fairly new (2014) package that tries to provide easy tools for the most common data manipulation tasks. It is built to work directly with data frames. The thinking behind it was largely inspired by the package `plyr` which has been in use for some time but suffered from being slow in some cases.` dplyr` addresses this by porting much of the computation to C++. An additional feature is the ability to work with data stored directly in an external database. The benefits of doing this are that the data can be managed natively in a relational database, queries can be conducted on that database, and only the results of the query returned.
>
> This addresses a common problem with R in that all operations are conducted in memory and thus the amount of data you can work with is limited by available memory. The database connections essentially remove that limitation in that you can have a database of many 100s GB, conduct queries on it directly and pull back just what you need for analysis in R.
{: .comment}

To use `dplyr`, we need first to install the package and then load it to be able to use it.

> <hands-on-title>Install and load dyplr</hands-on-title>
>
> 1. Load dyplr
>
>    ```R
>    ## load dyplr package
>    library("dplyr")
>    ```
>
>    > <tip-title>Didn't work?</tip-title>
>    > You might need to install dyplr first
>    >
>    > ```R
>    > > install.packages("dplyr")
>    > ```
>    >
>    > You might get asked to choose a CRAN mirror -- this is asking you to choose a site to download the package from. The choice doesn't matter too much.
>    {: .tip}
>
{: .hands_on}

We only need to install a package once per computer, but we need to load it every time we open a new R session and want to use that package.

## Select columns and filter rows

To select columns, instead with bracket, we could use the `select` function of `dplyr`

> <hands-on-title>Select column</hands-on-title>
>
> 1. Select `GeneID`, `Start`, `End`, `Strand` columns in `annotatedDEgenes`
>
>    ```R
>    > select(annotatedDEgenes, GeneID, Start, End, Strand)
>             GeneID    Start      End Strand
>    1   FBgn0039155 24141394 24147490      +
>    2   FBgn0003360 10780892 10786958      -
>    3   FBgn0026562 26869237 26871995      -
>    4   FBgn0025111 10778953 10786907      -
>    5   FBgn0029167 13846053 13860001      +
>    6   FBgn0039827 31196915 31203722      +
>    7   FBgn0035085 24945138 24946636      +
>    8   FBgn0034736 22550093 22552113      +
>    9   FBgn0264475   820758   821512      +
>    10  FBgn0000071  6762592  6765261      +
>    # ... with 120 more rows
>    ```
>
>    The first argument to this function is the data frame (`annotatedDEgenes`), and the subsequent arguments are the columns to keep.
>
> 2. Select all columns *except* certain ones
>
>    ```R
>    > select(annotatedDEgenes, -Chromosome)
>             GeneID   Base.mean  log2.FC.     StdErr Wald.Stats       P.value         P.adj    Start End Strand    Feature    Gene.Name
>    1   FBgn0039155  1086.97430 -4.148450 0.13494887 -30.740902 1.617921e-207 1.387691e-203 24141394  24147490      +  protein_coding         Kal1
>    2   FBgn0003360  6409.57713 -2.999777 0.10434506 -28.748628 9.422382e-182 4.040788e-178 10780892  10786958      -  protein_coding         sesB
>    3   FBgn0026562 65114.84056 -2.380164 0.08432692 -28.225437 2.850473e-175 8.149503e-172 26869237  26871995      -  protein_coding  BM-40-SPARC
>    4   FBgn0025111  2192.32237  2.699939 0.09794457  27.565988 2.846764e-167 6.104174e-164 10778953  10786907      -  protein_coding         Ant2
>    5   FBgn0029167  5430.06728 -2.105062 0.09254660 -22.745964 1.573283e-114 2.698810e-111 13846053  13860001      +  protein_coding          Hml
>    6   FBgn0039827   390.90178 -3.503014 0.16002962 -21.889786 3.250384e-106 4.646424e-103 31196915  31203722      +  protein_coding       CG1544
>    7   FBgn0035085   928.26381 -2.414074 0.11518516 -20.958204  1.579343e-97  1.935147e-94 24945138   24946636      + protein_coding       CG3770
>    8   FBgn0034736   330.38302 -3.018179 0.15815418 -19.083774  3.444661e-81  3.693107e-78 22550093   22552113      + protein_coding       CG6018
>    9   FBgn0264475   955.45445 -2.334486 0.12423003 -18.791643  8.840041e-79  8.424559e-76   820758   821512      +   lincRNA      CR43883
>    10  FBgn0000071   468.05793  2.360017 0.13564397  17.398615  8.452137e-68  7.249398e-65  6762592   6765261      +  protein_coding          Ama
>    # ... with 120 more rows
>    ```
{: .hands_on}

`dplyr` also provides useful functions to select columns based on their names. For instance, `starts_with()` allows you to select columns that starts with specific letters.

> <hands-on-title>Select columns that start with the letter "P"</hands-on-title>
>
> 1. Select columns that start with the letter "P"
>
>    ```R
>    > select(annotatedDEgenes, starts_with("P."))
>
>              P.value         P.adj
>    1   1.617921e-207 1.387691e-203
>    2   9.422382e-182 4.040788e-178
>    3   2.850473e-175 8.149503e-172
>    4   2.846764e-167 6.104174e-164
>    5   1.573283e-114 2.698810e-111
>    6   3.250384e-106 4.646424e-103
>    7    1.579343e-97  1.935147e-94
>    8    3.444661e-81  3.693107e-78
>    9    8.840041e-79  8.424559e-76
>    10   8.452137e-68  7.249398e-65
>    # ... with 120 more rows
>    ```
{: .hands_on}

> <question-title>Selecting on multiple conditions</question-title>
>
> Create a table that contains all the columns with the letter "s" in their name except for the column "Wald.Stats", and that contains the column "End".
>
> Hint: look at the help for the function `starts_with()` we've just covered.
>
> > <solution-title></solution-title>
> >
> > ```R
> > select(annotatedDEgenes, contains("s"), -Wald.Stats, End)
> > ```
>  {: .solution}
{: .question}

Similarly to choose rows, we can use the `filter()` function.

> <hands-on-title>Filter rows</hands-on-title>
>
> 1. Filter for rows on the forward strand
>
>    ```R
>    > filter(annotatedDEgenes, Strand == "+")
>            GeneID   Base.mean  log2.FC.     StdErr Wald.Stats       P.value         P.adj Chromosome Start      End Strand        Feature    Gene.Name
>    1  FBgn0039155  1086.97430 -4.148450 0.13494887 -30.740902 1.617921e-207 1.387691e-203      chr3R 24141394 24147490      + protein_coding         Kal1
>    2  FBgn0029167  5430.06728 -2.105062 0.09254660 -22.745964 1.573283e-114 2.698810e-111      chr3L 13846053 13860001      + protein_coding          Hml
>    3  FBgn0039827   390.90178 -3.503014 0.16002962 -21.889786 3.250384e-106 4.646424e-103      chr3R 31196915 31203722      + protein_coding       CG1544
>    4  FBgn0035085   928.26381 -2.414074 0.11518516 -20.958204  1.579343e-97  1.935147e-94      chr2R 24945138 24946636      + protein_coding       CG3770
>    5  FBgn0034736   330.38302 -3.018179 0.15815418 -19.083774  3.444661e-81  3.693107e-78      chr2R 22550093 22552113      + protein_coding       CG6018
>    6  FBgn0264475   955.45445 -2.334486 0.12423003 -18.791643  8.840041e-79  8.424559e-76      chr3L 820758   821512      +        lincRNA      CR43883
>    7  FBgn0000071   468.05793  2.360017 0.13564397  17.398615  8.452137e-68  7.249398e-65      chr3R 6762592  6765261      + protein_coding          Ama
>    8  FBgn0038832   429.85033 -2.340711 0.15483651 -15.117308  1.245307e-51  8.900828e-49      chr3R 20842139 20844981      + protein_coding      CG15695
>    9  FBgn0037754   394.46410 -2.071994 0.14494069 -14.295459  2.335613e-46  1.540966e-43      chr3R 9784652  9789323      + protein_coding       CG8500
>    10 FBgn0034897  1340.88978 -1.795107 0.12629203 -14.213934  7.508655e-46  4.293449e-43      chr2R 23713899 23734846      + protein_coding         Sesn
>    # ... with 120 more rows
>    ```
>
>    > <comment-title>Equivalent in base R code</comment-title>
>    > Note that this is equivalent to the base R code below, but is easier to read!
>    >
>    > ```R
>    > > annotatedDEgenes[annotatedDEgenes$Strand == "+",]
>    > ```
>    {: .comment}
>
>    `filter()` keeps all the rows that match the conditions that are provided
>
> 2. Filter for rows corresponding to genes in Chromosome X or 2R
>
>    ```R
>    ## rows for genes in Chromosome X or 2R
>    filter(annotatedDEgenes, Chromosome %in% c("chrX", "chr2R"))
>    ```
>
> 3. Filter for rows where the log2 fold change is greater than 2
>
>    ```R
>    ## rows where the log2 fold change is greater than 2
>    filter(annotatedDEgenes, log2.FC. >= 2)
>    ```
{: .hands_on}

`filter()` allows to combine multiple conditions. They need to be separated using a `,` as arguments to the function, they will be combined using the `&` (AND) logical operator. If you need to use the `|` (OR) logical operator, you can specify it explicitly:

> <hands-on-title>Filter rows for multiple conditions</hands-on-title>
>
> 1. Filter for genes on chrX and with adjusted p-value <= 1e-100
>
>    ```R
>    ## both are equivalent
>    filter(annotatedDEgenes, Chromosome == "chrX" & P.adj <= 1e-100)
>    filter(annotatedDEgenes, Chromosome == "chrX", P.adj <= 1e-100)
>    ```
>
> 2. Filter for genes on chrX and with log2 Fold Change smaller than -2 or larger than 2
>
>    ```R
>    filter(annotatedDEgenes, Chromosome == "chrX", (log2.FC. <= -2 | log2.FC. >= 2))
>    ```
{: .hands_on}

> <question-title>Practising with conditionals</question-title>
>
> Select all the rows for genes that:
> - start after position $$1e^6$$ (one million) and before position $$2e^6$$ (included) in their chromosome
> - have a log2 fold change greater than 1 or an adjusted p-value less than $$1e^{-75}$$.
>
> > <solution-title></solution-title>
> >
> > ```R
> > filter(annotatedDEgenes, Start >= 1e6 & End <= 2e6, (log2.FC. > 1 | P.adj < 1e-75)
> > ```
> {: .solution}
{: .question}

But what if you wanted to select and filter?

We can do this with **pipes**. Pipes, are a fairly recent addition to R. They let you take the output of one function and send it directly to the next, which is useful when you need to do many things to the same data set. It was possible to do this before pipes were added to R, but it was much messier and more difficult. Pipes in R look like `%>%` and are made available via the `magrittr` package, which is installed as part of `dplyr`.

> <comment-title>Pipes on RStudio</comment-title>
>
> With RStudio, you can type the pipe with
> - <kbd>Ctrl</kbd> + <kbd>Shift</kbd> + <kbd>M</kbd> if you're using a PC
> - <kbd>Cmd</kbd> + <kbd>Shift</kbd> + <kbd>M</kbd> if you're using a Mac
{: .comment}

> <hands-on-title>Select and filter</hands-on-title>
>
> 1. Filter for genes on forward strand, select `GeneID`, `Start`, `End`, `Chromosome` and display the first results using pipes
>
>    ```R
>    > annotatedDEgenes %>%
>       filter(Strand == "+") %>%
>       select(GeneID, Start, End, Chromosome) %>%
>       head()
>
>           GeneID    Start      End Chromosome
>    1 FBgn0039155 24141394 24147490      chr3R
>    2 FBgn0029167 13846053 13860001      chr3L
>    3 FBgn0039827 31196915 31203722      chr3R
>    4 FBgn0035085 24945138 24946636      chr2R
>    5 FBgn0034736 22550093 22552113      chr2R
>    6 FBgn0264475   820758   821512      chr3L
>    ```
>
{: .hands_on}

In the above code, we use the pipe to
1. Send the `annotatedDEgenes` dataset through `filter()`
2. Keep rows for genes on the '+' strand of their chromosome
3. Select through `select()` to keep only the `GeneID`, `Start`, `End`, and `Chromosome` columns
4. Show only the first 6 rows with `head`

> <comment-title>No need to give 1st argument to the functions</comment-title>
>
> Since `%>%` takes the object on its left and passes it as the first argument to the function on its right, we don't need to explicitly include the data frame as an argument to the `filter()` and `select()` functions any more.
>
{: .comment}

Some may find it helpful to read the pipe like the word "then". For instance, in the above example, we took the data frame `annotatedDEgenes`, *then* we `filter`ed for rows where `Strand` was +, *then* we `select`ed the `GeneID`, `Start`, `End`, and `Chromosome` columns, *then* we showed only the first six rows.

The `dplyr` functions by themselves are somewhat simple, but by combining them into linear workflows with the pipe, we can accomplish complex manipulations of data frames.

If we want to create a new object with this smaller version of the data we can do so by assigning it to a new object.

> <hands-on-title>Select and filter to create new object</hands-on-title>
>
> 1. Create an `plus_strand_genes` from `annotatedDEgenes` after filter for genes on forward strand and selection of `GeneID`, `Start`, `End`, `Chromosome`
>
>    ```R
>    plus_strand_genes <- annotatedDEgenes %>%
>     filter(Strand == "+") %>%
>     select(GeneID, Start,


topic: data-science
tutorial: python-typing
title: Python - Type annotations
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: b1dbbfdf875a261388a5cacd67de8e68
last_modified: 2026-02-26T21:12:14.905459
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-typing/tutorial.html

content:
In some languages type annotations are a core part of the language and types are checked at compile time, to ensure your code can never use the incorrect type of object. Python, and a few other dynamic languages, instead use ["Duck Typing"](https://en.wikipedia.org/wiki/Duck_typing) wherein the type of the object is less important than whether or not the correct methods or attributes are available.

However, we can provide type hints as we write python which will allow our editor to type check code as we go, even if it is not typically enforced at any point.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Types

Types used for annotations can be any of the base types:

```python
str
int
float
bool
None
...
```

or they can be relabeling of existing types, letting you create new types as needed to represent your internal data structures

```python
from typing import NewType

NameType = NewType("NameType", str)
Point2D = NewType("Point2D", tuple[float, float])
```

> <tip-title>Does `tuple` cause an issue?</tip-title>
> You might be on a python earlier than 3.9. Please update, or rewrite these as Tuple and List which must be imported.
{: .tip}

## But why?

Imagine for a minute you have a situation like the following, take a minute to read and understand the code:

```python
# Fetch the user and history list
(history_id, user_id) = GetUserAndCurrentHistory("hexylena")

# And make sure all of the permissions are correct
history = History.fetch(history_id)
history.share_with(user_id)
history.save()
```

> <question-title></question-title>
> 1. Can you be sure the `history_id` and `user_id` are in the correct order? It
>    seems like potentially not, given the ordering of "user" and "history" in the
>    function name, but without inspecting the definition of that function we
>    won't know.
> 2. What happens if `history_id` and `user_id` are swapped?
>
> > <solution-title></solution-title>
> > 1. This is unanswerable without the code.
> > 2. Depending on the magnitude of `history_id` and `user_id`, those may be within allowable ranges. Take for example
> > 
> >    User | History Id
> >    ---- | ----------
> >    1    | 1
> >    1    | 2
> >    2    | 3
> >    2    | 4
> >    
> >    Given `user_id=1` and `history_id=2` we may intend that the second row in our tables, history #2 owned by user #1, is shared with that user, as they're the owner. But if those are backwards, we'll get a situation where history #1 is actually associated with user #1, but instead we're sharing with user #2. We've created a situation where we've accidentally shared the wrong history with the wrong user! This could be a GDPR violation for our system and cause a lot of trouble.
> {: .solution}
{: .question}

However, if we have type definitions for the `UserId` and `HistoryId` that declare them as their own types:


```python
from typing import NewType

UserId = NewType("UserId", int)
HistoryId = NewType("HistoryId", int)
```

And then defined on our function, e.g.

```python
def GetUserAndCurrentHistory(username: str) -> tuple[UserId, HistoryId]:
    x = UserId(1) # Pretend this is fetching from the database
    y = HistoryId(2) # Likewise
    return (x, y)
```

we would be able to catch that, even if we call the variable `user_id`, it will still be typed checked. 

```python
history_id: HistoryId
user_id: UserId

(user_id, history_id) = GetUserAndCurrentHistory("hexylena")
(history_id, user_id) = GetUserAndCurrentHistory("hexylena")
```

If we're using a code editor with typing hints, e.g. VSCode with PyLance, we'll see something like:

![Screenshot of VSCode showing the functions from above. The version with history_id first has a bright red line under the function call of GetUserAndCurrentHistory. A popup tab shown on hovering over the function name shows that Expression of type UserId cannot be assigned to declared type HistoryId. UserId is incompatible with HistoryId](../../images/typing.png)

Here we see that we're not allowed to call this function this way, it's simply impossible.

> <question-title></question-title>
> What happens if you execute this code?
> > <solution-title></solution-title>
> > It executes happily. Types are **not enforced at runtime**. So this case where they're both custom types around an integer, Python sees that it expects an int in both versions of the function call, and that works fine for it. That is why we are repeatedly calling them "type hints", they're hints to your editor to show suggestions and help catch bugs, but they're not enforced.
> > If you modified the line `y = HistoryId(2)` to be something like `y = "test"`, the code will also execute fine. Python doesn't care that there's suddenly a string where you promised and asked for, an int. It simply does not matter.
> >
> > However, types *are* checked when you do operations involving them. Trying to get the `len()` of an integer? That will raise an `TypeError`, as integers don't support the `len()` call.
> {: .solution}
{: .question}

## Typing Variables

Adding types to variables is easy, you've seen a few examples already:

```python
a: str = "Hello"
b: int = 3
c: float = 3.14159
d: bool = True
```

### Complex Types

But you can go further than this with things like `tuple` and `list` types:

```python
e: list[int] = [1, 2, 3]
f: tuple[int, str] = (3, "Hi.")
g: list[tuple[int, int]] = [(1, 2), (3, 4)]
```

### Typing Functions

Likewise you've seen an example of adding type hints to a function:

```python
def reverse_list_of_ints(a: list[int]) -> list[int]:
    return a[::-1]
```

But this is a very specific function, right? We can reverse lists with more than just integers. For this, you can use `Any`:

```python
from typing import Any

def reverse_list(a: list[Any]) -> list[Any]:
    return a[::-1]
```

But this will lose the type information from the start of the function to the end. You said it was a `list[Any]` so your editor might not provide any type hints there, even though you could know, that calling it with a `list[int]` would always return the same type. Instead you can do

```python
from typing import TypeVar

T = TypeVar("T") # Implicitly any

def reverse_list(a: list[T]) -> list[T]:
    return a[::-1]
```

Now this will allow the function to accept a list of any type of value, int, float, etc. But it will also accept types you might not have intended:

```python
w: list[tuple[int, int]] = [(1, 2), (3, 4), (5, 8)]
reverse_list(w)
```

We can lock down what types we'll accept by using a `Union` instead of `Any`. With a `Union`, we can define that a type in that position might be any one of a few more specific types. Say your function can only accept strings, integers, or floats:

```python
from typing import Union

def reverse_list(a: list[Union[int, float, str]]) -> list[Union[int, float, str]]:
    return a[::-1]
```

Here we have used a `Union[A, B, ...]` to declare that it can only be one of these three types.

> <question-title></question-title>
> 1. Are both of these valid definitions?`
> 
>    ```python
>    q1: list[Union[int, float, str]] = [1, 2, 3]
>    q2: list[Union[int, float, str]] = [1, 2.3214, "asdf"]
>    ```
> 
> 2. If that wasn't what you expected, how would you define it so that it would be?
> > <solution-title></solution-title>
> > Yes, both are valid, but maybe you expected a homogeneous list. If you wanted that, you could instead do
> > 
> > ```python
> > q3: Union[list[int], list[float], list[str]] = [1, 2, 3]
> > q4: Union[list[int], list[float], list[str]] = [1, 2.3243, "asdf"] # Fails
> > ```
> {: .solution}
{: .question}


### Optional

Sometimes you have an argument to a function that is truly optional, maybe you have a different code path if it isn't there, or you simply process things differently but still correctly. You can explicitly declare this by defining it as `Optional`

```python
from typing import Optional

def pretty(lines: list[str], padding: Optional[str] = None) -> None:
    for line in lines:
        if padding:
            print(f"{padding} {line}")
        else:
            print(line)


lines = ["hello", "world", "你好", "世界"]

# Without the optional argument
pretty(lines)
# And with the optional
pretty(lines, "★")
```

While this superficially *looks* like a keyword argument with a default value, however it's subtly different. Here an explicit value of None is allowed, and we still know that it will either be a string, or it will be None. Not something that was possible with just a keyword argument.

## Testing for Types

You can use `mypy` to ensure that these type annotations are working in a project, this is a step you could add to your automated testing, if you have that. Using the `HistoryId`/`UserId` example from above, we can write that out into a script and test it out by running `mypy` on that file:

```console
$ mypy tmp.py
tmp.py:15: error: Incompatible types in assignment (expression has type "UserId", variable has type "HistoryId")
tmp.py:15: error: Incompatible types in assignment (expression has type "HistoryId", variable has type "UserId")
```

Here it reports the errors in the console, and you can use this to prevent bad code from being committed.

## Exercise

Here is an example module that would be stored in `corp/__init__.py`

```python
def repeat(x, n):
    """Return a list containing n references to x."""
    return [x]*n


def print_capitalized(x):
    """Print x capitalized, and return x."""
    print(x.capitalize())
    return x


def concatenate(x, y) :
    """Add two strings together."""
    return x + y
```

And here are some example invocations of that module, as found in `test.py`

```python
from corp import *

x = repeat("A", 3) # Should return ["A", "A", "A"]
y = print_capitalized("hElLo WorLd") # Should print Hello World
z = concatenate("Hi", "Bob") # HiBob
```

> <hands-on-title>Add type annotations</hands-on-title>
> 1. Add type annotations to each of those functions AND the variables `x`, `y`, `z`
> 2. How did you know which types were appropriate? 
> 3. Does `mypy` approve of your annotations? (Run `mypy test.py`, once you've written the above files out to their appropriate locations.)
{: .hands_on}

> <solution-title></solution-title>
> 1. The proper annotations: 
>
>    ```python
>    def repeat(x: str, n: int) -> list[str]:
>    # Or
>    from typing import TypeVar
>    T = TypeVar("T")
>    def repeat(x: T, n: int) -> list[T]:
> 
>    def print_capitalized(x: str) -> str:
> 
>    def concatenate(x: str, y:str) -> str:
>    ```
> 
>    and
> 
>    ```python
>    x: list[str] = ...
>    y: str = ...
>    z: str = ...
>    ```
> 
> 2. You might have discovered this by a combination of looking at the function definitions and their documentation, and perhaps also the sample invocations and what types were passed there.
> 3. We hope so!
>
{: .solution}

## Automation with MonkeyType

You can use MonkeyType to automatically apply type annotations to your code. Based on the execution of the code, it will make a best guess about what types are supported.

> <hands-on-title>Using MonkeyType to generate automatic annotations</hands-on-title>
> 
> 1. Create a folder for a module named `some`
> 2. Touch `some/__init__.py` to ensure it's importable as a python module
> 3. Create `some/module.py` and add the following contents:
> 
>    ```python
>    def add(a, b):
>        return a + B
>    ```
> 
> 4. Create a script that uses that module:
> 
>    ```
>    from some.module import add
>    
>    add(1, 2)
>    ```
> 
> 5. `pip install monkeytype`
> 6. Run MonkeyType to generate the annotations
> 
>    ```console
>    monkeytype run myscript.py
>    ```
> 
> 7. View the generated annotations
> 
>    ```console
>    monkeytype stub myscript.py
>    ```
{: .hands_on}

> <question-title></question-title>
> 1. What was the output of that command?
> 2. This function will accept strings as well, add a statement to exercise that in `myscript.py` and re-run `monkeytype run` and `monkeytype stub`. What is the new output?
>
> > <solution-title></solution-title>
> > 1. The expected output is:
> > 
> >    ```python
> >    def add(a: int, b: int) -> int: ...
> >    ```
> > 
> > 2. You can add a statement like `add("a", "b")` below `add(1, 2)` to see:
> > 
> >    ```python
> >    def add(a: Union[int, str], b: Union[int, str]) -> Union[int, str]: ...
> >    ```
> {: .solution}
{: .question}

> <question-title></question-title>
> Why is it different?
>
> > <solution-title></solution-title>
> > Because MonkeyType works by running the code provided (`myscript.py`) and annotating based on what executions it saw. In the first invocation it had not seen any calls to `add()` with strings, so it only reported `int` as acceptable types. However, the second time it saw `str`s as well. Can you think of another type that would be supported by this operation, that was not caught? (list!)
> {: .solution} 
{: .question}

> <question-title></question-title>
> 1. Does that type annotation make sense based on what you've learned today?
> 2. Can you write a better type annoation based on what you know?
>
> > <solution-title></solution-title>
> > 1. It works, but it's not a great type annotation. Here the description looks like it can accept two `int`s and return a `str` which isn't correct.
> > 2. Here is a better type annotation
> > 
> >    ```python
> >    from typing import TypeVar
> >    T = TypeVar("T", int, str, list)
> >    
> >    def add(a: T, b: T) -> T:
> >        return a + b
> >    ```
> {: .solution} 
{: .question}


topic: data-science
tutorial: bash-git
title: Version Control with Git
difficulty: intermediate
time_estimation: 65M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 9f8a31eb712dcf823b823c8bca01e882
last_modified: 2026-02-26T21:12:14.736398
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/bash-git/tutorial.html

content:
Version control is a way of tracking the change history of a project, and `git` is one of the most popular systems for doing that! This tutorial will guide you through the basics of using git for version control.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Basics

We'll start by exploring how version control can be used
to keep track of what one person did and when.
Even if you aren't collaborating with other people,
automated version control is much better than this situation:

![Cartoon titled 'final'.doc, showing a grad student and their advisor going through multiple revisions. The first named final.doc, then final_rev.2.doc, final_rev.6.comments.doc, a long filename with the revision number 18, until a final filename, revision 22, with special characters indicating frustration where the file name includes the text 'why did I come to grad school'.](../../images/bash-git/phd101212s.png "Piled Higher and Deeper by Jorge Cham, http://www.phdcomics.com")

We've all been in this situation before: it seems unnecessary to have
multiple nearly-identical versions of the same document. Some word
processors let us deal with this a little better, such as Microsoft
Word's
[Track Changes](https://support.office.com/en-us/article/Track-changes-in-Word-197ba630-0f5f-4a8e-9a77-3712475e806a),
Google Docs' [version history](https://support.google.com/docs/answer/190843?hl=en), or
LibreOffice's [Recording and Displaying Changes](https://help.libreoffice.org/Common/Recording_and_Displaying_Changes).

Version control systems start with a base version of the document and
then record changes you make each step of the way. You can
think of it as a recording of your progress: you can rewind to start at the base
document and play back each change you made, eventually arriving at your
more recent version.

![Changes Are Saved Sequentially, graphic shows three documents with text being added in each new revision.](../../images/bash-git/play-changes.svg)

Once you think of changes as separate from the document itself, you
can then think about "playing back" different sets of changes on the base document, ultimately
resulting in different versions of that document. For example, two users can make independent
sets of changes on the same document.

![Different Versions Can be Saved, showing a document splitting into two, with different changes.](../../images/bash-git/versions.svg)

Unless multiple users make changes to the same section of the document - a conflict - you can
incorporate two sets of changes into the same base document.

![Multiple Versions Can be Merged, shows two documents with different changes merging into a final document with both changes.](../../images/bash-git/merge.svg)

A version control system is a tool that keeps track of these changes for us,
effectively creating different versions of our files. It allows us to decide
which changes will be made to the next version (each record of these changes is
called a [commit](https://git-scm.com/docs/gitglossary#def_commit), and keeps useful metadata
about them. The complete history of commits for a particular project and their
metadata make up a [repository](https://git-scm.com/docs/gitglossary#def_repository).
Repositories can be kept in sync across different computers, facilitating
collaboration among different people.

> <tip-title>The Long History of Version Control Systems</tip-title>
>
> Automated version control systems are nothing new.
> Tools like [RCS](https://en.wikipedia.org/wiki/Revision_Control_System), [CVS](https://en.wikipedia.org/wiki/Concurrent_Versions_System), or [Subversion](https://en.wikipedia.org/wiki/Apache_Subversion) have been around since the early 1980s and are used by
> many large companies.
> However, many of these are now considered legacy systems (i.e., outdated) due to various
> limitations in their capabilities.
> More modern systems, such as Git and [Mercurial](https://swcarpentry.github.io/hg-novice/),
> are *distributed*, meaning that they do not need a centralized server to host the repository.
> These modern systems also include powerful merging tools that make it possible for
> multiple authors to work on
> the same files concurrently.
{: .tip}

> <question-title>Paper Writing</question-title>
>
> *   Imagine you drafted an excellent paragraph for a paper you are writing, but later ruin
>     it. How would you retrieve the *excellent* version of your conclusion? Is it even possible?
>
> *   Imagine you have 5 co-authors. How would you manage the changes and comments
>     they make to your paper?  If you use LibreOffice Writer or Microsoft Word, what happens if
>     you accept changes made using the `Track Changes` option? Do you have a
>     history of those changes?
>
> > <solution-title></solution-title>
> >
> > *   Recovering the excellent version is only possible if you created a copy
> >     of the old version of the paper. The danger of losing good versions
> >     often leads to the problematic workflow illustrated in the PhD Comics
> >     cartoon at the top of this page.
> >
> > *   Collaborative writing with traditional word processors is cumbersome.
> >     Either every collaborator has to work on a document sequentially
> >     (slowing down the process of writing), or you have to send out a
> >     version to all collaborators and manually merge their comments into
> >     your document. The 'track changes' or 'record changes' option can
> >     highlight changes for you and simplifies merging, but as soon as you
> >     accept changes you will lose their history. You will then no longer
> >     know who suggested that change, why it was suggested, or when it was
> >     merged into the rest of the document. Even online word processors like
> >     Google Docs or Microsoft Office Online do not fully resolve these
> >     problems.
> {: .solution}
{: .question }


Before diving in the tutorial, we need to open {% tool [RStudio](interactive_tool_rstudio) %}. If you do not know how or never interacted with RStudio, please follow the [dedicated tutorial]({% link topics/galaxy-interface/tutorials/rstudio/tutorial.md %}).

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Installing git</hands-on-title>
> The R Console and other interactive tools like RStudio are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. This is one of those cases and, in order to do that, we will use the terminal provided by the RStudio itself.
> We go to "Tools" and pick the "Shell..." option and we are good to go. Our workspace is the left, terminal window that just opened.
>
> Fortunately, [miniconda](https://docs.conda.io/en/latest/miniconda.html) is already installed. Miniconda is a package manager that simplifies the installation processes. We can and will use it to install every essential package for our tutorial. However, it is of critical importance that we do that in an new environment within our existing base and install our packages in said environment.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ conda create -n name_of_your_env nano git
> > $ conda activate name_of_your_env
> > ```
> {: .code-in}
>
>
> | Software | Version | Manual | Available for | Description |
> | -------- | ------------ | ------ | ------------- | ----------- |
> | [git](https://git-scm.com/) | 2.35.3 | [git Manual](https://git-scm.com/book/en/v2) | Linux, MacOS | Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. |
> | [GNU nano](https://www.nano-editor.org/) | 2.9.8 | [Nano manual](https://www.nano-editor.org/dist/latest/nano.html) | Linux, MacOS | GNU nano is a small and friendly text editor. |
>
{: .hands_on}

# Setting up git

When we use Git on a new computer for the first time,
we need to configure a few things. Below are a few examples
of configurations we will set as we get started with Git:

*   our name and email address,
*   what our preferred text editor is,
*   and that we want to use these settings globally (i.e. for every project).

On a command line, Git commands are written as `git verb options`,
where `verb` is what we actually want to do and `options` is additional optional information which may be needed for the `verb`. So here is how
Sherlock sets up his new laptop:

> <code-in-title>Setting up with bash</code-in-title>
> ```bash
> $ git config --global user.name "Sherlock Holmes"
> $ git config --global user.email "sherlock@baker.street"
> ```
{: .code-in}

Please use your own name and email address instead of Sherlocks's. This user name and email will be associated with your subsequent Git activity,
which means that any changes pushed to
[GitHub](https://github.com/),
[BitBucket](https://bitbucket.org/),
[GitLab](https://gitlab.com/) or
another Git host server
after this lesson will include this information.

For this lesson, we will be interacting with [GitHub](https://github.com/) and so the email address used should be the same as the one used when setting up your GitHub account. If you are concerned about privacy, please review [GitHub's instructions for keeping your email address private][git-privacy].

> <tip-title>Keeping your email private</tip-title>
>
> If you choose to use a private email address with GitHub, then use that same email address for the `user.email` value, e.g. `username@users.noreply.github.com` replacing `username` with your GitHub one.
{: .tip}


> <tip-title>Line Endings</tip-title>
>
> As with other keys, when you hit <kbd>Enter</kbd> or <kbd></kbd> or on Macs, <kbd>Return</kbd>, on your keyboard,
> your computer encodes this input as a character.
> Different operating systems use different character(s) to represent the end of a line.
> (You may also hear these referred to as newlines or line breaks.)
> Because Git uses these characters to compare files,
> it may cause unexpected issues when editing a file on different machines.
> Though it is beyond the scope of this lesson, you can read more about this issue
> [in the Pro Git book](https://www.git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_core_autocrlf).
>
> You can change the way Git recognizes and encodes line endings
> using the `core.autocrlf` command to `git config`.
> The following settings are recommended:
>
> > <code-in-title>Settings with bash</code-in-title>
> > On macOS and Linux:
> >
> >```bash
> >$ git config --global core.autocrlf input
> >```
> >
> >And on Windows:
> >
> >```bash
> >$ git config --global core.autocrlf true
> >```
> {: .code-in}
{: .tip}

Sherlock also has to set his favorite text editor, following this table:

| Editor             | Configuration command                            |
|:-------------------|:-------------------------------------------------|
| Atom | `$ git config --global core.editor "atom --wait"`|
| nano               | `$ git config --global core.editor "nano -w"`    |
| BBEdit (Mac, with command line tools) | `$ git config --global core.editor "bbedit -w"`    |
| Sublime Text (Mac) | `$ git config --global core.editor "/Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl -n -w"` |
| Sublime Text (Win, 32-bit install) | `$ git config --global core.editor "'c:/program files (x86)/sublime text 3/sublime_text.exe' -w"` |
| Sublime Text (Win, 64-bit install) | `$ git config --global core.editor "'c:/program files/sublime text 3/sublime_text.exe' -w"` |
| Notepad (Win)    | `$ git config --global core.editor "c:/Windows/System32/notepad.exe"`|
| Notepad++ (Win, 32-bit install)    | `$ git config --global core.editor "'c:/program files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"`|
| Notepad++ (Win, 64-bit install)    | `$ git config --global core.editor "'c:/program files/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"`|
| Kate (Linux)       | `$ git config --global core.editor "kate"`       |
| Gedit (Linux)      | `$ git config --global core.editor "gedit --wait --new-window"`   |
| Scratch (Linux)       | `$ git config --global core.editor "scratch-text-editor"`  |
| Emacs              | `$ git config --global core.editor "emacs"`   |
| Vim                | `$ git config --global core.editor "vim"`   |
| VS Code                | `$ git config --global core.editor "code --wait"`   |

It is possible to reconfigure the text editor for Git whenever you want to change it.

> <tip-title>Exiting Vim</tip-title>
> Note that Vim is the default editor for many programs. If you haven't used Vim before and wish to exit a session without saving
> your changes, press <kbd>Esc</kbd> then type `:q!` and hit <kbd>Enter</kbd> or <kbd></kbd> or on Macs, <kbd>Return</kbd>.
> If you want to save your changes and quit, press <kbd>Esc</kbd> then type `:wq` and hit <kbd>Enter</kbd> or <kbd></kbd> or on Macs, <kbd>Return</kbd>.
{: .tip}

Git (2.28+) allows configuration of the name of the branch created when you
initialize any new repository.  Sherlock decides to use that feature to set it to `main` so
it matches the cloud service he will eventually use.

> <code-in-title>Configure the name of the created branch</code-in-title>
> ```bash
> $ git config --global init.defaultBranch main
> ```
{: .code-in}

> <tip-title>Default Git branch naming</tip-title>
>
> Source file changes are associated with a "branch."
> For new learners in this lesson, it's enough to know that branches exist, and this lesson uses one branch.
> By default, Git will create a branch called `master`
> when you create a new repository with `git init` (as explained in the next Episode). This term evokes
> the racist practice of human slavery and the
> [software development community](https://github.com/github/renaming)  has moved to adopt
> more inclusive language.
>
> In 2020, most Git code hosting services transitioned to using `main` as the default
> branch. As an example, any new repository that is opened in GitHub and GitLab default
> to `main`.  However, Git has not yet made the same change.  As a result, local repositories
> must be manually configured have the same main branch name as most cloud services.
>
> For versions of Git prior to 2.28, the change can be made on an individual repository level.  The
> command for this is in the next episode.  Note that if this value is unset in your local Git
> configuration, the `init.defaultBranch` value defaults to `master`.
>
{: .tip}

The five commands we just ran above only need to be run once: the flag `--global` tells Git
to use the settings for every project, in your user account, on this computer.

You can check your settings at any time:

> <code-in-title>Checking your settings</code-in-title>
> ```bash
> $ git config --list
> ```
{: .code-in}

You can change your configuration as many times as you want: use the
same commands to choose another editor or update your email address.

> <tip-title>Proxy</tip-title>
>
> In some networks you need to use a
> [proxy](https://en.wikipedia.org/wiki/Proxy_server). If this is the case, you
> may also need to tell Git about the proxy:
>
> > <code-in-title>Git and proxy</code-in-title>
> > ```bash
> > $ git config --global http.proxy proxy-url
> > $ git config --global https.proxy proxy-url
> > ```
> {: .code-in}
>
> > <code-in-title>To disable the proxy, use</code-in-title>
> >
> > ```bash
> > $ git config --global --unset http.proxy
> > $ git config --global --unset https.proxy
> > ```
> {: .code-in}
{: .tip}

> <tip-title>Git Help and Manual</tip-title>
>
> Always remember that if you forget the subcommands or options of a `git` command, you can access the relevant list of options typing `git <command> -h` or access the corresponding Git manual by typing
> `git <command> --help`, e.g.:
>
> > <code-in-title>Access help </code-in-title>
> >```bash
> >$ git config -h
> >$ git config --help
> >```
> {: .code-in}
>
> While viewing the manual, remember the `:` is a prompt waiting for commands and you can press <kbd>Q</kbd> to exit the manual.
>
> More generally, you can get the list of available `git` commands and further resources of the Git manual typing:
>
> > <code-in-title>Access available commands</code-in-title>
> > ```bash
> > $ git help
> > ```
> {: .code-in}
>
{: .tip}

[git-privacy]: https://help.github.com/articles/keeping-your-email-address-private/

# Create a Repo

Once Git is configured,
we can start using it.

We will continue with the story of Sherlock who is investigating a crime and is collecting information about suspects.

![Cartoon of sherlock examining the git logo with his magnifying glass.](../../images/bash-git/sherlock_git.png)


First, let's create a directory for our work and then move into that directory:

> <code-in-title>Create a workspace</code-in-title>
> ```bash
> $ mkdir suspects
> $ cd suspects
> ```
{: .code-in}

Then we tell Git to make `suspects` a [repository](https://git-scm.com/docs/gitglossary#def_repository)
-- a place where Git can store versions of our files:

> <code-in-title>Turn our workspace into directory</code-in-title>
> ```bash
> $ git init
> ```
{: .code-in}

It is important to note that `git init` will create a repository that
includes subdirectories and their files---there is no need to create
separate repositories nested within the `suspects` repository, whether
subdirectories are present from the beginning or added later. Also, note
that the creation of the `suspects` directory and its initialization as a
repository are completely separate processes.

If we use `ls` to show the directory's contents,
it appears that nothing has changed:

> <code-in-title>Show directory content</code-in-title>
> ```bash
> $ ls
> ```
{: .code-in}

But if we add the `-a` flag to show everything,
we can see that Git has created a hidden directory within `suspects` called `.git`:

> <code-in-title>Show everything in our directory</code-in-title>
> ```bash
> $ ls -a
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> .	..	.git
> ```
{: .code-out}

Git uses this special subdirectory to store all the information about the project,
including all files and sub-directories located within the project's directory.
If we ever delete the `.git` subdirectory, we will lose the project's history.

Next, we will change the default branch to be called `main`.
This might be the default branch depending on your settings and version
of git.

> <code-in-title>Rename the branch</code-in-title>
> ```bash
> $ git checkout -b main
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> Switched to a new branch 'main'
> ```
{: .code-out}


We can check that everything is set up correctly
by asking Git to tell us the status of our project:

> <code-in-title>Check</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
>
> No commits yet
>
> nothing to commit (create/copy files and use "git add" to track)
> ```
{: .code-out}

If you are using a different version of `git`, the exact
wording of the output might be slightly different.

## Places to Create Git Repositories

 Along with tracking information about suspects (the project we have already created), Sherlock would also like to track information specific clues. So, Sherlock creates a `clues` project inside his `suspects` project with the following sequence of commands:

> <code-in-title>Create a project within a project</code-in-title>
> ```bash
> $ cd ~/Desktop   # return to Desktop directory
> $ cd suspects     # go into suspects directory, which is already a Git repository
> $ ls -a          # ensure the .git subdirectory is still present in the suspects directory
> $ mkdir clues    # make a subdirectory suspects/clues
> $ cd clues       # go into clues subdirectory
> $ git init       # make the clues subdirectory a Git repository
> $ ls -a          # ensure the .git subdirectory is present indicating we have created a new Git repository
> ```
{: .code-in}

> <question-title>Tracking in a subdirectory</question-title>
> Is the `git init` command, run inside the `clues` subdirectory, required for tracking files stored in the `clues` subdirectory?
>
> > <solution-title></solution-title>
> >
> > No. Sherlock does not need to make the `clues` subdirectory a Git repository
> > because the `suspects` repository will track all files, sub-directories, and
> > subdirectory files under the `suspects` directory.  Thus, in order to track
> > all information about clues, Sherlock only needed to add the `clues` subdirectory
> > to the `suspects` directory.
> >
> {: .solution}
{: .question}

> <tip-title>"Nested" repositories</tip-title>
> Additionally, Git repositories can interfere with each other if they are "nested":
> the outer repository will try to version-control
> the inner repository. Therefore, it's best to create each new Git
> repository in a separate directory. To be sure that there is no conflicting
> repository in the directory, check the output of `git status`. If it looks
> like the following, you are good to go to create a new repository as shown
> above:
>
> > <code-in-title>Before creating a new repository</code-in-title>
> > ```bash
> > $ git status
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> >```
> >fatal: Not a git repository (or any of the parent directories): .git
> >```
> {: .code-out}
>
{: .tip}

## Correcting `git init` Mistakes
Dr. Watson explains to Sherlock how a nested repository is redundant and may cause confusion down the road. Sherlock would like to remove the nested repository. How can Sherlock undo his last `git init` in the `clues` subdirectory?

### USE `rm` WITH CAUTION!

Removing files from a Git repository needs to be done with caution. But we have not learned yet how to tell Git to track a particular file; we will learn this in the next episode. Files that are not tracked by Git can easily be removed like any other "ordinary" files with
> <code-in-title>Remove files </code-in-title>
> ```bash
> $ rm filename
> ```
{: .code-in}

Similarly a directory can be removed using `rm -r dirname` or `rm -rf dirname`. If the files or folder being removed in this fashion are tracked by Git, then their removal becomes another change that we will need to track, as we will see later.

Git keeps all of its files in the `.git` directory. To recover from this little mistake, Sherlock can just remove the `.git` folder in the clues subdirectory by running the following command from inside the `suspects` directory:

> <code-in-title>Remove the `.git` folder</code-in-title>
> ```bash
> $ rm -rf clues/.git
> ```
{: .code-in}

But be careful! Running this command in the wrong directory will remove the entire Git history of a project you might want to keep. Therefore, always check your current directory using the command `pwd`.

# Tracking Changes

First let's make sure we're still in the right directory.
You should be in the `suspects` directory.

> <code-in-title>Check directory</code-in-title>
> ```bash
> $ cd ~/suspects
> ```
{: .code-in}

Let's create a file called `colonel.txt` that contains some notes about the colonel Smith's connection to the case.
We'll use `nano` to edit the file;
you can use whatever editor you like.
In particular, this does not have to be the `core.editor` you set globally earlier. But remember, the bash command to create or edit a new file will depend on the editor you choose (it might not be `nano`). For a refresher on text editors, check out ["Which Editor?"](https://swcarpentry.github.io/shell-novice/03-create/) in [The Unix Shell](https://swcarpentry.github.io/shell-novice/) lesson.

> <code-in-title>Edit file with nano in bash</code-in-title>
> ```bash
> $ nano colonel.txt
> ```
{: .code-in}

Type the text below into the `colonel.txt` file:

```
No alibi for the night of murder.
```


Let's first verify that the file was properly created by running the list command (`ls`):

> <code-in-title>Check new file</code-in-title>
> ```bash
> $ ls
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> colonel.txt
> ```
{: .code-out}

`colonel.txt` contains a single line, which we can see by running:

> <code-in-title>Inspect the new file</code-in-title>
> ```bash
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> ```
{: .code-out}

If we check the status of our project again,
Git tells us that it's noticed the new file:

> <code-in-title>Check</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
>
> No commits yet
>
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
>
> colonel.txt
>
> nothing added to commit but untracked files present (use "git add" to track)
> ```
{: .code-out}

The "untracked files" message means that there's a file in the directory that Git isn't keeping track of. We can tell Git to track a file using `git add`:

> <code-in-title>Track file</code-in-title>
> ```bash
> $ git add colonel.txt
> ```
{: .code-in}

and then check that the right thing happened:

> <code-in-title>Check</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
>
> No commits yet
>
> Changes to be committed:
>   (use "git rm --cached <file>..." to unstage)
>
>	new file:   colonel.txt
>
> ```
{: .code-out}

Git now knows that it's supposed to keep track of `colonel.txt`, but it hasn't recorded these changes as a commit yet. To get it to do that, we need to run one more command:

> <code-in-title>Record changes as a commit with a descriptive title</code-in-title>
> ```bash
> $ git commit -m "Start notes for colonel Smith as a suspect"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
>[main (root-commit) f22b25e] Start notes on colonel Smith as a suspect
> 1 file changed, 1 insertion(+)
> create mode 100644 colonel.txt
> ```
{: .code-out}

When we run `git commit`,
Git takes everything we have told it to save by using `git add`
and stores a copy permanently inside the special `.git` directory.
This permanent copy is called a [commit](https://git-scm.com/docs/gitglossary#def_commit)
(or [revision](https://git-scm.com/docs/gitglossary#def_revision)) and its short identifier is `f22b25e`. Your commit may have another identifier.

We use the `-m` flag (for "message")
to record a short, descriptive, and specific comment that will help us remember later on what we did and why.
If we just run `git commit` without the `-m` option,
Git will launch `nano` (or whatever other editor we configured as `core.editor`)
so that we can write a longer message.

[Good commit messages][commit-messages] start with a brief <50 characters) statement about the
changes made in the commit. Generally, the message should complete the sentence "If applied, this commit will" <commit message here>.
If you want to go into more detail, add a blank line between the summary line and your additional notes. Use this additional space to explain why you made changes and/or what their impact will be.

> <code-in-title>If we run `git status` now:</code-in-title>
>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> nothing to commit, working directory clean
> ```
{: .code-out}

it tells us everything is up to date.
If we want to know what we've done recently,
we can ask Git to show us the project's history using `git log`:

> <code-in-title>Access project's history</code-in-title>
> ```bash
> $ git log
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> commit f22b25e3233b4645dabd0d81e651fe074bd8e73b
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:   Thu Aug 22 09:51:46 2013 -0400
>
>   Start notes on colonel as a suspect
> ```
{: .code-out}

`git log` lists all commits  made to a repository in reverse chronological order.
The listing for each commit includes
the commit's full identifier
(which starts with the same characters as
the short identifier printed by the `git commit` command earlier),
the commit's author,
when it was created,
and the log message Git was given when the commit was created.

> <tip-title>Where Are My Changes?</tip-title>
>
> If we run `ls` at this point, we will still see just one file called `colonel.txt`.
> That's because Git saves information about files' history
> in the special `.git` directory mentioned earlier
> so that our filesystem doesn't become cluttered
> (and so that we can't accidentally edit or delete an old version).
{: .tip}

Now suppose Sherlock adds more information to the file.
(Again, we'll edit with `nano` and then `cat` the file to show its contents;
you may use a different editor, and don't need to `cat`.)

> <code-in-title>Edit with nano</code-in-title>
> ```bash
> $ nano colonel.txt
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> No clear motive. Seems high unlikely.
> ```
{: .code-out}

When we run `git status` now,
it tells us that a file, it already knows about, has been modified:

> <code-in-title>Find about current status</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> Changes not staged for commit:
>  (use "git add <file>..." to update what will be committed)
>  (use "git checkout -- <file>..." to discard changes in working directory)
>
>	modified:   colonel.txt
>
> no changes added to commit (use "git add" and/or "git commit -a")
> ```
{: .code-out}

The last line is the key phrase:
"no changes added to commit".
We have changed this file,
but we haven't told Git we will want to save those changes
(which we do with `git add`)
nor have we saved them (which we do with `git commit`).
So let's do that now. It is good practice to always review
our changes before saving them. We do this using `git diff`.
This shows us the differences between the current state
of the file and the most recently saved version:

> <code-in-title>Review the changes </code-in-title>
> ```bash
> $ git diff
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index df0654a..315bf3a 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1 +1,2 @@
>  No alibi for the night of murder.
> +No clear motive. Seems high unlikely.
> ```
{: .code-out}

The output is cryptic because
it is actually a series of commands for tools like editors and `patch`
telling them how to reconstruct one file given the other.
If we break it down into pieces:

1.  The first line tells us that Git is producing output similar to the Unix `diff` command
    comparing the old and new versions of the file.
2.  The second line tells exactly which versions of the file
    Git is comparing;
    `df0654a` and `315bf3a` are unique computer-generated labels for those versions.
3.  The third and fourth lines once again show the name of the file being changed.
4.  The remaining lines are the most interesting, they show us the actual differences
    and the lines on which they occur.
    In particular,
    the `+` marker in the first column shows where we added a line.

After reviewing our change, it's time to commit it:

> <code-in-title>Commit after checking changes</code-in-title>
> ```bash
> $ git commit -m "Add concerns about existence of motive for colonel"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git checkout -- <file>..." to discard changes in working directory)
>
> 	modified:   colonel.txt
>
> no changes added to commit (use "git add" and/or "git commit -a")
> ```
{: .code-out}

Whoops:
Git won't commit because we didn't use `git add` first.
Let's fix that:

> <code-in-title>First `add` then `commit`</code-in-title>
> ```bash
> $ git add colonel.txt
> $ git commit -m "Add concerns about existence of motive for colonel"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> [main 34961b1] Add concerns about existence of motive for colonel
>  1 file changed, 1 insertion(+)
> ```
{: .code-out}

Git insists that we add files to the set we want to commit
before actually committing anything. This allows us to commit our
changes in stages and capture changes in logical portions rather than
only large batches.
For example,
suppose we're adding a few citations to relevant research to our thesis.
We might want to commit those additions,
and the corresponding bibliography entries,
but *not* commit some of our work drafting the conclusion
(which we haven't finished yet).

To allow for this,
Git has a special *staging area*
where it keeps track of things that have been added to
the current [changeset](https://git-scm.com/docs/gitglossary#def_changeset)
but not yet committed.

> <tip-title>Staging Area</tip-title>
>
> If you think of Git as taking snapshots of changes over the life of a project,
> `git add` specifies *what* will go in a snapshot
> (putting things in the staging area),
> and `git commit` then *actually takes* the snapshot, and
> makes a permanent record of it (as a commit).
> If you don't have anything staged when you type `git commit`,
> Git will prompt you to use `git commit -a` or `git commit --all`,
> which is kind of like gathering *everyone* to take a group photo!
> However, it's almost always better to
> explicitly add things to the staging area, because you might
> commit changes you forgot you made. (Going back to the group photo simile,
> you might get an extra with incomplete makeup walking on
> the stage for the picture because you used `-a`!)
> Try to stage things manually,
> or you might find yourself searching for "git undo commit" more
> than you would like!
{: .tip}

![The Git Staging Area cartoon, a document is shown going into the staging area via "git add", and then into the repository via "git commit".](../../images/bash-git/git-staging-area.svg)

Let's watch as our changes to a file move from our editor
to the staging area
and into long-term storage.
First,
we'll add another line to the file:

> <code-in-title>Add and review new line with `nano` and `cat`</code-in-title>
> ```bash
> $ nano colonel.txt
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> No clear motive. Seems high unlikely.
> Fingerprints on victims glasses.
> ```
{: .code-out}

> <code-in-title>Review changes</code-in-title>
> ```bash
> $ git diff
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index 315bf3a..b36abfd 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1,2 +1,3 @@
>  No alibi for the night of murder.
>  No clear motive. Seems high unlikely.
> +Fingerprints on victims glasses.
> ```
{: .code-out}

So far, so good:
we've added one line to the end of the file
(shown with a `+` in the first column).
Now let's put that change in the staging area
and see what `git diff` reports:

> <code-in-title>See changes</code-in-title>
> ```bash
> $ git add colonel.txt
> $ git diff
> ```
{: .code-in}

But, there is no output:
as far as Git can tell,
there's no difference between what it's been asked to save permanently
and what's currently in the directory.
However,
if we do this:

> <code-in-title>See what is in the staging area</code-in-title>
> ```bash
> $ git diff --staged
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index 315bf3a..b36abfd 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1,2 +1,3 @@
>  No alibi for the night of murder.
>  No clear motive. Seems high unlikely.
> +Fingerprints on victims glasses.
> ```
{: .code-out}

it shows us the difference between
the last committed change
and what's in the staging area.
Let's save our changes:

> <code-in-title>Save changes</code-in-title>
> ```bash
> $ git commit -m "Make notes about colonel's fingerprints"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> [main 005937f] Make notes about colonel's fingerprints
>  1 file changed, 1 insertion(+)
> ```
{: .code-out}

> <code-in-title>Check new status</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> nothing to commit, working directory clean
> ```
{: .code-out}

> <code-in-title>and look at the history of what we've done so far: </code-in-title>
> ```bash
> $ git log
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> commit 005937fbe2a98fb83f0ade869025dc2636b4dad5 (HEAD -> main)
> Author: Holmes Sherlock <Holmes@tran.sylvan.ia>
> Date:   Thu Aug 22 10:14:07 2013 -0400
>
>     Make notes about colonel's fingerprints
>
> commit 34961b159c27df3b475cfe4415d94a6d1fcd064d
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:   Thu Aug 22 10:07:21 2013 -0400
>
>     Add concerns about existence of motive for colonel
>
> commit f22b25e3233b4645dabd0d81e651fe074bd8e73b
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:   Thu Aug 22 09:51:46 2013 -0400
>
>     Start notes on colonel as a suspect
> ```
{: .code-out}

> <tip-title>Word-based diffing</tip-title>
>
> Sometimes, e.g. in the case of the text documents a line-wise
> diff is too coarse. That is where the `--color-words` option of
> `git diff` comes in very useful as it highlights the changed
> words using colors.
{: .tip}

> <tip-title>Paging the Log</tip-title>
>
> When the output of `git log` is too long to fit in your screen,
> `git` uses a program to split it into pages of the size of your screen.
> When this "pager" is called, you will notice that the last line in your
> screen is a `:`, instead of your usual prompt.
>
> *   To get out of the pager, press <kbd>Q</kbd>.
> *   To move to the next page, press <kbd>Spacebar</kbd>.
> *   To search for `some_word` in all pages,
>     press <kbd></kbd>
>     and type `some_word`.
>     Navigate through matches pressing <kbd>N</kbd>.
{: .tip}

> <tip-title>Limit Log Size</tip-title>
>
> To avoid having `git log` cover your entire terminal screen, you can limit the
> number of commits that Git lists by using `-N`, where `N` is the number of
> commits that you want to view. For example, if you only want information from
> the last commit you can use:
>
> > <code-in-title>See specific number of commits</code-in-title>
> > ```bash
> > $ git log -1
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > commit 005937fbe2a98fb83f0ade869025dc2636b4dad5 (HEAD -> main)
> > Author: Sherlock Holmes <sherlock@baker.street>
> > Date:   Thu Aug 22 10:14:07 2013 -0400
> >
> >    Make notes about colonel's fingerprints
> > ```
> {: .code-out}
>
> You can also reduce the quantity of information using the
> `--oneline` option:
>
> > <code-in-title>See only basic information</code-in-title>
> > ```
> > $ git log --oneline
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > 005937f (HEAD -> main) Make notes about colonel's fingerprints
> > 34961b1 Add concerns about existence of motive for colonel
> > f22b25e Start notes on colonel as a base
> > ```
> {: .code-out}
>
> You can also combine the `--oneline` option with others. One useful
> combination adds `--graph` to display the commit history as a text-based
> graph and to indicate which commits are associated with the
> current `HEAD`, the current branch `main`, or
> [other Git references][git-references]:
>
> > <code-in-title>Extra options</code-in-title>
> > ```bash
> > $ git log --oneline --graph
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > * 005937f (HEAD -> main) Make notes about colonel's fingerprints
> > * 34961b1 Add concerns about existence of motive for colonel
> > * f22b25e Start notes on colonel as a base
> > ```
> {: .code-out}
{: .tip}

> <tip-title>Directories </tip-title>
>
> Two important facts you should know about directories in Git.
>
> 1. Git does not track directories on their own, only files within them.
>    Try it for yourself:
>
>    > <code-in-title>Test directory tracking  </code-in-title>
>    > ```bash
>    > $ mkdir mysteries
>    > $ git status
>    > $ git add mysteries
>    > $ git status
>    > ```
>    {: .code-in}
>
>    Note, our newly created empty directory `mysteries` does not appear in
>    the list of untracked files even if we explicitly add it (_via_ `git add`) to our
>    repository. This is the reason why you will sometimes see `.gitkeep` files
>    in otherwise empty directories. Unlike `.gitignore`, these files are not special
>    and their sole purpose is to populate a directory so that Git adds it to
>    the repository. In fact, you can name such files anything you like.
>
> 2. If you create a directory in your Git repository and populate it with files,
>    you can add all files in the directory at once by:
>
>     ```
>     git add <directory-with-files>
>     ```
>
>    Try it for yourself:
>    > <code-in-title>Add multiple files</code-in-title>
>    > ```bash
>    > $ touch mysteries/belgravia mysteries/reichenbachfall
>    > $ git status
>    > $ git add mysteries
>    > $ git status
>    > ```
>    {: .code-in}
>
>    Before moving on, we will commit these changes.
>
>    > <code-in-title>Commit the new changes</code-in-title>
>    > ```
>    > $ git commit -m "Add some initial thoughts on mysteries"
>    > ```
>    {: .code-in}
>
{: .tip}

To recap, when we want to add changes to our repository,
we first need to add the changed files to the staging area
(`git add`) and then commit the staged changes to the
repository (`git commit`):

![The Git Commit Workflow](../../images/bash-git/git-committing.svg)

# Let's put us to the test

> <question-title>Choosing a Commit Message</question-title>
>
> Which of the following commit messages would be most appropriate for the
> last commit made to `colonel.txt`?
>
> 1. "Changes"
> 2. "Added line 'Fingerprints on victims glasses.' to colonel.txt"
> 3. "Make notes about colonel's fingerprints"
>
> > <solution-title></solution-title>
> > Answer 1 is not descriptive enough, and the purpose of the commit is unclear;
> > and answer 2 is redundant to using "git diff" to see what changed in this commit;
> > but answer 3 is good: short, descriptive, and imperative.
> {: .solution}
{: .question }

> <question-title>Committing Changes to Git</question-title>
>
> Which command(s) below would save the changes of `myfile.txt`
> to my local Git repository?
>
> 1. ```
>    $ git commit -m "my recent changes"
>    ```
>
> 2. ```
>    $ git init myfile.txt
>    $ git commit -m "my recent changes"
>    ```
> 3. ```
>    $ git add myfile.txt
>    $ git commit -m "my recent changes"
>    ```
> 4. ```
>    $ git commit -m myfile.txt "my recent changes"
>    ```
>
> > <solution-title></solution-title>
> >
> > 1. Would only create a commit if files have already been staged.
> > 2. Would try to create a new repository.
> > 3. Is correct: first add the file to the staging area, then commit.
> > 4. Would try to commit a file "my recent changes" with the message myfile.txt.
> {: .solution}
{: .question }

> <question-title>Committing Multiple Files</question-title>
>
> The staging area can hold changes from any number of files
> that you want to commit as a single snapshot.
>
> 1. Add some text to `colonel.txt` noting your suspicion on judge Brown.
> 2. Create a new file `judge.txt` with your initial thoughts
> about the judge as a suspect.
> 3. Add changes from both files to the staging area,
> and commit those changes.
>
> > <solution-title></solution-title>
> >
> > The output below from `cat colonel.txt` reflects only content added during
> > this exercise. Your output may vary.
> >
> > First we make our changes to the `colonel.txt` and `judge.txt` files:
> > > <code-in-title>Edit `colonel.txt`</code-in-title>
> > > ```bash
> > > $ nano colonel.txt
> > > $ cat colonel.txt
> > > ```
> > {: .code-in}
> > > <code-out-title></code-out-title>
> > > ```
> > > Maybe judge Brown should also be considerable as a suspect.
> > > ```
> > {: .code-out}
> > > <code-in-title>Create and edit `judge.txt`</code-in-title>
> > > ```bash
> > > $ nano judge.txt
> > > $ cat judge.txt
> > > ```
> > {: .code-in}
> > > <code-out-title></code-out-title>
> > > ```
> > > Judge seems like a nice guy, but has a shady past.
> > > ```
> > {: .code-out}
> >
> > Now you can add both files to the staging area. We can do that in one line:
> > > <code-in-title>Add both files</code-in-title>
> > > ```bash
> > > $ git add colonel.txt judge.txt
> > > ```
> > {: .code-in}
> >
> > > <code-in-title>Or with multiple commands: </code-in-title>
> > > ```bash
> > > $ git add colonel.txt
> > > $ git add judge.txt
> > > ```
> > {: .code-in}
> >
> > Now the files are ready to commit. You can check that using `git status`. If you are ready to commit use:
> > > <code-in-title>Commit changes</code-in-title>
> > > ```bash
> > > $ git commit -m "Write plans to start a base on judge"
> > > ```
> > {: .code-in}
> > > <code-out-title></code-out-title>
> > > ```
> > > [main cc127c2]
> > >  Write plans to start a base on judge
> > >  2 files changed, 2 insertions(+)
> > >  create mode 100644 judge.txt
> > > ```
> > {: .code-out}
> {: .solution}
{: .question }

> <question-title>`bio` Repository</question-title>
>
> 1. Create a new Git repository on your computer called `bio`.
> 2. Write a three-line biography for yourself in a file called `me.txt`,
> commit your changes
> 3. Modify one line, add a fourth line
> 4. Display the differences
> between its updated state and its original state.
>
> > <solution-title></solution-title>
> >
> > If needed, move out of the `suspects` folder:
> > > <code-in-title>Change directory</code-in-title>
> > > ```bash
> > > $ cd ..
> > > ```
> > {: .code-in}
> >
> > Create a new folder called `bio` and 'move' into it:
> > > ### Create folder
> > > ```bash
> > > $ mkdir bio
> > > $ cd bio
> > > ```
> > {: .code-in}
> >
> > > <code-in-title>Initialise git:</code-in-title>
> > > ```bash
> > > $ git init
> > > ```
> > {: .code-in}
> >
> > Create your biography file `me.txt` using `nano` or another text editor.
> > Once in place, add and commit it to the repository:
> >
> > > <code-in-title>Create file and edit it</code-in-title>
> > > ```bash
> > > $ git add me.txt
> > > $ git commit -m "Add biography file"
> > > ```
> > {: .code-in}
> >
> > Modify the file as described (modify one line, add a fourth line).
> > To display the differences
> > between its updated state and its original state, use `git diff`:
> >
> > > <code-in-title>Display the differences</code-in-title>
> > > ```bash
> > > $ git diff me.txt
> > > ```
> > {: .code-in}
> {: .solution}
{: .question }

[commit-messages]: https://chris.beams.io/posts/git-commit/
[git-references]: https://git-scm.com/book/en/v2/Git-Internals-Git-References

# History Exploring

As we saw in the previous episode, we can refer to commits by their
identifiers.  You can refer to the _most recent commit_ of the working
directory by using the identifier `HEAD`.

We've been adding one line at a time to `colonel.txt`, so it's easy to track our
progress by looking, so let's do that using our `HEAD`s.  Before we start,
let's make a change to `colonel.txt`, adding yet another line.

> <code-in-title>Edit `colonel.txt`</code-in-title>
> ```bash
> $ nano colonel.txt
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> No clear motive. Seems high unlikely.
> Fingerprints on victims glasses.
> Maybe judge Brown should also be considerable as a suspect.
> ```
{: .code-out}

Now, let's see what we get.

> <code-in-title>Display the changes</code-in-title>
> ```bash
> $ git diff HEAD colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index b36abfd..0848c8d 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1,3 +1,4 @@
>  No alibi for the night of murder.
>  No clear motive. Seems high unlikely.
>  Fingerprints on victims glasses.
> +Maybe judge Brown should also be considerable as a suspect.
> ```
{: .code-out}

which is the same as what you would get if you leave out `HEAD` (try it).  The
real goodness in all this is when you can refer to previous commits.  We do
that by adding `~1`
(where "~" is "tilde", pronounced [**til**-d*uh*])
to refer to the commit one before `HEAD`.

> <code-in-title>Refer to previous commits</code-in-title>
> ```bash
> $ git diff HEAD~1 colonel.txt
> ```
{: .code-in}

If we want to see the differences between older commits we can use `git diff`
again, but with the notation `HEAD~1`, `HEAD~2`, and so on, to refer to them:

> <code-in-title>Refer to previous commits</code-in-title>
> ```bash
> $ git diff HEAD~3 colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index df0654a..b36abfd 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1 +1,4 @@
>  No alibi for the night of murder.
> +No clear motive. Seems high unlikely.
> +Fingerprints on victims glasses.
> +Maybe judge Brown should also be considerable as a suspect.
> ```
{: .code-out}

We could also use `git show` which shows us what changes we made at an older commit as
well as the commit message, rather than the _differences_ between a commit and our
working directory that we see by using `git diff`.

> <code-in-title>`git show` and `git diff` differences </code-in-title>
> ```bash
> $ git show HEAD~3 colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> commit f22b25e3233b4645dabd0d81e651fe074bd8e73b
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:


topic: data-science
tutorial: bash-variant-calling
title: Variant Calling Workflow
difficulty: intermediate
time_estimation: 50M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 3cb587173fe68d405598bccb6b92413d
last_modified: 2026-02-26T21:12:14.736612
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/bash-variant-calling/tutorial.html

content:
In this tutorial we are working with files from a long-term evolution study of an *E. coli* population (designated Ara-3). We will perform variant calling to see how the population changed over time. We care how this population changed relative to the original population, *E. coli* strain REL606. Therefore, we will align each of our samples to the *E. coli* REL606 reference genome, and see what differences exist in our reads versus the genome.

The whole proccess is designed to be run in RStudio as an Interactive Tool, directly in Galaxy.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/) lesson
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Software to use</hands-on-title>
>
> The R Console and other interactive tools like RStudio are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. This is one of those cases and, in order to do that, we will use the terminal provided by the RStudio itself.
> We go to "Tools" and pick the "Shell..." option and we are good to go. Our workspace is the left, terminal window that just opened.
>
> Fortunately, [miniconda](https://docs.conda.io/en/latest/miniconda.html) is already installed. Miniconda is a package manager that simplifies the installation processes. We can and will use it to install every essential package for our tutorial. However, it is of critical importance that we do that in an new environment within our existing base and install our packages in said environment.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ conda create -n name_of_your_env bwa samtools bcftools
> > $ conda activate name_of_your_env
> > ```
> {: .code-in}
>
>
> | Software | Version | Manual | Available for | Description |
> | -------- | ------------ | ------ | ------------- | ----------- |
> | [BWA](http://bio-bwa.sourceforge.net/) | 0.7.17 | [BWA Manual](http://bio-bwa.sourceforge.net/bwa.shtml) | Linux, MacOS | Mapping DNA sequences against reference genome. |
> | [SAMtools](http://samtools.sourceforge.net/) | 1.15.1 | [SAMtools Manual](http://www.htslib.org/doc/samtools.html) | Linux, MacOS | Utilities for manipulating alignments in the SAM format. |
> | [BCFtools](https://samtools.github.io/bcftools/) | 1.15.1 | [BCFtools manual](https://samtools.github.io/bcftools/bcftools.html) | Linux, MacOS | Utilities for variant calling and manipulating VCFs and BCFs. |
> | [IGV](http://software.broadinstitute.org/software/igv/home) | [IGV Download](https://software.broadinstitute.org/software/igv/download) | [IGV User Guide](https://software.broadinstitute.org/software/igv/UserGuide) | Linux, MacOS, Windows | Visualization and interactive exploration of large genomics datasets. |
>
{: .hands_on}

# Alignment to a reference genome

![Simplified workflow that starts with the raw sequence reads and ends up to genome alignment](../../images/bash-variant-calling/variant_calling_workflow_align.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

We perform read alignment or mapping to determine where in the genome our reads originated from. There are a number of tools to
choose from and, while there is no gold standard, there are some tools that are better suited for particular NGS analyses. We will be
using the [Burrows Wheeler Aligner (BWA)](http://bio-bwa.sourceforge.net/), which is a software package for mapping low-divergent
sequences against a large reference genome.

The alignment process consists of two steps:

1. Indexing the reference genome
2. Aligning the reads to the reference genome

# Setting up

> <hands-on-title>Download Genome</hands-on-title>
>
> First we download the reference genome for *E. coli* REL606. Although we could copy or move the file with `cp` or `mv`, most genomics workflows begin with a download step, so we will practice that here.
>
> > ###  {% icon code-in %} Input: Bash
> > ```bash
> > $ cd ~/dc_workshop
> > $ mkdir -p data/ref_genome
> > $ curl -L -o data/ref_genome/ecoli_rel606.fasta.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/017/985/GCA_000017985.1_ASM1798v1/GCA_000017985.1_ASM1798v1_genomic.fna.gz
> > $ gunzip data/ref_genome/ecoli_rel606.fasta.gz
> > ```
> {: .code-in}
>
> > <question-title></question-title>
> >
> > We saved this file as `data/ref_genome/ecoli_rel606.fasta.gz` and then decompressed it.
> > What is the real name of the genome?
> >
> > > <solution-title></solution-title>
> > >
> > > ```bash
> > > $ head data/ref_genome/ecoli_rel606.fasta
> > > ```
> > >
> > >
> > > The name of the sequence follows the `> ` character. The name is `CP000819.1 Escherichia coli B str. REL606, complete genome`.
> > > Keep this chromosome name (`CP000819.1`) in mind, as we will use it later in the lesson.
> > {: .solution}
> {: .question}
>
> We will also download a set of trimmed FASTQ files to work with that will enable us to run our variant calling workflow quite quickly.
>
> > <code-in-title>Getting easy-to-work FASTQ files</code-in-title>
> > ```bash
> > $ curl -L -o sub.tar.gz https://ndownloader.figshare.com/files/14418248
> > $ tar xvf sub.tar.gz
> > $ mv sub/ ~/dc_workshop/data/trimmed_fastq_small
> > ```
> {: .code-in}
>
> You will also need to create directories for the results that will be generated as part of this workflow. We can do this in a single line of code, because `mkdir` can accept multiple new directory names as input.
>
> > <code-in-title>Create result directories</code-in-title>
> > ```bash
> > $ mkdir -p results/sam results/bam results/bcf results/vcf
> > ```
> {: .code-in}
{: .hands_on}


> <hands-on-title>Index the reference genome</hands-on-title>
> Our first step is to index the reference genome for use by BWA. Indexing allows the aligner to quickly find potential alignment sites for query sequences in a genome, which saves time during alignment. Indexing the reference only has to be run once. The only reason you would want to create a new index is if you are working with a different reference genome or you are using a different tool for alignment.
>
> > <code-in-title>Indexing with `bwa`</code-in-title>
> > ```bash
> > $ bwa index data/ref_genome/ecoli_rel606.fasta
> > ```
> {: .code-in}
>
> While the index is created, you will see output that looks something like this:
>
> > <code-out-title></code-out-title>
> > ```
> > [bwa_index] Pack FASTA... 0.04 sec
> > [bwa_index] Construct BWT for the packed sequence...
> > [bwa_index] 1.05 seconds elapse.
> > [bwa_index] Update BWT... 0.03 sec
> > [bwa_index] Pack forward-only FASTA... 0.02 sec
> > [bwa_index] Construct SA from BWT and Occ... 0.57 sec
> > [main] Version: 0.7.17-r1188
> > [main] CMD: bwa index data/ref_genome/ecoli_rel606.fasta
> > [main] Real time: 1.765 sec; CPU: 1.715 sec
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Align reads to reference genome</hands-on-title>
>
> The alignment process consists of choosing an appropriate reference genome to map our reads against and then deciding on an
> aligner. We will use the BWA-MEM algorithm, which is the latest and is generally recommended for high-quality queries as it
> is faster and more accurate.
>
> An example of what a `bwa` command looks like is below. This command will not run, as we do not have the files `ref_genome.fa`, `input_file_R1.fastq`, or `input_file_R2.fastq`.
>
>
> ```
> $ bwa mem ref_genome.fasta input_file_R1.fastq input_file_R2.fastq > output.sam
> ```
>
>
> Have a look at the [bwa options page](http://bio-bwa.sourceforge.net/bwa.shtml). While we are running bwa with the default
> parameters here, your use case might require a change of parameters. *NOTE: Always read the manual page for any tool before using
> and make sure the options you use are appropriate for your data.*
>
> We are going to start by aligning the reads from just one of the
> samples in our dataset (`SRR2584866`). Later, we will be
> iterating this whole process on all of our sample files.
>
> > <code-in-title>Aligning with bwa</code-in-title>
> > ```bash
> > $ bwa mem data/ref_genome/ecoli_rel606.fasta data/trimmed_fastq_small/SRR2584866_1.trim.sub.fastq data/trimmed_fastq_small/SRR2584866_2.trim.sub.fastq > results/sam/SRR2584866.aligned.sam
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > [M::bwa_idx_load_from_disk] read 0 ALT contigs
> > [M::process] read 77446 sequences (10000033 bp)...
> > [M::process] read 77296 sequences (10000182 bp)...
> > [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (48, 36728, 21, 61)
> > [M::mem_pestat] analyzing insert size distribution for orientation FF...
> > [M::mem_pestat] (25, 50, 75) percentile: (420, 660, 1774)
> > [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 4482)
> > [M::mem_pestat] mean and std.dev: (784.68, 700.87)
> > [M::mem_pestat] low and high boundaries for proper pairs: (1, 5836)
> > [M::mem_pestat] analyzing insert size distribution for orientation FR...
> > ```
> {: .code-out}
{: .hands_on}

# SAM/BAM format

The [SAM file](https://genome.sph.umich.edu/wiki/SAM),
is a tab-delimited text file that contains information for each individual read and its alignment to the genome. While we do not
have time to go into detail about the features of the SAM format, the paper by
[Heng Li et al.](http://bioinformatics.oxfordjournals.org/content/25/16/2078.full) provides a lot more detail on the specification.

**The compressed binary version of SAM is called a BAM file.** We use this version to reduce size and to allow for *indexing*, which enables efficient random access of the data contained within the file.

The file begins with a **header**, which is optional. The header is used to describe the source of data, reference sequence, method of
alignment, etc., this will change depending on the aligner being used. Following the header is the **alignment section**. Each line
that follows corresponds to alignment information for a single read. Each alignment line has **11 mandatory fields** for essential
mapping information and a variable number of other fields for aligner specific information. An example entry from a SAM file is
displayed below with the different fields highlighted.

![sam file example with fields description, showing qname, flag, rname, pos, maqp, and cigar as the first fields](../../images/bash-variant-calling/sam_bam.png)

Image from [Data Wrangling and Processing for Genomics](https://datacarpentry.org/wrangling-genomics/)

![sam file example with fields description, showing seq, qual, mrnm, mpos, and size as the second set of fields.](../../images/bash-variant-calling/sam_bam3.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

> <hands-on-title>SAM to BAM</hands-on-title>
> > <code-in-title>bash</code-in-title>
> > We will convert the SAM file to BAM format using the `samtools` program with the `view` command and tell this command that the input is in SAM format (`-S`) and to output BAM format (`-b`):
> >
> > ```bash
> > $ samtools view -S -b results/sam/SRR2584866.aligned.sam > results/bam/SRR2584866.aligned.bam
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > [samopen] SAM header is present: 1 sequences.
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Sort BAM file by coordinates</hands-on-title>
>
> Next we sort the BAM file using the `sort` command from `samtools`. `-o` tells the command where to write the output.
> > <code-in-title>`sort` command</code-in-title>
> > ```bash
> > $ samtools sort -o results/bam/SRR2584866.aligned.sorted.bam results/bam/SRR2584866.aligned.bam
> > ```
> {: .code-in}
>
> Our files are pretty small, so we will not see this output. If you run the workflow with larger files, you will see something like this:
> > <code-out-title></code-out-title>
> >
> > ```
> > [bam_sort_core] merging from 2 files...
> > ```
> {: .code-out}
>
> SAM/BAM files can be sorted in multiple ways, e.g. by location of alignment on the chromosome, by read name, etc. It is important to be aware that different alignment tools will output differently sorted SAM/BAM, and different downstream tools require differently sorted alignment files as input.
>
> You can use samtools to learn more about this bam file as well.
>
> > <code-in-title>`flagstat` command</code-in-title>
> > ```bash
> > samtools flagstat results/bam/SRR2584866.aligned.sorted.bam
> > ```
> {: .code-in}
>
> This will give you the following statistics about your sorted bam file:
>
> > <code-out-title></code-out-title>
> > ```
> > 351169 + 0 in total (QC-passed reads + QC-failed reads)
> > 0 + 0 secondary
> > 1169 + 0 supplementary
> > 0 + 0 duplicates
> > 351103 + 0 mapped (99.98% : N/A)
> > 350000 + 0 paired in sequencing
> > 175000 + 0 read1
> > 175000 + 0 read2
> > 346688 + 0 properly paired (99.05% : N/A)
> > 349876 + 0 with itself and mate mapped
> > 58 + 0 singletons (0.02% : N/A)
> > 0 + 0 with mate mapped to a different chr
> > 0 + 0 with mate mapped to a different chr (mapQ> =5)
> > ```
> {: .code-out}
{: .hands_on}

# Variant calling

A variant call is a conclusion that there is a nucleotide difference vs. some reference at a given position in an individual genome
or transcriptome, often referred to as a Single Nucleotide Variant (SNV). The call is usually accompanied by an estimate of
variant frequency and some measure of confidence. Similar to other steps in this workflow, there are a number of tools available for
variant calling. In this workshop we will be using `bcftools`, but there are a few things we need to do before actually calling the
variants.

![Workflow that leads from sequence reads to "variant calling" step](../../images/bash-variant-calling/variant_calling_workflow.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

> <hands-on-title>Step 1: Calculate the read coverage of positions in the genome</hands-on-title>
>
> Do the first pass on variant calling by counting read coverage with
> [bcftools](https://samtools.github.io/bcftools/bcftools.html). We will
> use the command `mpileup`. The flag `-O b` tells bcftools to generate a
> bcf format output file, `-o` specifies where to write the output file, and `-f` flags the path to the reference genome:
> > <code-in-title>`mpileup` command</code-in-title>
> > ```bash
> > $ bcftools mpileup -O b -o results/bcf/SRR2584866_raw.bcf -f data/ref_genome/ecoli_rel606.fasta results/bam/SRR2584866.aligned.sorted.bam
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > [mpileup] 1 samples in 1 input files
> > ```
> {: .code-out}
>
> We have now generated a file with coverage information for every base.
>
{: .hands_on}

> <hands-on-title>Step 2: Detect the single nucleotide variants (SNVs</hands-on-title>
>
> Identify SNVs using bcftools `call`. We have to specify ploidy with the flag `--ploidy`, which is one for the haploid *E. coli*. `-m` allows for multiallelic and rare-variant calling, `-v` tells the program to output variant sites only (not every site in the genome), and `-o` specifies where to write the output file:
> > <code-in-title>`call` command</code-in-title>
> > ```bash
> > $ bcftools call --ploidy 1 -m -v -o results/vcf/SRR2584866_variants.vcf results/bcf/SRR2584866_raw.bcf
> > ```
> {: .code-in}
>
{: .hands_on}

> <hands-on-title>Step 3: Filter and report the SNV variants in variant calling format (VCF</hands-on-title>
>
> Filter the SNVs for the final output in VCF format, using `vcfutils.pl`:
> > <code-in-title>Filtering with `vcfutils.pl` command</code-in-title>
> > ```bash
> > $ vcfutils.pl varFilter results/vcf/SRR2584866_variants.vcf  > results/vcf/SRR2584866_final_variants.vcf
> > ```
> {: .code-in}
>
{: .hands_on}


> <hands-on-title>Explore the VCF format:</hands-on-title>
>
> > <code-in-title></code-in-title>
> > ```bash
> > $ less -S results/vcf/SRR2584866_final_variants.vcf
> > ```
> {: .code-in}
>
> You will see the header (which describes the format), the time and date the file was
> created, the version of bcftools that was used, the command line parameters used, and
> some additional information:
>
> > <code-out-title></code-out-title>
> > ```
> > ##fileformat=VCFv4.2
> > ##FILTER<ID=PASS,Description="All filters passed">
> > ##bcftoolsVersion=1.8+htslib-1.8
> > ##bcftoolsCommand=mpileup -O b -o results/bcf/SRR2584866_raw.bcf -f data/ref_genome/ecoli_rel606.fasta results/bam/SRR2584866.aligned.sorted.bam
> > ##reference=file://data/ref_genome/ecoli_rel606.fasta
> > ##contig<ID=CP000819.1,length=4629812>
> > ##ALT<ID=*,Description="Represents allele(s) other than observed.">
> > ##INFO<ID=INDEL,Number=0,Type=Flag,Description="Indicates that the variant is an INDEL.">
> > ##INFO<ID=IDV,Number=1,Type=Integer,Description="Maximum number of reads supporting an indel">
> > ##INFO<ID=IMF,Number=1,Type=Float,Description="Maximum fraction of reads supporting an indel">
> > ##INFO<ID=DP,Number=1,Type=Integer,Description="Raw read depth">
> > ##INFO<ID=VDB,Number=1,Type=Float,Description="Variant Distance Bias for filtering splice-site artefacts in RNA-seq data (bigger is better)",Version=
> > ##INFO<ID=RPB,Number=1,Type=Float,Description="Mann-Whitney U test of Read Position Bias (bigger is better)">
> > ##INFO<ID=MQB,Number=1,Type=Float,Description="Mann-Whitney U test of Mapping Quality Bias (bigger is better)">
> > ##INFO<ID=BQB,Number=1,Type=Float,Description="Mann-Whitney U test of Base Quality Bias (bigger is better)">
> > ##INFO<ID=MQSB,Number=1,Type=Float,Description="Mann-Whitney U test of Mapping Quality vs Strand Bias (bigger is better)">
> > ##INFO<ID=SGB,Number=1,Type=Float,Description="Segregation based metric.">
> > ##INFO<ID=MQ0F,Number=1,Type=Float,Description="Fraction of MQ0 reads (smaller is better)">
> > ##FORMAT<ID=PL,Number=G,Type=Integer,Description="List of Phred-scaled genotype likelihoods">
> > ##FORMAT<ID=GT,Number=1,Type=String,Description="Genotype">
> > ##INFO<ID=ICB,Number=1,Type=Float,Description="Inbreeding Coefficient Binomial test (bigger is better)">
> > ##INFO<ID=HOB,Number=1,Type=Float,Description="Bias in the number of HOMs number (smaller is better)">
> > ##INFO<ID=AC,Number=A,Type=Integer,Description="Allele count in genotypes for each ALT allele, in the same order as listed">
> > ##INFO<ID=AN,Number=1,Type=Integer,Description="Total number of alleles in called genotypes">
> > ##INFO<ID=DP4,Number=4,Type=Integer,Description="Number of high-quality ref-forward , ref-reverse, alt-forward and alt-reverse bases">
> > ##INFO<ID=MQ,Number=1,Type=Integer,Description="Average mapping quality">
> > ##bcftools_callVersion=1.8+htslib-1.8
> > ##bcftools_callCommand=call --ploidy 1 -m -v -o results/bcf/SRR2584866_variants.vcf results/bcf/SRR2584866_raw.bcf; Date=Tue Oct  9 18:48:10 2018
> > ```
> >
> > Followed by information on each of the variations observed:
> >
> > ```
> > #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  results/bam/SRR2584866.aligned.sorted.bam
> > CP000819.1      1521    .       C       T       207     .       DP=9;VDB=0.993024;SGB=-0.662043;MQSB=0.974597;MQ0F=0;AC=1;AN=1;DP4=0,0,4,5;MQ=60
> > CP000819.1      1612    .       A       G       225     .       DP=13;VDB=0.52194;SGB=-0.676189;MQSB=0.950952;MQ0F=0;AC=1;AN=1;DP4=0,0,6,5;MQ=60
> > CP000819.1      9092    .       A       G       225     .       DP=14;VDB=0.717543;SGB=-0.670168;MQSB=0.916482;MQ0F=0;AC=1;AN=1;DP4=0,0,7,3;MQ=60
> > CP000819.1      9972    .       T       G       214     .       DP=10;VDB=0.022095;SGB=-0.670168;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,2,8;MQ=60      GT:PL
> > CP000819.1      10563   .       G       A       225     .       DP=11;VDB=0.958658;SGB=-0.670168;MQSB=0.952347;MQ0F=0;AC=1;AN=1;DP4=0,0,5,5;MQ=60
> > CP000819.1      22257   .       C       T       127     .       DP=5;VDB=0.0765947;SGB=-0.590765;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,2,3;MQ=60      GT:PL
> > CP000819.1      38971   .       A       G       225     .       DP=14;VDB=0.872139;SGB=-0.680642;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,4,8;MQ=60      GT:PL
> > CP000819.1      42306   .       A       G       225     .       DP=15;VDB=0.969686;SGB=-0.686358;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,5,9;MQ=60      GT:PL
> > CP000819.1      45277   .       A       G       225     .       DP=15;VDB=0.470998;SGB=-0.680642;MQSB=0.95494;MQ0F=0;AC=1;AN=1;DP4=0,0,7,5;MQ=60
> > CP000819.1      56613   .       C       G       183     .       DP=12;VDB=0.879703;SGB=-0.676189;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,8,3;MQ=60      GT:PL
> > CP000819.1      62118   .       A       G       225     .       DP=19;VDB=0.414981;SGB=-0.691153;MQSB=0.906029;MQ0F=0;AC=1;AN=1;DP4=0,0,8,10;MQ=59
> > CP000819.1      64042   .       G       A       225     .       DP=18;VDB=0.451328;SGB=-0.689466;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,7,9;MQ=60      GT:PL
> > ```
> {: .code-out}
>
{: .hands_on}

This is a lot of information, so let's take some time to make sure we understand our output.

The first few columns represent the information we have about a predicted variation.

| column | info |
| ------- | ---------- |
| CHROM | contig location where the variation occurs |
| POS | position within the contig where the variation occurs |
| ID | a `.` until we add annotation information |
| REF | reference genotype (forward strand) |
| ALT | sample genotype (forward strand) |
| QUAL | Phred-scaled probability that the observed variant exists at this site (higher is better) |
| FILTER | a `.` if no quality filters have been applied, PASS if a filter is passed, or the name of the filters this variant failed |

In an ideal world, the information in the `QUAL` column would be all we needed to filter out bad variant calls.
However, in reality we need to filter on multiple other metrics.

The last two columns contain the genotypes and can be tricky to decode.

| column | info |
| ------- | ---------- |
| FORMAT | lists in order the metrics presented in the final column |
| results | lists the values associated with those metrics in order |

For our file, the metrics presented are GT:PL:GQ.

| metric | definition |
| ------- | ---------- |
| AD, DP | the depth per allele by sample and coverage |
| GT | the genotype for the sample at this loci. For a diploid organism, the GT field indicates the two alleles carried by the sample, encoded by a 0 for the REF allele, 1 for the first ALT allele, 2 for the second ALT allele, etc. A 0/0 means homozygous reference, 0/1 is heterozygous, and 1/1 is homozygous for the alternate allele. |
| PL | the likelihoods of the given genotypes |
| GQ | the Phred-scaled confidence for the genotype |

The Broad Institute's [VCF guide](https://www.broadinstitute.org/gatk/guide/article?id=1268) is an excellent place
to learn more about the VCF file format.

> <question-title></question-title>
>
> Use the `grep` and `wc` commands you have learned to assess how many variants are in the vcf file.
>
> > <solution-title></solution-title>
> >
> > > {% icon code-in%}
> > > ```bash
> > > $ grep -v "#" results/vcf/SRR2584866_final_variants.vcf | wc -l
> > > ```
> > {: .code-in}
> > > {% icon code-out%}
> > > ```
> > > 766
> > > ```
> > {: .code-out}
> >
> > There are 766 variants in this file.
> {: .solution}
{: .question}

> <hands-on-title>Assess the alignment (visualization) - optional step</hands-on-title>
>
> It is often instructive to look at your data in a genome browser. Visualization will allow you to get a "feel" for
> the data, as well as detecting abnormalities and problems. Also, exploring the data in such a way may give you
> ideas for further analyses.  As such, visualization tools are useful for exploratory analysis. In this lesson we
> will describe two different tools for visualization: a light-weight command-line based one and the Broad
> Institute's Integrative Genomics Viewer (IGV) which requires
> software installation and transfer of files.
>
> > <code-in-title> In order for us to visualize the alignment files, we will need to index the BAM file using `samtools`:</code-in-title>
> >
> > ```bash
> > $ samtools index results/bam/SRR2584866.aligned.sorted.bam
> > ```
> {: .code-in}
>
> ## Viewing with `tview`
>
> [Samtools](http://www.htslib.org/) implements a very simple text alignment viewer based on the GNU
> `ncurses` library, called `tview`. This alignment viewer works with short indels and shows [MAQ](http://maq.sourceforge.net/) consensus.
> It uses different colors to display mapping quality or base quality, subjected to users' choice. Samtools viewer is known to work with a 130 GB alignment swiftly. Due to its text interface, displaying alignments over network is also very fast.
>
> In order to visualize our mapped reads, we use `tview`, giving it the sorted bam file and the reference file:
>
> > <code-in-title>Visualize with `tview`</code-in-title>
> >
> > ```bash
> > $ samtools tview results/bam/SRR2584866.aligned.sorted.bam data/ref_genome/ecoli_rel606.fasta
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > 1         11        21        31        41        51        61        71        81        91        101       111       121
> > AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACTGGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATAC
> > ..................................................................................................................................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ..................N................. ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,........................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ..................N................. ,,,,,,,,,,,,,,,,,,,,,,,,,,,.............................
> > ...................................,g,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ....................................   ................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,....................................   ....................................      ,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ....................................  ,,a,,,,,,,,,,,,,,,,,,,,,,,,,,,,,     .......
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, .............................  ,,,,,,,,,,,,,,,,,g,,,,,    ,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ...........................T.......   ,,,,,,,,,,,,,,,,,,,,,,,c,          ......
> > ......................... ................................   ,g,,,,,,,,,,,,,,,,,,,      ...........................
> > ,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,,,,,,,,,,,,,,,,,,,       ..........................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,   ................................T..  ..............................   ,,,,,,
> > ...........................       ,,,,,,g,,,,,,,,,,,,,,,,,   ....................................         ,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,, ....................................  ...................................        ....
> > ....................................  ........................  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,      ....
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,   ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ........................            .................................. .............................     ....
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,   ....................................        ..........................
> > ...............................       ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ....................................
> > ...................................  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ..................................
> > .................................... ,,,,,,,,,,,,,,,,,,a,,,,,,,,,,,,,,,,,        ,,,,,,,,,,,,,,,,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ............................ ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ```
> {: .code-out}
>
> The first line of output shows the genome coordinates in our reference genome. The second line shows the reference
> genome sequence. The third line shows the consensus sequence determined from the sequence reads. A `.` indicates
> a match to the reference sequence, so we can see that the consensus from our sample matches the reference in most
> locations. That is good! If that was not the case, we should probably reconsider our choice of reference.
>
> Below the horizontal line, we can see all of the reads in our sample aligned with the reference genome. Only
> positions where the called base differs from the reference are shown. You can use the arrow keys on your keyboard
> to scroll or type `?` for a help menu. To navigate to a specific position, type `g`. A dialogue box will appear. In
> this box, type the name of the "chromosome" followed by a colon and the position of the variant you would like to view
> (e.g. for this sample, type `CP000819.1:50` to view the 50th base. Type `Ctrl^C` or `q` to exit `tview`.
>
> > <question-title></question-title>
> >
> > Visualize the alignment of the reads for our `SRR2584866` sample. What variant is present at
> > position 4377265? What is the canonical nucleotide in that position?
> >
> > > <solution-title></solution-title>
> > >
> > > ```bash
> > > $ samtools tview ~/dc_workshop/results/bam/SRR2584866.aligned.sorted.bam ~/dc_workshop/data/ref_genome/ecoli_rel606.fasta
> > > ```
> > >
> > > Then type `g`. In the dialogue box, type `CP000819.1:4377265`.
> > > `G` is the variant. `A` is canonical. This variant possibly changes the phenotype of this sample to hypermutable. It occurs
> > > in the gene *mutL*, which controls DNA mismatch repair.
> > {: .solution}
> {: .question}
>
> ## Viewing with IGV
>
> [IGV](http://www.broadinstitute.org/igv/) is a stand-alone browser, which has the advantage of being installed locally and providing fast access. Web-based genome browsers, like [Ensembl](http://www.ensembl.org/index.html) or the [UCSC browser](https://genome.ucsc.edu/), are slower, but provide more functionality. They not only allow for more polished and flexible visualization, but also provide easy access to a wealth of annotations and external data sources. This makes it straightforward to relate your data with information about repeat regions, known genes, epigenetic features or areas of cross-species conservation, to name just a few.
>
>In order to use IGV, we will need to download the files, that we are going to use as input, to our local machine. We can do this easily via the Galaxy RStudio interface, in which we were working all along. 
>We will work on the lower right panel in the RStudio, on the "Files" tab, wehre we simply need to select the desired files by checking the boxes left to their name.
>Then, we click on the option "Export", which we can find on the said panel in the "More" dropdown tab. After assigning a name for the file to be downloaded we hit "Download" and our selection are localy downloaded. 
>The files we are going to use are listed below:
>
>~~~
>~/dc_workshop/results/bam/SRR2584866.aligned.sorted.bam 
>~/dc_workshop/results/bam/SRR2584866.aligned.sorted.bam.bai 
>~/dc_workshop/data/ref_genome/ecoli_rel606.fasta 
>~/dc_workshop/results/vcf/SRR2584866_final_variants.vcf 
>~~~
>
> Next, we need to open the IGV software. If you have not done so already, you can download IGV from the [Broad Institute's software page](https://www.broadinstitute.org/software/igv/download), double-click the `.zip` file
> to unzip it, and then drag the program into your Applications folder.
>
> 1. Open IGV.
> 2. Load our reference genome file (`ecoli_rel606.fasta`) into IGV using the **"Load Genomes from File..."** option under the **"Genomes"** pull-down menu.
> 3. Load our BAM file (`SRR2584866.aligned.sorted.bam`) using the **"Load from File..."** option under the **"File"** pull-down menu.
> 4.  Do the same with our VCF file (`SRR2584866_final_variants.vcf`).
>
> Your IGV browser should look like the screenshot below:
>
> ![IGV screenshot](../../images/bash-variant-calling/igv-screenshot.png)
>
> There should be two tracks: one coresponding to our BAM file and the other for our VCF file.
>
> In the **VCF track**, each bar across the top of the plot shows the allele fraction for a single locus. The second bar shows
> the genotypes for each locus in each *sample*. We only have one sample called here, so we only see a single line. Dark blue =
> heterozygous, Cyan = homozygous variant, Grey = reference.  Filtered entries are transparent.
>
> Zoom in to inspect variants you see in your filtered VCF file to become more familiar with IGV. See how quality information
> corresponds to alignment information at those loci.
> Use [this website](http://software.broadinstitute.org/software/igv/AlignmentData) and the links therein to understand how IGV colors the alignments.
>
{: .hands_on}

> <tip-title>BWA alignment options</tip-title>
> BWA consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM. The first algorithm is designed for Illumina sequence
> reads up to 100bp, while the other two are for sequences ranging from 70bp to 1Mbp. BWA-MEM and BWA-SW share similar features such
> as long-read support and split alignment, but BWA-MEM, which is the latest, is generally recommended for high-quality queries as it
> is faster and more accurate.
{: .tip}


topic: data-science
tutorial: online-resources-gene
title: Learning about one gene across biological resources and formats
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 1cbe6da6453b2b59cfd38d3535134eca
last_modified: 2026-02-26T21:12:14.894780
zenodo_link: https://zenodo.org/record/8304465
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/online-resources-gene/tutorial.html

content:
When we do a bioinformatics analysis, e.g. RNA-seq, we might end up with a list of gene names. We then need to explore these genes. But how can we do that? What are the resources available for that? And how to navigate through them?

The aim of this tutorial is to familiarize ourselves with that, using Human opsins as an example.

Human opsins are found in the cells of your retina. Opsins catch light and begin the sequence of signals that result in vision. We will proceed by asking questions about opsins and opsin genes, and then using different bioinformatics databases and resources to answer them.

> <comment-title></comment-title>
> This tutorial is a bit atypical: we will not work in Galaxy but mostly outside of it, navigating databases and tools through their own web interfaces. The scope of this tutorial is to illustrate several sources of biological data in different file formats, and representing different information.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
{: .agenda}

# Searching Human Opsins

To seach Human Opsins, we will start by checking the [NCBI Genome Data Viewer](https://www.ncbi.nlm.nih.gov/genome/gdv). The NCBI Genome Data Viewer (GDV) ({% cite rangwala2021accessing %}) is a genome browser supporting the exploration and analysis of annotated eukaryotic genome assemblies. The GDV browser displays biological information mapped to a genome, including gene annotation, variation data, BLAST alignments, and experimental study data from the NCBI GEO and dbGaP databases. GDV release notes describe new features relating to this browser. 

> <hands-on-title>Open NCBI Genome Data Viewer</hands-on-title>
>
> 1. Open the NCBI Genome Data Viewer at [www.ncbi.nlm.nih.gov/genome/gdv](https://www.ncbi.nlm.nih.gov/genome/gdv/)
>
{: .hands-on}

The homepage includes a simple "tree of life" where the human node is highlighted because it is the default organism to search. We can change that in the *Search organisms* box but we will leave for now as we are interested in Human Opsins. 

![Genome Data Viewer home page screenshot, the word "opsin" is written in the search box and the result is previewed.](./images/GenomeDataViewerpage.png "Genome Data Viewer home page")

The panel on the right reports multiple assemblies of the genome of interest, and a map of the chromosomes in that genome. We can search for Opsins there.

> <hands-on-title>Search for OpsinsOpen NCBI Genome Data Viewer</hands-on-title>
>
> 1. Type `opsin` in the *Search in genome* box
> 2. Click on the magnifier icon or press <kbd>Enter<kbd>
>
{: .hands-on}

Below the box is now displayed a table with genes related to opsin together with their names and location, i.e. the chromosome number, as well ass the start and end position

In the list of genes related to the search term opsin, there are the rhodopsin gene (RHO), and three cone pigments, short-, medium-, and long-wavelength sensitive opsins (for blue, green, and red light detection). There are other entities, e.g. a -LCR (Locus Control region), putative genes and receptors. 

Multiple hits are on the X chromosome, one of the sex-determining chromosomes.

> <question-title></question-title>
>
> 1. How many genes have been found in Chromosome X?
> 2. How many are protein coding genes?
>
> > <solution-title></solution-title>
> >
> > 1. The hits in ChrX are: 
> > - OPSIN-LCR
> > - OPN1LW
> > - OP1MW
> > - OPN1MW2
> > - OPN1MW3
> >
> > 2. By hovering over each gene, a box open and we can click on *Details* to learn more about each gene. Then we learn that the first (OPSIN-LCR) is not protein coding but a gene regulatory region and the other are protein coding genes. So there are 4 protein coding genes related to opsins in Chromosome X. In particular, Chromosome X includes one red pigment gene (OPN1LW) and three green pigment genes (OPN1MW, OPN1MW2 and OPN1MW3 in the reference genome assembly).
> >
> {: .solution}
{: .question}

Let's now focus on one specific opsin, the gene OPN1LW. 

> <hands-on-title>Open Genome Browser for gene OPN1LW</hands-on-title>
>
> 1. Click on the blue arrow that appears in the results table when you hover your mouse on the OPN1LW row
>
{: .hands-on}

You should have landed in [this page](https://www.ncbi.nlm.nih.gov/genome/gdv/browser/genome/?id=GCF_000001405.40), that is the genome view of gene OPN1LW. 

![Genome Data Viewer of gene OPN1LW, screenshot of the two main panels of the viewer, with chromosomes on the left and the feature viewer on the right.](./images/GenomeDataViewerofgeneOPN1LW.png "Genome Data Viewer of gene OPN1LW")

There is a lot of information in this page, let's focus on one section at the time. 

1. The Genome Data Viewer, on the top, tells us that we are looking at the data from the organism `Homo sapiens`, assembly `GRCh38.p14` and in particular at `Chr X` (Chromosome X). Each of these information has a unique ID.
2. The entire Chromosome is represented directly below, and the positions along the short (`p`) and long (`q`) arms are highlihgted.
3. Below, a blue box highlights that we are now focusing on the Region corresponding to the Gene `OPN1LW`. 

    There are multiple ways to interact with the viewer below. Try for example to hover with the mouse on the dots representing exons in the blue box. 

4. In the graph below, the gene requence is a green line with the exons (protein coding fragments) represented by green rectangles. 

    Hover with the mouse on the green line corresponding to `NM_020061.6` (our gene of interest) to get more detailed information.

    > <question-title></question-title>
    >
    > 1. What is the location of the OPN1LW segment?
    > 2. What is the length of the OPN1LW segment?
    > 3. What are introns and exons?
    > 4. How many exons and introns are in the OPN1LW gene?
    > 5. What is the total length of the coding region?
    > 6. What is the distribution between coding and non coding regions? What does that mean in term of biology?
    > 7. What is the lenght of the protein in number of amino acids?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. From 154,144,243 to 154,159,032
    > > 2. 1,4790 nucleotides, found at *Span on 14790 nt, nucleotides)*
    > > 3. Eukaryotic genes are often interrupted by non-coding regions called intervening sequences or introns. The coding regions are called exons. 
    > > 4. From this diagram, you can see that the OPN1LW gene consists of 6 exons and 5 introns, and that the introns are far larger than the exons. 
    > > 5. The CDS length is 1,095 nucleotides. 
    > > 6. Of the 14790 nt in the gene, only 1095 nt code for protein, which means that less than 8% of the base pairs contain the code. When this gene is expressed in cells in the human retina, an RNA copy of the entire gene is synthesized. Then the intron regions are cut out, and the exon regions joined together to produce the mature mRNA (a process called splicing). which will be translated by ribosomes as they make the red opsin protein. In this case, 92% of the initial RNA transcript is tossed out, leaving the pure protein code. 
    > > 7. The length of the resulting protein is 364 aa, amino acids.
    > {: .solution}
    {: .question}

But what is the sequence of this gene? There are multiple ways to retrieve this information, we will go through what we think is one of the most intituitive. 

> <hands-on-title>Open Genome Browser for gene OPN1LW</hands-on-title>
>
> 1. Click on the {% icon tool %} *Tools* section on the top right of the box showing the gene
> 2. Click on *Sequence Text View*
{: .hands-on}

This panel reports the DNA sequence of the introns (in green), as well as the one of the exons (in pink, including the translated protein sequence below). 

![Screenshot of the sequence view of the NHI resource, text is highlighted in different colors.](./images/SequenceTextView.png "Sequence Text View")

This sequence box is not showing the entire gene at the moment, but a subsequence of it. You can move upstream and downstream the genetic code with the arrows *Prev Page* and *Next Page*, or start from a specific position with the *Go To Position* button. We suggest to start with the start of the coding part of the gene, which as we learned earlier is at position 154,144,243. 

> <hands-on-title>Go to a specific position in Sequence View</hands-on-title>
>
> 1. Click on *Go To Position*
> 2. Type on `154144243`
> 
>    We have to remove the commas to validate the value
{: .hands-on}

The sequence highlighted in purple here signals a regulatory region. 

> <question-title></question-title>
>
> 1. What is the first amino acid of the resulting protein product?
> 2. What is the last one?
> 3. Can you keep a note of the first three and last three AAs of this protein?
>
> > <solution-title></solution-title>
> >
> > 1. The correspondent protein starts with Methionine, M (they all do).
> > 2. The last AA of the last exon (found in the 2nd page) is Alanine (A). After that, the stop codon TGA comes, which is not translated into an AA.
> > 3. The first three AAs are: M,A,Q; the last three: S,P,A.
> >
> {: .solution}
{: .question}

We can now close the *Sequence View*. 

From this resource, we can also get files, in different format, describing the gene. They are available from the *Download* section. 

1. *Download FASTA* will allow us to download the simplest file format to represent the nucleotide sequence of all the visible range of the genome (longer than the gene only).
2. *Download GenBank flat file* will allow us to access the annotation avaible on this page (and beyond) in a flat text format.
3. *Download Track Data* allows us to inpect two of the file formats we presented in the slides: the GFF (GFF3) and BED formats. If you change the tracks, each one may or may not be available. 

# Finding more information about our gene

Let's now get an overview of the information we have (in the literature) about our gene, using the NCBI resources

> <hands-on-title>Go to a specific position in Sequence View</hands-on-title>
>
> 1. Open the NCBI search at [www.ncbi.nlm.nih.gov/search](https://www.ncbi.nlm.nih.gov/search/)
> 2. Type `OPN1LW` in the *Search NCBI* search box
> 
{: .hands-on}

![Screenshot of the NIH result page, with cards named Literature, Genes, Proteins, genomes, Clinical and PubChem](./images/NIHresults.png "Results when searching for `OPN1LW` on NCBI").

## Literature

Let's start with the literature and in particular *PubMed* or *PubMed Central* results

> <details-title>What's the difference between PubMed and PubMed Central? </details-title>
>
> PubMed is a biomedical literature database which contains the abstracts of publications in the database. 
>
> PubMed Central is a full text repository, which contains the full text of publications in the database.
>
> While the exact number of hits may vary in time from the screenshot above, any gene name should have more hits in PubMed Central (searched in the full texts of publications) than in PubMed (searched only in the abstracts).  
>
{: .details}

> <hands-on-title>Open PubMed</hands-on-title>
>
> 1. Click on *PubMed* in the *Literature* box
> 
{: .hands-on}

You have entered PubMed, a free database of scientific literature, to the results of a complete search for articles directly associated with this gene locus. 

By clicking on the title of each article, you can see abstracts of the article. If you are on a university campus where there is online access to specific journals, you might also see links to full articles. PubMed is your entry point to a wide variety of scientific literature in the life sciences. On the left side of any PubMed page, you will find links to a description of the database, help, and tutorials on searching. 

> <question-title></question-title>
>
> 1. Can you guess which type of conditions are associated to this gene?
>
> > <solution-title></solution-title>
> >
> > 1. We will answer this question later
> >
> {: .solution}
{: .question}

> <hands-on-title>Back to NCBI Search page</hands-on-title>
>
> 1. Go back the [NCBI Search page](https://www.ncbi.nlm.nih.gov/search/all/?term=OPN1LW)
> 
{: .hands-on}

## Clinical

Let's now focus on the *Clinical* box, and specially on *OMIM*. OMIM, the Online Mendeliam Inheritance in Man (and woman!), is a catalog of human genes and genetic disorders. 

> <hands-on-title>Open OMIM</hands-on-title>
>
> 1. Click on *OMIM* in the *Clinical* box
> 
{: .hands-on}

Each OMIM entry is a genetic disorder (here mostly types of colorblindness) associated with mutations in this gene. 

> <hands-on-title>Read as much as your interest dictates</hands-on-title>
>
> 1. Follow links to get more information about each entry
> 
{: .hands-on}

> <comment-title>Read as much as your interest dictates</comment-title>
>
> For more information about OMIM itself, click the OMIM logo at the top of the page. Through OMIM, a wealth of information is available for countless genes in the human genome, and all information is backed up by references to the latest research articles.
> 
{: .comment}

How do variations in the gene affect the protein product, and its functions? Let's go back to the NIH page and investigate access the list of Single Nucleotide Polymorphisms (SNPs) that were detected by genetics studies in the gene. 

> <hands-on-title>Open dbSNP</hands-on-title>
>
> 1. Go back the [NCBI Search page](https://www.ncbi.nlm.nih.gov/search/all/?term=OPN1LW)
> 2. Click on *dbSNP* in the *Clinical* box
> 
{: .hands-on}

![Screenshot of the dbSNPs page about gene OPN1LW. Three main panels, the one on the left to filter the search based on tags, the central showing results, the right for a more detailed and programmatic search.](./images/dbSNPs.png "dbSNP in OPN1LW")

> <question-title></question-title>
>
> 1. What is the clinical significance of the rs5986963 and rs5986964 (first 2 variants listed at the time of creation of this tutorial)?
> 2. What is the functional consequence of rs104894912?
> 3. What is the functional consequence of rs104894913?
>
> > <solution-title></solution-title>
> >
> > 1. The Clinical significance is `benign` so it seems that they have no effect on the final protein product
> > 2. rs104894912 mutation leads to a `stop_gained` variant, which truncates the resulting protein too early and is therefore `pathogenic`
> > 3. rs104894913 mutation leads to a `missense_variant`, also `pathogenic`.
> >
> {: .solution}
{: .question}

Let's investigate about more on the rs104894913 variant

> <hands-on-title>Learn more about a variant dbSNP</hands-on-title>
>
> 1. Click on `rs104894913` to open its [dedicated page](https://www.ncbi.nlm.nih.gov/snp/rs104894913)
> 2. Click on *Clinical Significance* 
>
>    > <question-title></question-title>
>    > 
>    > What type of condition is associated with the rs104894913 variant? 
>    > 
>    > > <solution-title></solution-title>
>    > >
>    > > The name of the associated disease is "Protan defect". A quick internet search with your search engine will clarify that this is a type of color blindness.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Click on the *Variant details*
>
>    > <question-title></question-title>
>    > 
>    > 1. Which substitution is associated with this variant?
>    > 2. What is impact of this subtitution in term of codon and amino acid?
>    > 3. At which position of the protein is this substitution?
>    > 
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The substitution `NC_000023.10:g.153424319G>A` corresponds to change from a Guanine (G) to an Adenine (A)
>    > > 2. This substitution change the codon `GGG`, a Glycine, into `GAG`, a Glutathione
>    > > 3. `p.Gly338Glu` means that the substitution is at position 338 of the protein.
>    > {: .solution}
>    {: .question}
{: .hands-on}

What does this mean this substitution for the protein? Let's have a deeper look at this protein. 

## Protein

> <hands-on-title>Open Protein</hands-on-title>
>
> 1. Go back the [NCBI Search page](https://www.ncbi.nlm.nih.gov/search/all/?term=OPN1LW)
> 2. Click on *Protein* in the *Proteins* box
> 3. Click on `OPN1LW – opsin 1, long wave sensitive` in the box on top
> 
{: .hands-on}

![Screenshot of the Opsin 1 NIH protein page, two main panels. The one on the left reporting information about the gene, the one on the right is a table of content and a series of links to other resources](./images/Opsin1NIH.png "Opsin 1 NIH protein page")

This page presents once again some data that we are familiar with (e.g. distribution of the exons along the gene sequence). 

> <hands-on-title>Download the protein sequences</hands-on-title>
>
> 1. Click on *Download Datasets*
> 2. Select 
>    - `Gene Sequences (FASTA)`
>    - `Transcript sequences (FASTA)`
>    - `Protein sequences (FASTA)`
> 3. Click on *Download* button
> 4. Open the downloaded ZIP file
{: .hands-on}

> <question-title></question-title>
>
> 1. What does the folder contain? 
> 2. Do you think they implemented good data practices?
>
> > <solution-title></solution-title>
> >
> > 1. The folder includes
> >    - a folder `ncbi_datasets` with different subfolders in it leadig some data files (multiple formats),
> >    - a `README.md` (a Markdown file), which is designed to "travel" together with the data and explain how was the data retreived, what is the structure of the data containing subfolder, and where to find extensive documentation. 
> > 2. It is definitely a good data management practice to guide users (not only your collaborators, but also yourself in the not-so-far future, when you will forget where does that file in your Downloads folder come from) to the data source and the data structure.
> >
> {: .solution}
{: .question}

# Searching by sequence

What could we do with these sequences that we just downloaded? Let's assume that we just sequenced the transcripts that we isolated through an experiment - so we know the sequence of our entity of interest, but don't know what it is. What we need to do in this case is to search the entire database of sequences known to science and match our unknown entity with an entry that has some annotation. Let's do it. 

> <hands-on-title>Search the protein sequence against all protein sequences</hands-on-title>
>
> 1. Open (with the simplest text editor you have installed) the `protein.faa` file that you just downloaded.
> 2. Copy its contents
> 3. Open BLAST [blast.ncbi.nlm.nih.gov](https://blast.ncbi.nlm.nih.gov/Blast.cgi)
> 4. Click on the `Protein BLAST, protein > protein`
> 
>     We will indeed use a protein sequence to search against a database of proteins
>
> 5. Paste the protein sequence into the big text box
> 6. Check the rest of parameters
> 7. Click the blue button `BLAST`
{: .hands-on}

This phase will take some time, there is afterall some server somewhere that is comparing the entirety of known sequences to your target.  When the search is complete, the result should look similar to the one below: 

![Screenshot of BLAST results, one big header on top and the results listed as a table on bottom](./images/BLASTresults.png "BLAST results")

> <hands-on-title>Graphic Summary of the protein sequences</hands-on-title>
>
> 1. Click on the tab *Graphic Summary*
{: .hands-on}

We access a box containing lots of colored lines. Each line represents a hit from your blast search. If you click on a red line, the narrow box just above the box gives a brief description of the hit. 

> <hands-on-title>Descriptions of the protein sequences</hands-on-title>
>
> 1. Click on the tab *Descriptions*
{: .hands-on}

> <question-title></question-title>
>
> 1. What is the first hit? Is it expected?
> 2. What are the other hits? For which organisms?
>
> > <solution-title></solution-title>
> >
> > 1. The first hit is our red opsin. That's encouraging, because the best match should be to the query sequence itself, and you got this sequence from that gene entry. 
> > 2. Other hits are other opsins. They include entries from other primates (e.g. `Pan troglogytes`). 
> >
> >
> {: .solution}
{: .question}

The hits are for our red opsin in human but also other opsins in other primates. We could want that, for example if we wanted to use this data to build a phylogenetic tree. If we instead are pretty sure that our sequence of interest is human, we could also have filtered the search only in human sequences.

> <hands-on-title>Filter a BLAST Search</hands-on-title>
>
> 1. Click on *Edit Search*
> 2. Type `Homo sapiens` in *Organism* field
> 3. Click the blue button `BLAST`
{: .hands-on}

With this new search, we find the other opsins (green, blue, rod-cell pigment) in the list. Other hits have lower numbers of matching residues. If you click on any of the colored lines in the *Graphic Summary*, you'll open more information about that hit, and you can see how much similarity each one has to the red opsin, our original query sequence. As you go down the list, each succeeding sequence has less in common with red opsin. Each sequence is shown in comparison with red opsin in what is called a pairwise sequence alignment. Later, you'll make multiple sequence alignments from which you can discern relationships among genes.

> <details-title>More details on BLAST scores</details-title>
>
> The displays contain two prominent measures of the significance of the hit:
>
> 1. the BLAST Score - lableled Score (bits)
>
>    The BLAST Score indicates the quality of the best alignment between the query sequence and the found sequence (hit). The higher the score, the better the alignment. Scores are reduced by mismatches and gaps in the best alignment. Calculation of the score is complex, involving a substitution matrix, which is a table that assigns a score to each pair of residues aligned. The most widely used matrix for protein alignment is known as BLOSUM62.
>
> 2. the Expectation Value (labeled Expect or E)
>
>     The expectation value E of a hit tells whether the hit is likely be result from chance likeness between hit and query, or from common ancestry of hit and query. () 
>
>     > <comment-title>Filter a BLAST Search</comment-title>
>     > 
>     > If E is smaller than $$10\mathrm{e}{-100}$$, it is sometimes given as 0.0.
>     {: .comment}
>
>     The expectation value is the number of hits you would expect to occur purely by chance if you searched for your sequence in a random genome the size of the human genome. 
>
>      $$E = 25$$ means that you could expect to find 25 matches in a genome of this size, purely by chance. So a hit with $$E = 25$$ is probably a chance match, and does not imply that the hit sequence shares common ancestry with your search sequence. 
>
>      Expectation values of around 0.1 may or may not be biologically significant (other tests would be needed to decide).
>
>      But very small values of E mean that the hit is biologically significant. The correspondence between your search sequence and this hit must arise from common ancestry of the sequences, because the odds are are simply too low that the match could arise by chance. For example, $$E = 10\mathrm{e}{-18}$$ for a hit in the human genome means that you would expect only one chance match in one billion billion different genomes the same size of the human genome.
>
>      The reason we believe that we all come from common ancestors is that massive sequence similarity in all organisms is simply too unlikely to be a chance occurrence. Any family of similar sequences across many organisms must have evolved from a common sequence in a remote ancestor.
>
{: .details}

> <hands-on-title>Dowloading </hands-on-title>
>
> 1. Click on *Descriptions* tab
> 2. Click at any sequence hit
> 3. Click on *Download*
> 4. Select `FASTA (aligned sequences)`
{: .hands-on}

It will download a new, slightly different, type of file: an aligned FASTA. If you want, explore it before the next section. 

While in the previous sections of this tutorial we extensively used the web interfaces of the tools (genomic viewers, quick literature scanning, reading annotations, etc.), this BLAST search is an example of a step that you could fully automate with Galaxy. 

> <hands-on-title> Similarity search with BLAST in Galaxy </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. Import the protein sequence via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/protein.faa
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 1. {% tool [NCBI BLAST+ blastp](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2) %} with the following parameters:
>    - *"Protein query sequence(s)"*: `protein.faa`
>    - *"Subject database/sequences"*: `Locally installed BLAST database`
>    - *"Protein BLAST database"*: `SwissProt`
>
>      To search against only annotated sequences in UniProt, we need select the latest release of `SwissProt`
>
>    - *"Set expectation value cutoff"*: `0.001`
>    - *"Output format"*: `Tabular (extended 25 columns)` 
>
{: .hands_on}

> <question-title></question-title>
>
> Do you think we are looking at exactly the same results as our original search for `opsin` in [www.ncbi.nlm.nih.gov/genome/gdv](https://www.ncbi.nlm.nih.gov/genome/gdv/)? Why?
>
> > <solution-title></solution-title>
> >
> > The results might be similar, but there are definitely some differences. Indeed, not only a text search is different than a sequence search in terms of method, but also this second round we started from the sequence of one specific opsin, so one branch of the entire protein family tree. Some of the family members are more similar between each other, so this type of search looks at the whole family from a quite biased perspective. 
> >
> {: .solution}
{: .question}

# More information about our protein

So far, we explored this information about opsins:
- how to know which proteins of a certain type exist in a genome,
- how to know where they are along the genome,
- how to get more information about a gene of interest,
- how to download their sequences in different formats,
- how to use these files to perform a similarity search.

You might be curious about how to know more about the proteins they code for, now. We have already collected some information (e.g. diseases associated), but in the next steps we will cross it with data about the protein structure, localisation, interactors, functions, etc.

The portal to visit to obtain all information about a protein is [UniProt](https://www.uniprot.org/). We can search it using a text search, or the gene or protein name. Let's go for our usual `OPN1LW` keyword. 

> <hands-on-title>Searching on UniProt</hands-on-title>
>
> 1. Open  [UniProt](https://www.uniprot.org/)
> 2. Type `OPN1LW` in the search bar
> 3. Select the card view
{: .hands-on}

The first hit should be `P04000 · OPSR_HUMAN`. Before opening the page, two things to notice: 

1. The name of the protein `OPSR_HUMAN` is different than the gene name, as well as their IDs are. 
2. This entry has a golden star, which means that was manually annotated and curated. 

> <hands-on-title>Open a result on UniProt</hands-on-title>
>
> 1. Click on `P04000 · OPSR_HUMAN`
{: .hands-on}

![Screenshot of the UniProt entry page header](./images/UniProt.png "UniProt page")

This is a long page with a lot of information, we designed an [entire tutorial]({% link topics/data-science/tutorials/online-resources-protein/tutorial.md %}) to go through it.


