# tutorials

topic: data-science
tutorial: python-multiprocessing
title: Python - Multiprocessing
difficulty: advanced
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: c9f92599d9dd26f461427d60eb2936a3
last_modified: 2026-02-26T21:12:14.903902
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-multiprocessing/tutorial.html

content:
This tutorial will introduce you to the basics of Threads and Processes in
Python and how you can use them to parallelise your code.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Threads vs Processes

For many languages, threads can be extremely efficient, as they are rather light weight and don't require many resources to create new threads. "Threads are cheap". However Python has a major limitation with the {GIL}. Only one thread can be executing at once, whether it's the main thread, or one of the others you've spawned. So we look to alternative concurrency mechanisms like processes for sharing the load across multiple CPU cores.

For **threads**, if you are doing "lightweight" work like fetching data from the web where a lot of time is spent waiting for the server to respond, but very little computational work within Python is required, then threads are a great fit!

However if you are doing computationally intensive work (imagine calculating prime numbers or similar complex tasks) then each thread will be contending for CPU time. Python is a single processes and can only have one thread running at a time due to the {GIL}. So it will switch between multiple threads and try and make progress on each, but it cannot execute them truly simultaneously. Here we need to switch to processes.

**Processes** are relatively heavier weight as they essentially start a new python process, before executing the individual function. This is part of the reason for a "Pool", to amortise the expensive cost of setting up processes, before allowing them to do a lot of work quickly. Here each process can be using it's own CPU core fully, and thus make more sense for computationally expensive tasks.


Let's dive straight into an example: here we're using the `multiprocessing` library which uses processes and is relatively simple to understand:

```python
from multiprocessing import Pool
```

Importantly we should define a *pure* function, i.e. a function that only works on the inputs available to it, and doesn't use or affect global state (i.e.  without side effects). (printing is ok.)

```python
def f(x):
    return x * x
```

We spawn a process Pool, with 5 processes And then use the convenient map
function to send inputs multiply to the specified function `f`

```python
with Pool(5) as p:
    print(p.map(f, range(10)))
```

This will create 5 new python processes, and they will begin to consume tasks from the queue as quickly as they can. As soon as a process has finished handling one task, it will begin work on the next task.

## Pools & Paralellism

Let's do another example, the classic "print time", to see how the threads are actually executing across 4 processes.

```python
import time

def g(x):
    print(time.time())
    time.sleep(1)

with Pool(4) as p:
    print(p.map(g, range(12)))
```

> <question-title></question-title>
> What did you see here?
>
> Was it clear and easy to read?
> > <solution-title></solution-title>
> > It prints four timestamps immediately, all around the same time. Then one second later it prints 4 more, and one second later, a final four.
> > Our Pool of 4 processes processes the function the maximal amount of times possible concurrently. Once each of those functions returns, then it can move on to processing the next tasks.
> > This is precisely the situation of e.g. 4 queues at the grocery; as soon as one customer is processed, they immediately begin on the next, until no more remain.
> >
> > You might also see a situation where the numbers are interleaved in a completely unreadable way. Here they write immediately, but there is no synchronisation or limiting on who can write at one time, and the result is a mess of interleaved print statements.
> {: .solution}
{: .question}

## When to parallelise

Some guidelines:

- When you can isolate a pure function
- That do not require shared state, nor ordering
- That do not modify global state (pure!)
- That take a significant amount of time, relative to the time it takes to rewrite your code to support parallelising.

Some common examples of this are slow tasks like requesting data from multiple websites, e.g. using the `requests` library. Or doing some computationally expensive calculation. The last point however is especially important, as it is not always trivial to rearchitect your code to handle being spread across multiple threads or processes.

Let's parallelize this program which fetches some metadata from multiple Galaxy servers:

```python
import requests
import time

servers = [
    "https://usegalaxy.org",
    "https://usegalaxy.org.au",
    "https://usegalaxy.eu",
    "https://usegalaxy.fr",
    "https://usegalaxy.be",
]

data = {}
start = time.time()
for url in servers:
    print(url)
    try:
        response = requests.get(url + "/api/version", timeout=2).json()
        data[url] = response['version_major']
    except requests.exceptions.ConnectTimeout:
        data[url] = None
    except requests.exceptions.ReadTimeout:
        data[url] = None


# How long did it take to execute
print(time.time() - start)

for k, v in data.items():
    print(k, v)
```

If we look at this, we can see one hot spot in the code, where it's quite slow, requesting data from a remote server. If we want to speed this up we'll need to isolate it into a pure function. Here we can see a possibility for a function that requests the data, with the input of the server url, and output of the version.

```python
def fetch_version(server_url):
    try:
        response = requests.get(server_url + "/api/version", timeout=2).json()
        return response['version_major']
    except requests.exceptions.ConnectTimeout:
        return None
    except requests.exceptions.ReadTimeout:
        return None
```

This now lacks side effects (like modifying the `data` object), and can be used in a map statement.

```python
start = time.time()
with Pool(4) as p:
    versions = p.map(fetch_version, servers)
    data = dict(zip(servers, versions))
print(time.time() - start)

for k, v in data.items():
    print(k, v)
```

Same result, and now this is a lot more efficient!

## Sizing your Pool

This depends largely on profilling your code or knowing the performance characteristics of it. In the above example, there is very little computation work executed as part of this function, it's essentially all network I/O, no CPU or memory usage.

As such, we can probably set our pool size to be very large, a multiple of our systems' processor count. It will spawn many processes that do very little.

If however this were a more complicated task (e.g. calculating a large number, machine learning, etc.), then we might wish to set our pool size to the number of CPUs - 1, as each process will potentially consume it's CPU allocation completely, and we wish to have some left over for the managing program and any other work going on, on the system.

> <question-title></question-title>
> Try changing the pool size and see what effect it has on runtime. Start from a value of 1, and go up to 5, the number of servers in our list.
> > <solution-title></solution-title>
> > Given the small sample size (5 servers), and the variability of response times (in local testing between 3-8 seconds for the single-pool version), you can see varying results but *generally* it should decrease as the pool size increases. However, sometimes you will see the Pool=5 version take the same or longer than Pool=4.
> >
> > Especially in the case that 1 server (or 1 request one time) dominates the request time, this can "hide" the improvements as the others complete quickly on the remaining N-1 processes.
> {: .solution}
{: .question}

```python
results = []
for i in range(1, 6):
    start = time.time()
    with Pool(i) as p:
        versions = p.map(fetch_version, servers)
        data = dict(zip(servers, versions))
    duration = time.time() - start
    results.append(duration)

# Plot it
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.plot(range(1, 6), results)
ax.set(ylabel='time (s)', xlabel='Pool Size', title='Pool size vs runtime')
ax.grid()
# Uncomment if your notebook cannot display images inline.
# fig.savefig("pool-vs-runtime.png")
plt.show()
```

You might see a result similar to the following:

![graph of the above, showing a decrease in runtime from 2.6 seconds to approximately 1.1 seconds as the pool size increases from 1 to 5. As the pool size increases from 1-2 there is a large improvement, but as it increases from 4-5 the improvement is very small, on the order of milliseconds.](../../images/pool-vs-runtime.png)


## Threads

Let's convert our previous example from processes to threads, as processes aren't strictly necessary for such a light weight use case as fetching data from the internet where you're blocking on network rather than CPU resources.

```python
import multiprocessing.pool
import requests
import time

data = {}
start = time.time()

pool = multiprocessing.pool.ThreadPool(processes=4)
results = pool.map(fetch_version, servers, chunksize=1)
data = {k: v for (k, v) in zip(servers, results)}
pool.close()

print(time.time() - start)

for k, v in data.items():
    print(k, v)
```

This is a bit more complicated to write, but again, if you're not blocking on CPU resources, then this is potentially approximately as effecient as thread pools.

> <question-title></question-title>
> What result did you get? Was it slower, faster, or about the same as processes?
{: .question}


topic: data-science
tutorial: cli-bashcrawl
title: CLI Educational Game - Bashcrawl
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["game", "bash"]
content_hash: d1b8d76110b12d84dd6086c8088c1761
last_modified: 2026-02-26T21:12:14.737273
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-bashcrawl/tutorial.html

content:
This is not a tutorial like most GTN content but a fun exercise for you to play around and learn a bit about the command line, and hopefully re-inforce the skills you covered in Basic and Advanced CLI skills.

> <hands-on-title>Open a Jupyter Terminal</hands-on-title>
> To best follow this tutorial, you should open a terminal.
>
> Here are some instructions for how to do this on various environments.
>
> **Jupyter** on UseGalaxy.\* and **MyBinder.org**
>
> 1. Use the File → New → Terminal menu to launch a terminal.
>
>    ![screenshot of jupyterlab showing the File menu expanded to show new and terminal option.](../../images/carpentries-cli/mybinder-launch.png)
>
> 2. Disable "Simple" mode in the bottom left hand corner, if it activated.
>
>    ![screenshot of jupyterlab showing a toggle labelled simple](../../images/carpentries-cli/mybinder-simple.png)
>
> 3. Drag one of the terminal or notebook tabs to the side to have the training materials and terminal side-by-side
>
>    ![screenshot of jupyterlab with notebook and terminal side-by-side.](../../images/carpentries-cli/mybinder-final.png)
>
> **CoCalc**
>
> 1. Use the *Split View* functionality of cocalc to split your view into two portions.
>
>    ![screenshot of cocalc button to split views](../../images/carpentries-cli/cocalc-split.png)
>
> 2. Change the view of one panel to a terminal
>
>    ![screenshot of cocalc swapping view port to that of a terminal](../../images/carpentries-cli/cocalc-terminal.png)
>
{: .hands_on}

## Setup

```bash
git clone https://gitlab.com/slackermedia/bashcrawl
cd bashcrawl/entrance
# Run: cat scroll
```

Have fun, and good luck!


topic: data-science
tutorial: data-manipulation-olympics-jq
title: Data Manipulation Olympics - JQ
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa", "json"]
content_hash: e4ef880573819b91e6ee25f2405a3e0b
last_modified: 2026-02-26T21:12:14.738827
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics-jq/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->

Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

If you've opened this tutorial via the {% icon level %} icon in Galaxy (top menu bar), you can click on the tool names in the last column to quickly open them in Galaxy and start using them on your own!

Operation                | Description                                                         | Galaxy Tool
------------------------ | ------------------------------------                                | ----------------
Compute an expression    | Over each row, add it as a new column                               | `.key1 * .key2`
Concatenate datasets     | one after the other                                                 | `cat *.json | jq`
Convert format           | Change the file format                                              | Often xsv, other tools.
Counting                 | Count the number of objects in an array                             | `length`
Cut Columns              | By header name                                                      | `.key`
Filter                   | Remove objects based on values                                      | `select(.key == "value")`
Find and Replace         | in a specific column                                                | `gsub("(?<a>...)", "xyz\\(.a)abc")`
Group on a column        | And perform simple operations (count, mean, min, max etc)           | `group_by(.key1, -.key2)`
Join two Datasets        | side by side on a specified field                                   | `[JOIN(INDEX(input[]; .right); .[]; .left; add)]`
Paste                    | Two files side by side                                              | `-s '[.[][]]'`
Remove Beginning         | Good for removing header lines                                      | `last`, `index`
Select First lines       | Good for finding top 10s or saving header lines                     | `first`, `index`
Sort on a column         | Change the order of the rows based on values in one or more columns | `sort`, `sort_by(.key)`
Split file               | Based on values of a column                                         | n/a (requires bash/awk/etc.)
Unique                   | Remove duplicate rows                                               | `unique`

In this tutorial, these functions are explained in more detail, and we provide some exercises for you to practice.

# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"]({% link topics/introduction/tutorials/data-manipulation-olympics/images/cover.jpg %})


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!


# Download Data

Before we can do any manipulation, we will need some data. Let's download our table with Olympics results now.

We can use `jq` just by itself to pretty print the results:

```bash
cat olympics.json | jq -S . | head -n 20
```

> <question-title></question-title>
>
> 1. What is the format of the file?
> 2. How is it structured?
> 3. How many lines are in the file? (Hint: `wc`)
> 4. How many columns?
>
> > <solution-title></solution-title>
> >
> > 1. It is a JSON file, or a Javascript Object Notation
> > 2. It is structured as an array (we can tell by the leading `[`) with objects representing an athlete's participation in an event. If an athlete competes in multiple events, there is an entry for each event.
> > 3. `1`. The entire JSON file is on a single line, which makes it rather unpleasant to access via normal text editors. However it contains 234,522 records (`cat olympics.json | jq -c '.[]' | wc -l`)
> > 4. There are 17 columns in this file. There are multiple ways to find this answer:
> >    - Count the fields manually
> >    - Advanced JQ: `cat olympics.json| jq '.[0] | keys | length'`
> {: .solution}
{: .question}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.json` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.json` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.

# File Format Conversion

The file we uploaded is a `.json` file. This stands for *JavaScript Object Notation*. This means that this is a file containing arrays (`[]`) and objects (`{}`). 

First we need to understand a bit about file structure, and how JQ can convert back and forth between different structures:

The `.` operator is the identity operator, it gives you back exactly the same structure

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.'
```

You can use `.[]` to loop over the items in an array:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.[]'
```

And supply a number to access an individual array item:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.[0]'
```

And `.<key>` to access a specific key

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.[].a'
```

But that's not all! Sometimes you may want the resulting output to be back in an array, rather than by line:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '[.[].a]'
```

By wrapping the entire expression in `[]` we say we want the output of that statement in an array. We can actually do the same to get each entry as a new object:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '{"results": .[].a}'
```

Or an object with an array:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '{"results": [.[].a]}'
```

Or an array of objects:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '[{"results": .[].a}]'
```

We'll need to use these reshaping abilities as we go, to get data in the right shape to be worked with. Sometimes you'll want an array of objects, or objects with items in their keys.

## Pipes

Lastly, pipes are incredibly important in JQ, we'll use them to combine multiple filters, or make our pipelines more readable:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.'
```

Here we pipe the entire incoming object (JQ operates on streams of JSON objects, so, there can be multiple), to the identity operator again but this time with `[]` to loop over every object in that item. (Really this is the same as `.[]` but we use a pipe for demonstration.)

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '. | .[]'
```

We can then pass that stream of individual items (no longer a single JSON object) on to another filter, one extracting an `.a`:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '. | .[] | .a'
```


# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

In `jq` we can use the `sort` or `sort_by` function. From the documentation:

```bash
echo '[8,3,null,6]' | jq '.sort'
```

`sort_by` however lets us sort multiple entries in an array, by a child key:

```bash
echo '[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]' | jq '.sort_by(.foo)'
echo '[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]' | jq '.sort_by(.bar)'
```

So let's sort our file in chronological order, based on the year of the Olympic games:

> <question-title></question-title>
>
> 1. Which key contains the year?
> 2. Do we want ascending or descending order if we want the oldest games at the top?
>
> > <solution-title></solution-title>
> >
> > 1. `.year`
> > 2. The file should be sorted in ascending (increasing) order
> >
> {: .solution}
{: .question}

```bash
cat olympics.json | jq -c 'sort_by(.year) | .[]'
```

> <tip-title>What does `.[]` do?</tip-title>
> We use this to first take the identity operator, taking exactly the array that was passed to us, and then `[]` lets us loop over the array, returning a stream of multiple individual objects. The `-c` flag lets us see that more easily, with one object per line.
{: .tip}

If we wanted to do it in reverse, we could just use the `reverse` function, or just use `-` prefixed to our sort factor.

```bash
cat olympics.json | jq -c 'sort_by(.year) | reverse | .[]'
cat olympics.json | jq -c 'sort_by(-.year) | .[]'
```

> <question-title></question-title>
>
> 1. Write a JQ command to access only the first entry.
> 2. Which athlete is listed at the top of the file now?
>
> > <solution-title></solution-title>
> > 1. We can simply access the 0th element:
> >    ```bash
> >    cat olympics.json | jq -c 'sort_by(.year) | .[0]'
> >    ```
> > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
> >
> {: .solution}
{: .question}

**Rename** the resulting file to somthing meaningful (e.g. `olympics-chronological.json`)

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, an then within each year, we sort again alphabetically by name. The `sort_by` function can do this too!

We will sort the file in chronological order based on the year of the Olympic games

```bash
cat olympics.json | jq -c 'sort_by(.year, .name) | .[]'
```

> <question-title></question-title>
>
> Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>
> > <solution-title></solution-title>
> >
> > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
> >
> {: .solution}
{: .question}


## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> > <br>
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <tip-title>Removing null values</tip-title>
> > This will be covered more during the Filtering section, but for now simply use this filter:
> > ```bash
> > cat olympics.json | jq '[.[] | select(.height != null)] | ...your sort_by here...'
> > ```
> {: .tip}
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We can use `.height`, and because we want the tallest on top, we will need to sort in *descending* (decreasing) order. Unfortunately you might discover there are null values.
> > 2. Rerun the same query as step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Lyton Mphande from Seol is the shortest at 127 cm.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > 1. `jq -c '[.[] | select(.height != null)] | sort_by(-.height) | .[0]' < olympics.json`
> > 2. `jq -c '[.[] | select(.height != null)] | sort_by(.height) | .[0]' < olympics.json`
> > 3. `jq -c '[.[] | select(.height != null)] | sort_by(-.year, -.height) | .[0]' < olympics.json`
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).


We will filter the file to show only winter Olympics
Look at the `olympics.json` file and answer the following questions

> <question-title></question-title>
>
> 1. Which key contains this information?
> 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>
> > <solution-title></solution-title>
> >
> > 1. `.season`
> > 2. The values can be `Summer` or `Winter` (`jq -c '[.[].season] | sort | unique' < olympics.json`)
> >
> {: .solution}
{: .question}

We'll be using the `select()` filter to select entries matching specific conditions:

> <question-title></question-title>
>
> 1. How would you write the expressions for the following conditions:
>    1. `enrolled` must be 'Yes'
>    2. `age` must be smaller than 75
>    3. `height` cannot be null
>    4. `birthplace` cannot be empty
>
> 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    How would you write expressions for the following filtering conditions:
>    1. `height` is larger than 200 or smaller than 160
>    2. `height` is larger than 200 and smaller than 210
>
> > <solution-title></solution-title>
> >
> > 1. The answers are:
> >    1. `select(.enrolled == "Yes")`
> >    2. `select(.age < 75)`
> >    3. `select(.height != null)``
> >    4. `select(.birthplace != "")`
> >
> > 2. The answers are:
> >    1. `select(.height > 200 or .height < 160)`
> >    2. `select(.height > 200 and .height < 210)`
> >
> {: .solution}
{: .question}



Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics. Make sure it is contained in an array, in case we want to do further sorting.

```bash
jq -c '[.[] | select(.season == "Winter")]' < olympics.json > winter.json
```

> <question-title></question-title>
>
> How many lines are in this file? (Hint: use the `length` filter.)
>
> > <solution-title></solution-title>
> >
> > 44,680
> >
> {: .solution}
{: .question}


**Repeat** the step for the Summer Olympics

```bash
jq -c '[.[] | select(.season == "Summer")]' < olympics.json > summer.json
```


> <question-title></question-title>
>
> 1. How many lines do you expect in the this file?
> 2. How many lines are in this file? Were you right?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Use the `length` filter, assuming your data is in an array, per the original prompt.
> > 2. Be careful to consider whether these counts include the header line of the file or not
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > 1. The original file has 234,522 lines, and the Winter Olympics had 44,680 lines. So we would expect 234,522 - 44,680 = 189,842 rows of data.
> > It is always useful to take a moment to think about the expected outcome, this makes it easier to spot mistakes and will save you time in the long run.
> >
> {: .solution}
{: .question}

## Exercises

Ok, time to train! let's see if you can use the `select` filter to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm? (inclusive)
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> > - You may want to `[ ... query ...] | length` to get a nice length
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `select(.medal == "Gold")`)
> >  2. 24,633  (Expression: `select(.medal == "Gold" or .medal == "Silver" or .medal == "Bronze")`, or `select(.medal != null)`)
> >  3. 131     (Expression: `select(.medal == "Gold" and .year == 2018)` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `select(.medal != null and .height >= 170 and .height <=180)`)
> >  5. 2,333   (Expression: `select(.medal != null and (.height < 160 or .height > 190))` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count.

Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)

We'll need to use the `group_by` filter which takes a key, and then emits arrays with objects with those matching keys:

```bash
echo '[{"foo":1, "bar":10}, {"foo":3, "bar":100}, {"foo":1, "bar":1}]' | jq 'group_by(.foo)
```

So let's try that with ours:

```bash
jq '. | group_by(.year)' < olympics.json
```

We now have a structure like

```
[
    [
        {"year" 1896, ...},
        {"year" 1896, ...},
        {"year" 1896, ...}
    ],
    [
        {"year" 1900, ...},
        {"year" 1900, ...},
        {"year" 1900, ...}
    ],
    ...
]
```

We'll re-structure to one "year group" per line:

```bash
jq '. | group_by(.year) | .[]' < olympics.json
```

So we can again use the `length` function to determine how many items are in each via

```bash
jq '. | group_by(.year) | .[] | [. | length]' < olympics.json
```

And all that's left is to include the game name:

```bash
jq '. | group_by(.year) | .[] | [. | length, .[0].games]' < olympics.json
```

But the output format is not so nice, so let's introduce the `@tsv` filter to turn the resulting data into a table:

```bash
jq '. | group_by(.year) | .[] | [. | length, .[0].games] | @tsv' < olympics.json
```

Where did the " and `\t` come from? We don't want those! That is a result of the resulting data being in a JSON string, so we can tell jq to use `raw` mode with `-r`

```bash
jq -r '. | group_by(.year) | .[] | [. | length, .[0].games] | @tsv' < olympics.json
```

> <question-title></question-title>
>
> 1. How many different Olympic games are in our file?
> 2. Which Olympic games had the most participations? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>
> > <solution-title></solution-title>
> >
> > 1. 52 games (`[. | group_by(.games) | .[] | [. | length, .[0].games]] | length`)
> >
> >    The resulting file looks something like:
> >
> >     ```
> >     615	1896 Summer Olympics
> >     2503	1900 Summer Olympics
> >     2643	1904 Summer Olympics
> >     3213	1908 Summer Olympics
> >     4610	1912 Summer Olympics
> >     3448	1920 Summer Olympics
> >     5242	1924 Summer Olympics
> >     358	1924 Winter Olympics
> >     4493	1928 Summer Olympics
> >     ...
> >     ```
> >
> > 2. 1996 Summer Olympics. (10501 participations)
> >
> {: .solution}
{: .question}

You may have guessed that like `sort_by`, that we could have selected multiple columns in the `group_by` step. This lets us count on combinations of columns.

Let's try counting the number of men and women in each olympic games.

```bash
jq '. | group_by(.games, .sex) | .[] | [. | length, .[0].games, .[0].sex] | @tsv' -r < olympics.json
```


> <question-title></question-title>
>
> You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>
> 1. How many women were in the first Olympic games?
>
> 2. Which Olympic games had the most women participants?
>
> > <solution-title></solution-title>
> >
> > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice).
> >    The file looks something like this:
> >    ```
> >    2	F	1896 Summer Olympics
> >    43	F	1900 Summer Olympics
> >    17	F	1904 Summer Olympics
> >    55	F	1908 Summer Olympics
> >    97	F	1912 Summer Olympics
> >    132	F	1920 Summer Olympics
> >    269	F	1924 Summer Olympics
> >    ```
> >
> > 2. 2020 Summer Olympics (4652)
> >
> {: .solution}
{: .question}


Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting tool above, we would get a results file for each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

We can use the `unique` filter in our pipeline to discover this. First let's do our group by and iterate over each resulting group:

```bash
jq '. | group_by(.games) | .[]' < olympics.json
```

For fun this time let's organise the output as an object. We know one value we want is the games:

```bash
jq '. | group_by(.games) | .[] | {"games": .[0].games}' < olympics.json
```

But the other we want is the list of sports. We can use our `.[]` to loop over every sport item in that group

```bash
jq '. | group_by(.games) | .[] | {"games": .[0].games, "sports": [.[].sport] }' < olympics.json
```

We're almost there! We just need to unique this list and determine it's length:

```bash
jq '. | group_by(.games) | .[] | {"games": .[0].games, "sports": [.[].sport] | unique | length }' < olympics.json
```

Or if you preferred as a table. But here it seems we can wrap it in `()` to make the parsing clearer:

```bash
jq '. | group_by(.games) | .[] | [.[0].games, ([.[].sport] | unique | length)] | @tsv' -r < olympics.json
```

> <question-title></question-title>
>
> 2. How many sport were in the first Olympics? How many in the latest?
> 3. Which Olympics had the most different sports?
>
> > <solution-title></solution-title>
> >
> > 2. 10 and 38.
> > 3. The 2020 Summer Olympics had the most different sports (38)
> >
> {: .solution}
{: .question}

Save the output as something descriptive.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting instances of a key, we can use `group_by(.team)` and then loop over that to print out the length, and the team name of each of those items.
> > 2. This is basically the same question as "how many women" participated, try modifying that query.
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations (`cat olympics.json | jq '. | group_by(.team) | .[] |  [. | length, .[0].team] | @tsv' -r`)
> >  2. 15 and 250. (`cat olympics.json | jq '. | group_by(.games) | .[] | [.[0].games, ([.[].team] | unique | length)] | @tsv' -r`)
> >
> {: .solution}
{: .question}

# Computing

Sometimes we want to use the data in our column to compute a new value, and add that to the table. For instance, for our dataset we could caluclate athtletes BMI (using height and weight columns), or their age at time of participation (from year of birth and year of the Olymics). By adding these computed values as a new colum to our datset, we can more easily query the dataset for these values. We can do these types of operations using the `+=` operation on a given object.

As an example, let's calculate the age of each athlete at the time of participation, and add this as a new column to our dataset.

First let's structure our query to loop over every object in the array:

```bash
jq '.[]' < olympics.json
```

Then let's try adding a static value to it:

```bash
jq '.[] | . += {"marco": "polo"}' < olympics.json
```

Do you see the resulting key in the new file? Ok, let's replace it with a computed value:

```bash
jq '.[] | . += {"age": .year - .birth_year}' < olympics.json
```

This errors, but JQ is a bit sensitive to syntax, so we can add some `()` to make things more explicit

```bash
jq '.[] | . += {"age": (.year - .birth_year)}' < olympics.json
```

Ah and then we discover null values!

```bash
jq '.[] | select(.birth_year != null) | . += {"age": (.year - .birth_year)}' < olympics.json
```

And there we go! But this is structured as a stream of objects, maybe we want it again as an array `[]` to ensure it remains in the same shape as our input dataset.

```bash
jq '[.[] | select(.birth_year != null) | . += {"age": (.year - .birth_year)}]' < olympics.json > olympics-with-age.json
```

> <question-title></question-title>
>
> 1. How old was Arnaud Boetsch during his Olympic tennis participation?
>
> > <solution-title></solution-title>
> >
> > 2. Arnaud Boetsch is listed on the first two lines, who turned 27 the year of their Olympics.
> >
> {: .solution}
{: .question}

This was a simple computation, but much more complex mathematical expressions can be computed with this tool. See the help section at the bottom of the tool for a list of all supported operations. In the exercise below, we will compute the BMI for each athlete as an example.


## Exercises

BMI stands for Body Mass Index, is a metric to provide a very crude measure of how healthy your weight is. The formula to compute BMI is:

$$ BMI = weight / (height^2) $$

(with weight in kilograms and height in meters).

> <tip-title>BMI is problematic</tip-title>
> [Adolphe Quetelet](https://en.wikipedia.org/wiki/Adolphe_Quetelet), who invented [BMI](https://en.wikipedia.org/wiki/Body_mass_index), was not a doctor of medicine, instead he was a Belgian astronomer, mathematician, statistician, and sociologist. His data consisted probably entirely of cisgender, white, european men and women {% cite Eknoyan_2007 %}.
> In his defense, he did argue it should not be used at individual levels, however it came to be used as such due to simplicity. (See [Wikipedia's History section](https://en.wikipedia.org/wiki/Body_mass_index#History)). BMI is a poor measure of health, especially for populations with high muscle content, and cannot be simply re-used as-is with anyone other than Anglo Saxons {% cite Caleyachetty_2021 %}, [Wikipedia:Limitations](https://en.wikipedia.org/wiki/Body_mass_index#Limitations).
>
> However as it is an easy to calculate metric, we include the calculation here.
{:.tip}


Let's compute this data for all athletes and add it as a new column!

> <question-title noprefix>Exercise: Calculating BMI</question-title>
>
> 1. How would you express this calculation in jq?
>    - Remember that our height is in cm, and the formula expects height in meters
>    - And that we have null values we must filter out
>
> 2. What is the BMI for Arnaud Boetsch?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - division is `/` and multiplication is ` * ` .
> > - Generally we cannot use `^`
> > - Parentheses may be required.
> > - use `select(.value != null)` to remove nulls.
> > - If that isn't eough, you can check that the type is a number with `(.value|type) == "number"` to ensure it really is a number and not e.g. a string.
> > - remember to wrap everything in `[...]` to retain the data shape
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> > 1. other variations are possible:
> >    ```bash
> >    jq '[.[] | select((.height|type) == "number" and (.weight|type) == "number") | . + {"bmi": (.weight / (.height/100 * .height/100))}]' < olympics.json > olympics-bmi.json
> >    ```
> > 2. 22.69
> >    ```bash
> >    jq '.[] | select(.name=="Arnaud Boetsch") | .bmi' < olympics-bmi.json
> >    ```
> {: .solution}
{: .question}


# Find and Replace

Often you may need to change the contents of a file a bit to fit the expectations of an analysis tool. For instance, our file uses `NA` for missing values, but other conventions included leaving the cell empty, or using `NaN` (not a number) instead. Or, when working with chromosomal data, you may need to add or remove the `chr` prefix from a column before using it as input to a certain tool. In such situations, we can find all occurrences of a certain pattern in our file, and replace it with another value.

If we want to perform such a replacement on a single column in our data, we can use the `gsub` function.

A few of the basics of regular expression, plus some links to further resources are given in the box below:

{% snippet faqs/galaxy/analysis_regular_expressions.md %}

Let's start with a simple example:
Our file uses a mix of `Athina` and `Athens` to indicate the Capital City of Greece in the `city` column.
Let's standardize this by replacing occurrences of `Athina` with `Athens`.

Let's start by filtering out the old spelling:

```bash
jq '.[] | select(.city == "Athina")' < olympics.json
```

We can access just the .city and try out our gsub next:

```bash
jq '.[] | select(.city == "Athina") | .city | gsub("Athina"; "Athens")' < olympics.json
```

Looks good! Let's use the technique we learned in [Computing](#computing) to update the key with a new key.

```bash
jq '.[] | select(.city == "Athina") | . += {"city": (.city | gsub("Athina"; "Athens"))}' < olympics.json
```

Also looks good, ok, that's it!

```bash
jq '[.[] | . += {"city": (.city | gsub("Athina"; "Athens"))}]' < olympics.json
```


Look at the file before and after. Athlete 7 (Patrick Chila) near the top of the `olympics.tsv` file, had a value of Athina in the city column. Verify that it has been changed to Athens.

This was rather simple example, so let's try a few more examples with slightly more complex expressions.


## Exercises

You may have noticed that our file has a lot of missing data. Especially for the earlier years, things like height, weight and birthday of athletes was not registered, or simply not known. In some columns you see these missing values have been replaced with an `NA` (not available) value. In other columns (for example birth place), the cells have simply been left empty.

Different tools may expect different ways of handling missing data. So you may have to change your missing data from empty to `NA`, `NaN`, or something else, between analysis steps

> <hands-on-title>Fill empty cells</hands-on-title>
>
> We will now replace empty cells in the `birth_place` column, to use `null` instead.
> Read the [documentation for PCRE and gsub](https://stedolan.github.io/jq/manual/#RegularexpressionsPCRE)
>
> > <question-title></question-title>
> >
> > 1. Should we use gsub?
> > 2. What is the expression for a line with just whitespace?
> >
> > > <solution-title noprefix>Answer</solution-title>
> > >
> > > 1. Yes, definitely
> > > 2. `^ *$` indicates an empty line, potentially with spaces. (`^` indicated the beginning, zero or more spaces, and `$` the end). The value in the column is treated as the line, since we are looking only in 1 column.
> > >
> > {: .solution}
> >
> > > <solution-title noprefix>Hints</solution-title>
> > > Note that you cannot use `gsub` to replace something with null. Instead you'll need to use `(if ... then "value" else "other" end)`.
> > > You might want to use the gsub instead to ensure lines with single blank characters, and empty lines, all map to the same null result.
> > {: .solution}
> >
> > > <solution-title noprefix>Full Solution</solution-title>
> > > other variations are possible:
> > > ```bash
> > > cat olympics.json | jq '.[] | (if (.birth_place|gsub("^ *$"; "")) == "" then null else .birth_place end)'
> > > ```
> > {: .solution}
> {: .question}
>
{: .hands_on}

Let's do another example, this one using capture groups.

Look at the `birth_day` column. It has values in a format like `12 December`. Suppose we have a tool that expects this data to be in the reverse format, `December 12`. The file is too big to change this manually in every column. But with regular expression tools we can make this replacement easily

We will now change the format in birthday column from `day month` to `month day`, as our boss is American and requested the silly format.

Read the [documentation for PCRE and gsub](https://stedolan.github.io/jq/manual/#RegularexpressionsPCRE)
- Read the text, what settings do you think we need to use?
- Read the Regular expressions 101 FAQ below

  {% snippet faqs/galaxy/analysis_regular_expressions.md %}

> <question-title></question-title>
>
> 1. How do we match on the birthday format? How strict/exact shoule we be here?
> 2. How do we captures both the day and the month?
> 3. How do we refer to the values we captured (for the replacement value)
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Birthday is one or more digits, followed by a space, followed by one or more letters.
> > 2. Remember that you can capture values using parentheses `(..)`
> > 3. We can name our capturing groups for easier referencing, by providing a name. `(?<day>...)` can be later referred to as `\(.day)`.
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > 1. There are multiple solutions here, depending on how strict you want to be
> >    - `\d+ ([a-zA-Z]+)` (not strict, would also match on `142 Septober`
> >    - `[0-9]{1,2} (January|February|March|April|May|June|July|August|September|October|November|December)` (this is much more strict, only matches on 1 or 2 digits, followed by a space, followed by one of the months. But this would still match on 42 April, and may miss it if one month names didn't start with a capital letter)
> >    - `[123]?[0-9] [a-zA-Z]+` this will only allow dates below 40
> >
> >    There are different ways to express this, and there is no one perfect solution. If you know your data is clean, and you do not have to worry about values like 42 Septober,
> >    then you can be less strict. If your data is less clean, you may be more worried about capturing things that aren't valid birthdays and might want to be a bit stricter.
> >    It all depends on your data and use case.
> >
> > 2. `(?<day>[\d]{1,2}) (?<mon>[a-zA-Z]+)` captures both the day and the month
> > 3. We can use `\(.day)` to refer to the day we captured, and `\(.mon)` for the month in our replacement.
> >
> > If you'd like to be fancier than just using `gsub`, you can also try the other capturing functions. E.g. `capture` returns an object with the named matches, which you can add back directly in to the original object:
> >
> > ```bash
> > jq '.[] | . += (.birth_day | capture("(?<birth_day>[0-9]+) (?<birth_month>[A-Z][a-z]+)\\s*"))' < olympics.json
> > ```
> >
> {: .solution}
{: .question}


# Removing Columns

We can remove columns from a table using either `del(.key)` if we just want to remove a few values, or otherwise by selecting, if we just want to keep a few values.

Suppose we want to simplify our file a bit. All we want is file with 4 columns: athlete name, sport, olympic games, and medals.

We can most easily do this by returning an array:

```bash
jq '.[] | [.name, .sport, .games, .medal]' < olympics.json
```

But perhaps we want something a bit more structured:

```bash
jq '.[] | {"name": .name, "sport": .sport, "games": .games, "medal": .medal}' < olympics.json
```

And of course we'll keep our list shape:

```bash
jq '[.[] | {"name": .name, "sport": .sport, "games": .games, "medal": .medal}]' < olympics.json > subset.json
```

Notice that during this step, we also changed the order of the keys. This tool can also be used to re-arrange keys, if you supply all key names but in a different order.

## Exercises

> <question-title noprefix>Exercise: Removing Columns</question-title>
>
> 1. Create a file that is similar to `olympics.json`, but without the first column (athlete_id column)
> 2. Which of the two methods would be easier to use and when? (this can be a personal preference, but think about whether you would subtract or add)
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Try the `del(.key)` function
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > 1. `jq '[.[] | del(.athlete_id)]' < olympics.json`
> > 2. The `del()` function is significantly easier if you are just removing keys, but may be cumbersome if you have a lot of key and only want a few. In that case it may be faster to type those out instead.
> {: .solution}
{: .question}


# Unique

Sometimes, in the course of our data manipulations, we may end up with a file that has duplicate values. In order to filter these out, we can use the `unique` filter.

Let's say we would like to create a list of all unique athletes (id and name).

First we will cut just the `athlete_id` and `name` columns from our dataset

```bash
jq '.[] | {"name": .name, "athlete_id": .athlete_id}' < olympics.json
```

> <question-title></question-title>
>
> 1. Do you see duplication? Why is that?
>
> > <solution-title></solution-title>
> >
> > 1. Yes. For all athletes who participated more than once, the row will be identical.
> >
> {: .solution}
{: .question}

Now let's remove those duplicates. For this filter to work, it needs to be passed an array, not individual values in a stream:

```bash
jq '[.[] | {"name": .name, "athlete_id": .athlete_id}] | unique' < olympics.json
```

> <question-title></question-title>
>
> How many unique athletes do we have?
>
> > <solution-title></solution-title>
> > 94,733
> >
> > ```bash
> > jq '[.[] | {"name": .name, "athlete_id": .athlete_id}] | unique | length' < olympics.json
> > ```
> >
> {: .solution}
{: .question}


# Joining Files

This file contains a lot of information, but we may want to add more information. For example if we had a file with information about each country (population, capital city, etc), we could join that information with our Olympics data, to get a single file with all information in every row.

For example, if we would like to be able to group by continent, to e.g. count athletes, medals etc per continent, we will have to add a `continent` column to our file. To do this we would need a second file that maps each country to the corresponding continent. This is what we will do in the next hands-on section.

We obtained country information data from [DataHub](https://datahub.io/core/country-codes). More information about this file can be found in the description there. It has 56 columns with a wide variety of data about each country (from country codes, to capital city, languages spoken, etc)

Download

```
{{page.zenodo_link}}/files/country-information.json
```

> <question-title></question-title>
>
> 1. How many keys does this file have?
> 2. Which keys(s) in this file are the same as in the `olympics.tsv` file?
>
> > <solution-title></solution-title>
> >
> > 1. The country information file has 56 columns (see [DataHub](https://datahub.io/core/country-codes) for more details).
> > 2. Both files have a `NOC` column with the 3-letter country code (`NOC` stands for National Olympic Committee). However, one is lowercase.
> {: .solution}
{: .question}

We would now like to take our Olympics dataset as the basis, and add columns to every row of this file with some information about the country. In order to join, we will need to have one column that is shared between the two files, on which we can match. The `NOC` column is perfect for this because it is a defined standard. Both files also contain a column with the country name in it, which is also a possible candidate to use for joining, but because it is less standardised, it is safer to use the NOC column. For example, if one file uses "Netherlands", while the other uses "The Netherlands" to indicate the same country, the joining will fail for these rows. So always make sure the columns you join on are compatible!

We can use the join and index commands, per [SQL Style Operators](https://stedolan.github.io/jq/manual/#SQL-StyleOperators) documentation and [this stack overflow answer](https://stackoverflow.com/a/72609410)

```bash
jq '[JOIN(INDEX(input[]; .NOC); .[]; .noc; add)]' olympics.json country-information.json
```

> <question-title></question-title>
>
> 1. What do you expect the output to look like? Were you right?
> 2. How many columns are in the resulting file? What about the NOC column?
> 3. What is a possible downside to this approach?
>
> > <solution-title></solution-title>
> >
> > 1. All the columns from the country information file are added to the end of each row of our olympics dataset
> > 2. Our olympics datset had 17 columns, the country information file has 56 columns. Therefore we have 17+56=73 columns columns in our resulting file. This also means the NOC column
> >    we joined on appears twice in our output.
> > 3. There is a lot of data duplication in this file now. The exact same country information is added to every line of every athlete from a certain country.
> >    This means much larger file size.
> >    If you do not need all these columns, it could save you a lot of space to remove unneeded columns from the `country-information.json` file, before joining.
> >
> {: .solution}
{: .question}

# Concatenating

Concatenation of two files simple means putting the contents of the two files together, one after the other. Our dataset was created in 2021, but since then we've had another Olympic event, the 2022 Winter Olympics in Beijing. If we have the same data for this latest Olympics, we could simply add the rows from the 2022 games to our current file with data, in order to create a single file with all data from 1896 to 2022.

First, let's get this data for the 2022 Olympics

```
wget {{page.zenodo_link}}/files/olympics_2022.json
```

View the new dataset, does it have the same structure as our original `olympics.json` file?

> <question-title></question-title>
>
> 1. Does the new file have the same structure?
> 2. Can we simply add the lines of the new files to the end of our existing olympics dataset?
>
> > <solution-title></solution-title>
> >
> > 1. Yes, this file has all the same columns, in the same order, so concatenation should be relatively straightforward.
> > 2. If we simply put the contents of this file after our existing dataset, we will have a second header line in the middle of our data rows. It is best to remove the header line from the second dataset after we have verified it is compatible. This way we will only add the real data rows to our dataset.
> >
> {: .solution}
{: .question}

Since this new dataset has the exact same structure (number and order of columns), we can simple add the lines from this file to the end of our existing `olympics.json` file.
For this, we'll need to use a new argument to `jq`, namely `-s` or `--slurp` which allows us to access all files as if they were one big indexed array.

```bash
cat olympics.json olympics_2022.json | jq -s '.[0] + .[1]' > olympics-combined.json
```

Alternatively we could loop over every array in the parent, and combine them. This would support more than two files:

```bash
cat olympics.json olympics_2022.json | jq -s '[.[][]]' > olympics-combined.json
```

Now this only works so simply because our two datasets had the same structure; two arrays. If your data comes from different sources, you may have to do some additional data manipulation before you can concatenate, e.g. to make sure the columns match, or how each file deals with missing data (empty cells, `NA`, `NaN` or something else).


# Splitting Files

This dataset contains a lot of data, we might want to split the data into one file per Olympic games, or have one file for all the winter games, and another file for all the summer games. In these situtations, where we want to create new, smaller files, based on the values in a column, we can use the tool.

> <tip-title>Not generically possible with only JQ</tip-title>
> It is not generically possible to e.g. split a file by value, with only jq. We need to use other tools, e.g. for loops, in addition to accomplish this.
{: .tip}

We would like to create a separate file for each Olympic event. First let's get a list of every event.

```bash
jq '[.[] | .games] | unique[]' -r < olympics.json
```

Easily done. Now we'll construct a loop over these results

```bash
jq '[.[] | .games] | unique[]' -r < olympics.json | while read -r game; do
    echo "Game: $game";
done
```

Here we can then use this to create a new file per the game identifier:

```bash
jq '[.[] | .games] | unique[]' -r < olympics.json | while read -r game; do
    echo "Selecting: $game";
    jq '[.[] | select(.games == "'$game'")]' < olympics.json > olympics-"$game".json
done
```


> <question-title></question-title>
>
> 1. How many different files do we get?
> 2. How many lines are in the file for the 1908 Summer Olympics?
>
> > <solution-title></solution-title>
> >
> > 1. We get 1 collection with 52 files in it, one per Olympic games:
> > 2. 61049 lines, 3213 entries.
> {: .solution}
{: .question}


## Exercises

Let's practice this a bit more, see if you can use the split file to answer the following questions:

> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. Create two files, one for Summer Olympics, one for Winter Olympics. Which has more lines?
> 2. Split the file by sport, how many sports have there been at the Olympics?
> 3. Split the file by medal, would you expect the output files to be equal sizes?
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The summer olympics file has more lines than the winter olympics file (the first winter Olympics wasn't held until 1924)
> >  2. Each file in the resulting collection represents a sport, there are 91 datasets, representing 91 different sports in the history of the Olympics.
> >  3. The files for gold silver and bronze are roughly equal, but not exactly (think of situations like shared second place, then 2 silver medals are handed out and no bronze medal)
> {: .solution}
>
{: .question}



# Conclusion

These tools and operations covered in the tutorial are just a few examples of some of the most common operations. There are many more tools available in Galaxy that perform other data manipulations. We encourage you to look around the `General Text Tools` section in the Galaxy toolbox (left panel) to find and explore more data manipulation tools. The more comfortable you are performing these kinds of steps, the more you can get out of Galaxy!


# Exercises: Putting it all together!

This section provides a number of exercises that require you to combine two or more of the techniques you learned in this tutorial. This is a great w


topic: data-science
tutorial: sql-python
title: SQL with Python
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["SQL", "Python"]
content_hash: 0c86c853e687219b51e2adfe5f918128
last_modified: 2026-02-26T21:12:14.912370
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-python/tutorial.html

content:
This tutorial will introduce you to accessing a SQL database from within Python. Experience with both SQL and Python is a pre-requisite.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

For this tutorial we need to download a database that we will use for the queries.

```python
!wget -c http://swcarpentry.github.io/sql-novice-survey/files/survey.db
```


# Programming with Databases - Python

Let's have a look at how to access a database from
a general-purpose programming language like Python.
Other languages use almost exactly the same model:
library and function names may differ,
but the concepts are the same.

Here's a short Python program that selects latitudes and longitudes
from an SQLite database stored in a file called `survey.db`:

```python
import sqlite3

connection = sqlite3.connect("survey.db")
cursor = connection.cursor()
cursor.execute("SELECT Site.lat, Site.long FROM Site;")
results = cursor.fetchall()
for r in results:
    print(r)
cursor.close()
connection.close()
```

The program starts by importing the `sqlite3` library.
If we were connecting to MySQL, DB2, or some other database,
we would import a different library,
but all of them provide the same functions,
so that the rest of our program does not have to change
(at least, not much)
if we switch from one database to another.

Line 2 establishes a connection to the database.
Since we're using SQLite,
all we need to specify is the name of the database file.
Other systems may require us to provide a username and password as well.
Line 3 then uses this connection to create a cursor.
Just like the cursor in an editor,
its role is to keep track of where we are in the database.

On line 4, we use that cursor to ask the database to execute a query for us.
The query is written in SQL,
and passed to `cursor.execute` as a string.
It's our job to make sure that SQL is properly formatted;
if it isn't,
or if something goes wrong when it is being executed,
the database will report an error.

The database returns the results of the query to us
in response to the `cursor.fetchall` call on line 5.
This result is a list with one entry for each record in the result set;
if we loop over that list (line 6) and print those list entries (line 7),
we can see that each one is a tuple
with one element for each field we asked for.

Finally, lines 8 and 9 close our cursor and our connection,
since the database can only keep a limited number of these open at one time.
Since establishing a connection takes time,
though,
we shouldn't open a connection,
do one operation,
then close the connection,
only to reopen it a few microseconds later to do another operation.
Instead,
it's normal to create one connection that stays open for the lifetime of the program.

Queries in real applications will often depend on values provided by users.
For example,
this function takes a user's ID as a parameter and returns their name:

```python
import sqlite3

def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id='" + person_id + "';"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query)
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

print("Full name for dyer:", get_name('survey.db', 'dyer'))
```

We use string concatenation on the first line of this function
to construct a query containing the user ID we have been given.
This seems simple enough,
but what happens if someone gives us this string as input?

```
dyer'; DROP TABLE Survey; SELECT '
```

It looks like there's garbage after the user's ID,
but it is very carefully chosen garbage.
If we insert this string into our query,
the result is:

```
SELECT personal || ' ' || family FROM Person WHERE id='dyer'; DROP TABLE Survey; SELECT '';
```

If we execute this,
it will erase one of the tables in our database.

This is called an SQL injection attack,
and it has been used to attack thousands of programs over the years.
In particular,
many web sites that take data from users insert values directly into queries
without checking them carefully first.
A very [relevant XKCD](https://xkcd.com/327/) that explains the
dangers of using raw input in queries a little more succinctly:

![A 4 panel comic, in the first panel a person is shown answering the phone, hearing that their son's school has some computer trouble. In panel 2 they apologises asking if their child broke something. In panel 3, the unseen person on the other end of the phone call asks if they really named their son Robert'); Drop table students;--? They respond saying 'oh yes. little bobby tables we call him.' In the 4th panel the caller says 'well we have lost this years student records, I hope you're happy.' They respond 'And I hope you've learned to sanitize your database inputs'.](../../images/xkcd/exploits_of_a_mom.png)

Since a villain might try to smuggle commands into our queries in many different ways,
the safest way to deal with this threat is
to replace characters like quotes with their escaped equivalents,
so that we can safely put whatever the user gives us inside a string.
We can do this by using a prepared statement
instead of formatting our statements as strings.
Here's what our example program looks like if we do this:

```python
import sqlite3

def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id=?;"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, [person_id])
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

print("Full name for dyer:", get_name('survey.db', 'dyer'))
```

The key changes are in the query string and the `execute` call.
Instead of formatting the query ourselves,
we put question marks in the query template where we want to insert values.
When we call `execute`,
we provide a list
that contains as many values as there are question marks in the query.
The library matches values to question marks in order,
and translates any special characters in the values
into their escaped equivalents
so that they are safe to use.

We can also use `sqlite3`'s cursor to make changes to our database,
such as inserting a new name.
For instance, we can define a new function called `add_name` like so:

```python
import sqlite3

def add_name(database_file, new_person):
    query = "INSERT INTO Person (id, personal, family) VALUES (?, ?, ?);"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, list(new_person))
    cursor.close()
    connection.close()


def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id=?;"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, [person_id])
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

# Insert a new name
add_name('survey.db', ('barrett', 'Mary', 'Barrett'))
# Check it exists
print("Full name for barrett:", get_name('survey.db', 'barrett'))
```


Note that in versions of sqlite3 >= 2.5, the `get_name` function described
above will fail with an `IndexError: list index out of range`,
even though we added Mary's
entry into the table using `add_name`.
This is because we must perform a `connection.commit()` before closing
the connection, in order to save our changes to the database.

```python
import sqlite3

def add_name(database_file, new_person):
    query = "INSERT INTO Person (id, personal, family) VALUES (?, ?, ?);"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, list(new_person))
    cursor.close()
    connection.commit()
    connection.close()


def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id=?;"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, [person_id])
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

# Insert a new name
add_name('survey.db', ('barrett', 'Mary', 'Barrett'))
# Check it exists
print("Full name for barrett:", get_name('survey.db', 'barrett'))
```


> <question-title>Filling a Table vs. Printing Values</question-title>
>
> Write a Python program that creates a new database in a file called
> `original.db` containing a single table called `Pressure`, with a
> single field called `reading`, and inserts 100,000 random numbers
> between 10.0 and 25.0.  How long does it take this program to run?
> How long does it take to run a program that simply writes those
> random numbers to a file?
>
> > <solution-title></solution-title>
> > ```
> > import sqlite3
> > # import random number generator
> > from numpy.random import uniform
> >
> > random_numbers = uniform(low=10.0, high=25.0, size=100000)
> >
> > connection = sqlite3.connect("original.db")
> > cursor = connection.cursor()
> > cursor.execute("CREATE TABLE Pressure (reading float not null)")
> > query = "INSERT INTO Pressure (reading) VALUES (?);"
> >
> > for number in random_numbers:
> >     cursor.execute(query, [number])
> >
> > cursor.close()
> > connection.commit() # save changes to file for next exercise
> > connection.close()
> > ```
> > {: .python}
> >
> > For comparison, the following program writes the random numbers
> > into the file `random_numbers.txt`:
> >
> > ```
> > from numpy.random import uniform
> >
> > random_numbers = uniform(low=10.0, high=25.0, size=100000)
> > with open('random_numbers.txt', 'w') as outfile:
> >     for number in random_numbers:
> >         # need to add linebreak \n
> >         outfile.write("{}\n".format(number))
> > ```
> > {: .python}
> {: .solution}
{: .question}

> <question-title>Filtering in SQL vs. Filtering in Python</question-title>
>
> Write a Python program that creates a new database called
> `backup.db` with the same structure as `original.db` and copies all
> the values greater than 20.0 from `original.db` to `backup.db`.
> Which is faster: filtering values in the query, or reading
> everything into memory and filtering in Python?
>
> > <solution-title></solution-title>
> > The first example reads all the data into memory and filters the
> > numbers using the if statement in Python.
> >
> > ```
> > import sqlite3
> >
> > connection_original = sqlite3.connect("original.db")
> > cursor_original = connection_original.cursor()
> > cursor_original.execute("SELECT * FROM Pressure;")
> > results = cursor_original.fetchall()
> > cursor_original.close()
> > connection_original.close()
> >
> > connection_backup = sqlite3.connect("backup.db")
> > cursor_backup = connection_backup.cursor()
> > cursor_backup.execute("CREATE TABLE Pressure (reading float not null)")
> > query = "INSERT INTO Pressure (reading) VALUES (?);"
> >
> > for entry in results:
> >     # number is saved in first column of the table
> >     if entry[0] > 20.0:
> >         cursor_backup.execute(query, entry)
> >
> > cursor_backup.close()
> > connection_backup.commit()
> > connection_backup.close()
> > ```
> >
> > In contrast the following example uses the conditional ``SELECT`` statement
> > to filter the numbers in SQL.
> > The only lines that changed are in line 5, where the values are fetched
> > from `original.db` and the for loop starting in line 15 used to insert
> > the numbers into `backup.db`.
> > Note how this version does not require the use of Python's if statement.
> >
> > ```
> > import sqlite3
> >
> > connection_original = sqlite3.connect("original.db")
> > cursor_original = connection_original.cursor()
> > cursor_original.execute("SELECT * FROM Pressure WHERE reading > 20.0;")
> > results = cursor_original.fetchall()
> > cursor_original.close()
> > connection_original.close()
> >
> > connection_backup = sqlite3.connect("backup.db")
> > cursor_backup = connection_backup.cursor()
> > cursor_backup.execute("CREATE TABLE Pressure (reading float not null)")
> > query = "INSERT INTO Pressure (reading) VALUES (?);"
> >
> > for entry in results:
> >     cursor_backup.execute(query, entry)
> >
> > cursor_backup.close()
> > connection_backup.commit()
> > connection_backup.close()
> > ```
> >
> {: .solution}
{: .question}

> <question-title>Generating Insert Statements</question-title>
>
> One of our colleagues has sent us a
> CSV
> file containing
> temperature readings by Robert Olmstead, which is formatted like
> this:
>
> ```
> Taken,Temp
> 619,-21.5
> 622,-15.5
> ```
> {: .output}
>
> Write a small Python program that reads this file in and prints out
> the SQL `INSERT` statements needed to add these records to the
> survey database.  Note: you will need to add an entry for Olmstead
> to the `Person` table.  If you are testing your program repeatedly,
> you may want to investigate SQL's `INSERT or REPLACE` command.
{: .question}


topic: data-science
tutorial: sql-advanced
title: Advanced SQL
difficulty: introductory
time_estimation: 3H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["SQL"]
content_hash: b5d50610d3a32491f4d19a9d95aaadf4
last_modified: 2026-02-26T21:12:14.910439
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-advanced/tutorial.html

content:
> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


```sql
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c http://swcarpentry.github.io/sql-novice-survey/files/survey.db
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///survey.db")
%load_ext sql
%sql sqlite:///survey.db
%config SqlMagic.displaycon=False
```

# Aggregation

We now want to calculate ranges and averages for our data.
We know how to select all of the dates from the `Visited` table:

```sql
SELECT dated FROM Visited;
```


but to combine them,
we must use an aggregation function
such as `min` or `max`.
Each of these functions takes a set of records as input,
and produces a single record as output:

```sql
SELECT min(dated) FROM Visited;
```

![SQL Aggregation](../../images/carpentries-sql/sql-aggregation.svg)

```sql
SELECT max(dated) FROM Visited;
```

`min` and `max` are just two of
the aggregation functions built into SQL.
Three others are `avg`,
`count`,
and `sum`:

```sql
SELECT avg(reading) FROM Survey WHERE quant = 'sal';
```

```sql
SELECT count(reading) FROM Survey WHERE quant = 'sal';
```

```sql
SELECT sum(reading) FROM Survey WHERE quant = 'sal';
```

We used `count(reading)` here,
but we could just as easily have counted `quant`
or any other field in the table,
or even used `count(*)`,
since the function doesn't care about the values themselves,
just how many values there are.

SQL lets us do several aggregations at once.
We can,
for example,
find the range of sensible salinity measurements:

```sql
SELECT min(reading), max(reading) FROM Survey WHERE quant = 'sal' AND reading <= 1.0;
```

We can also combine aggregated results with raw results,
although the output might surprise you:

```sql
SELECT person, count(*) FROM Survey WHERE quant = 'sal' AND reading <= 1.0;
```

Why does Lake's name appear rather than Roerich's or Dyer's?
The answer is that when it has to aggregate a field,
but isn't told how to,
the database manager chooses an actual value from the input set.
It might use the first one processed,
the last one,
or something else entirely.

Another important fact is that when there are no values to aggregate ---
for example, where there are no rows satisfying the `WHERE` clause ---
aggregation's result is "don't know"
rather than zero or some other arbitrary value:

```sql
SELECT person, max(reading), sum(reading) FROM Survey WHERE quant = 'missing';
```

One final important feature of aggregation functions is that
they are inconsistent with the rest of SQL in a very useful way.
If we add two values,
and one of them is null,
the result is null.
By extension,
if we use `sum` to add all the values in a set,
and any of those values are null,
the result should also be null.
It's much more useful,
though,
for aggregation functions to ignore null values
and only combine those that are non-null.
This behavior lets us write our queries as:

```sql
SELECT min(dated) FROM Visited;
```

instead of always having to filter explicitly:

```sql
SELECT min(dated) FROM Visited WHERE dated IS NOT NULL;
```

Aggregating all records at once doesn't always make sense.
For example,
suppose we suspect that there is a systematic bias in our data,
and that some scientists' radiation readings are higher than others.
We know that this doesn't work:

```sql
SELECT person, count(reading), round(avg(reading), 2)
FROM  Survey
WHERE quant = 'rad';
```

because the database manager selects a single arbitrary scientist's name
rather than aggregating separately for each scientist.
Since there are only five scientists,
we could write five queries of the form:

```sql
SELECT person, count(reading), round(avg(reading), 2)
FROM  Survey
WHERE quant = 'rad'
AND   person = 'dyer';
```

but this would be tedious,
and if we ever had a data set with fifty or five hundred scientists,
the chances of us getting all of those queries right is small.

What we need to do is
tell the database manager to aggregate the hours for each scientist separately
using a `GROUP BY` clause:

```sql
SELECT   person, count(reading), round(avg(reading), 2)
FROM     Survey
WHERE    quant = 'rad'
GROUP BY person;
```

`GROUP BY` does exactly what its name implies:
groups all the records with the same value for the specified field together
so that aggregation can process each batch separately.
Since all the records in each batch have the same value for `person`,
it no longer matters that the database manager
is picking an arbitrary one to display
alongside the aggregated `reading` values.

> <tip-title>Know Excel? It's just a pivot table.</tip-title>
> `GROUP BY` is basically just a pivot table for Excel users, it lets you build
> nice summary tables which aggregate your results.
>
> And if you didn't already know the Excel equivalent, now you know what to
> look for when you need it!
{: .tip}

Just as we can sort by multiple criteria at once,
we can also group by multiple criteria.
To get the average reading by scientist and quantity measured,
for example,
we just add another field to the `GROUP BY` clause:

```sql
SELECT   person, quant, count(reading), round(avg(reading), 2)
FROM     Survey
GROUP BY person, quant;
```

Note that we have added `quant` to the list of fields displayed,
since the results wouldn't make much sense otherwise.

Let's go one step further and remove all the entries
where we don't know who took the measurement:

```sql
SELECT   person, quant, count(reading), round(avg(reading), 2)
FROM     Survey
WHERE    person IS NOT NULL
GROUP BY person, quant
ORDER BY person, quant;
```


Looking more closely,
this query:

1.  selected records from the `Survey` table where the `person` field was not null;
2.  grouped those records into subsets so that the `person` and `quant` values in each subset were the same;
3.  ordered those subsets first by `person`, and then within each sub-group by `quant`; and
4.  counted the number of records in each subset, calculated the average `reading` in each, and chose a `person` and `quant` value from each (it doesn't matter which ones, since they're all equal).

> <question-title>Counting Temperature Readings</question-title>
>
> How many temperature readings did Frank Pabodie record,
> and what was their average value?
>
> > <solution-title></solution-title>
> >
> > ```
> > SELECT count(reading), avg(reading) FROM Survey WHERE quant = 'temp' AND person = 'pb';
> > ```
> >
> > |count(reading)|avg(reading)|
> > |--------------|------------|
> > |2             |-20.0       |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Averaging with NULL</question-title>
>
> The average of a set of values is the sum of the values
> divided by the number of values.
> Does this mean that the `avg` function returns 2.0 or 3.0
> when given the values 1.0, `null`, and 5.0?
>
> > <solution-title></solution-title>
> > The answer is 3.0.
> > `NULL` is not a value; it is the absence of a value.
> > As such it is not included in the calculation.
> >
> > You can confirm this, by executing this code:
> >
> > ```
> > SELECT AVG(a) FROM (
> >     SELECT 1 AS a
> >     UNION ALL SELECT NULL
> >     UNION ALL SELECT 5);
> > ```
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>What Does This Query Do?</question-title>
>
> We want to calculate the difference between
> each individual radiation reading
> and the average of all the radiation readings.
> We write the query:
>
> ```
> SELECT reading - avg(reading) FROM Survey WHERE quant = 'rad';
> ```
>
> What does this actually produce, and can you think of why?
>
> > <solution-title></solution-title>
> > The query produces only one row of results when we what we really want is a result for each of the readings.
> > The `avg()` function produces only a single value, and because it is run first, the table is reduced to a single row.
> > The `reading` value is simply an arbitrary one.
> >
> > To achieve what we wanted, we would have to run two queries:
> >
> > ```
> > SELECT avg(reading) FROM Survey WHERE quant='rad';
> > ```
> >
> > This produces the average value (6.5625), which we can then insert into a second query:
> >
> > ```
> > SELECT reading - 6.5625 FROM Survey WHERE quant = 'rad';
> > ```
> >
> > This produces what we want, but we can combine this into a single query using subqueries.
> >
> > ```
> > SELECT reading - (SELECT avg(reading) FROM Survey WHERE quant='rad') FROM Survey WHERE quant = 'rad';
> > ```
> >
> > This way we don't have execute two queries.
> >
> > In summary what we have done is to replace `avg(reading)` with `(SELECT avg(reading) FROM Survey WHERE quant='rad')` in the original query.
> >
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Ordering When Concatenating</question-title>
>
> The function `group_concat(field, separator)`
> concatenates all the values in a field
> using the specified separator character
> (or ',' if the separator isn't specified).
> Use this to produce a one-line list of scientists' names,
> such as:
>
> ```
> William Dyer, Frank Pabodie, Anderson Lake, Valentina Roerich, Frank Danforth
> ```
>
> Can you find a way to order the list by surname?
{: .question}

```sql
-- Try solutions here!
```


# Combining Data

In order to submit our data to a web site
that aggregates historical meteorological data,
we might need to format it as
latitude, longitude, date, quantity, and reading.
However,
our latitudes and longitudes are in the `Site` table,
while the dates of measurements are in the `Visited` table
and the readings themselves are in the `Survey` table.
We need to combine these tables somehow.

This figure shows the relations between the tables:

![Survey Database Structure](../../images/carpentries-sql/sql-join-structure.svg)

The SQL command to do this is `JOIN`.
To see how it works,
let's start by joining the `Site` and `Visited` tables:

```sql
SELECT * FROM Site JOIN Visited;
```

`JOIN` creates
the cross product
of two tables,
i.e.,
it joins each record of one table with each record of the other table
to give all possible combinations.
Since there are three records in `Site`
and eight in `Visited`,
the join's output has 24 records (3 * 8 = 24) .
And since each table has three fields,
the output has six fields (3 + 3 = 6).

What the join *hasn't* done is
figure out if the records being joined have anything to do with each other.
It has no way of knowing whether they do or not until we tell it how.
To do that,
we add a clause specifying that
we're only interested in combinations that have the same site name,
thus we need to use a filter:

```sql
SELECT * FROM Site JOIN Visited ON Site.name = Visited.site;
```

`ON` is very similar to `WHERE`,
and for all the queries in this lesson you can use them interchangeably.
There are differences in how they affect [outer joins][outer],
but that's beyond the scope of this lesson.
Once we add this to our query,
the database manager throws away records
that combined information about two different sites,
leaving us with just the ones we want.

Notice that we used `Table.field` to specify field names
in the output of the join.
We do this because tables can have fields with the same name,
and we need to be specific which ones we're talking about.
For example,
if we joined the `Person` and `Visited` tables,
the result would inherit a field called `id`
from each of the original tables.

We can now use the same dotted notation
to select the three columns we actually want
out of our join:

```sql
SELECT Site.lat, Site.long, Visited.dated
FROM   Site JOIN Visited
ON     Site.name = Visited.site;
```


If joining two tables is good,
joining many tables must be better.
In fact,
we can join any number of tables
simply by adding more `JOIN` clauses to our query,
and more `ON` tests to filter out combinations of records
that don't make sense:

```sql
SELECT Site.lat, Site.long, Visited.dated, Survey.quant, Survey.reading
FROM   Site JOIN Visited JOIN Survey
ON     Site.name = Visited.site
AND    Visited.id = Survey.taken
AND    Visited.dated IS NOT NULL;
```

We can tell which records from `Site`, `Visited`, and `Survey`
correspond with each other
because those tables contain
primary keys
and foreign keys.
A primary key is a value,
or combination of values,
that uniquely identifies each record in a table.
A foreign key is a value (or combination of values) from one table
that identifies a unique record in another table.
Another way of saying this is that
a foreign key is the primary key of one table
that appears in some other table.
In our database,
`Person.id` is the primary key in the `Person` table,
while `Survey.person` is a foreign key
relating the `Survey` table's entries
to entries in `Person`.

Most database designers believe that
every table should have a well-defined primary key.
They also believe that this key should be separate from the data itself,
so that if we ever need to change the data,
we only need to make one change in one place.
One easy way to do this is
to create an arbitrary, unique ID for each record
as we add it to the database.
This is actually very common:
those IDs have names like "student numbers" and "patient numbers",
and they almost always turn out to have originally been
a unique record identifier in some database system or other.
As the query below demonstrates,
SQLite [automatically numbers records][rowid] as they're added to tables,
and we can use those record numbers in queries:

```sql
SELECT rowid, * FROM Person;
```

> <question-title>Listing Radiation Readings</question-title>
>
> Write a query that lists all radiation readings from the DR-1 site.
> > <solution-title></solution-title>
> >
> > ```
> > SELECT Survey.reading
> > FROM Site JOIN Visited JOIN Survey
> > ON Site.name = Visited.site
> > AND Visited.id = Survey.taken
> > WHERE Site.name = 'DR-1'
> > AND Survey.quant = 'rad';
> > ```
> >
> > |reading   |
> > |----------|
> > |9.82      |
> > |7.8       |
> > |11.25     |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Where's Frank?</question-title>
>
> Write a query that lists all sites visited by people named "Frank".
> > <solution-title></solution-title>
> >
> > ```
> > SELECT DISTINCT Site.name
> > FROM Site JOIN Visited JOIN Survey JOIN Person
> > ON Site.name = Visited.site
> > AND Visited.id = Survey.taken
> > AND Survey.person = Person.id
> > WHERE Person.personal = 'Frank';
> > ```
> >
> > |name   |
> > |-------|
> > |DR-3   |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Reading Queries</question-title>
>
> Describe in your own words what the following query produces:
>
> ```
> SELECT Site.name FROM Site JOIN Visited
> ON Site.lat < -49.0 AND Site.name = Visited.site AND Visited.dated >= '1932-01-01';
> ```
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Who Has Been Where?</question-title>
>
> Write a query that shows each site with exact location (lat, long) ordered by visited date,
> followed by personal name and family name of the person who visited the site
> and the type of measurement taken and its reading. Please avoid all null values.
> Tip: you should get 15 records with 8 fields.
> > <solution-title></solution-title>
> >
> > ```
> > SELECT Site.name, Site.lat, Site.long, Person.personal, Person.family, Survey.quant, Survey.reading, Visited.dated
> > FROM Site JOIN Visited JOIN Survey JOIN Person
> > ON Site.name = Visited.site
> > AND Visited.id = Survey.taken
> > AND Survey.person = Person.id
> > WHERE Survey.person IS NOT NULL
> > AND Visited.dated IS NOT NULL
> > ORDER BY Visited.dated;
> > ```
> >
> > name   |  lat        |  long       |  personal   | family   | quant     | reading   |     dated
> >--------|-------------|-------------|-------------|----------|-----------|-----------|-----------
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   rad     |    9.82   |   1927-02-08
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   sal     |    0.13   |   1927-02-08
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   rad     |    7.8    |   1927-02-10
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   sal     |    0.09   |   1927-02-10
> >DR-3    |    -47.15   |   -126.72   |  Anderson   | Lake     |   sal     |    0.05   |   1930-01-07
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   rad     |    8.41   |   1930-01-07
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   temp    |    -21.5  |   1930-01-07
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   rad     |    7.22   |   1930-01-12
> >DR-3    |    -47.15   |   -126.72   |  Anderson   | Lake     |   sal     |    0.1    |   1930-02-26
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   rad     |    4.35   |   1930-02-26
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   temp    |    -18.5  |   1930-02-26
> >MSK-4   |    -48.87   |   -123.4    |  Anderson   | Lake     |   rad     |    1.46   |   1932-01-14
> >MSK-4   |    -48.87   |   -123.4    |  Anderson   | Lake     |   sal     |    0.21   |   1932-01-14
> >MSK-4   |    -48.87   |   -123.4    |  Valentina  | Roerich  |   sal     |    22.5   |   1932-01-14
> >DR-1    |    -49.85   |   -128.57   |  Valentina  | Roerich  |   rad     |    11.25  |   1932-03-22
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

A good visual explanation of joins can be found [in the SQL Join Visualizer][joinref]

[outer]: https://en.wikipedia.org/wiki/Join_%28SQL%29#Outer_join
[rowid]: https://www.sqlite.org/lang_createtable.html#rowid
[joinref]: https://sql-joins.leopard.in.ua/

# Data Hygiene

Now that we have seen how joins work, we can see why the relational
model is so useful and how best to use it.  The first rule is that
every value should be atomic, i.e., not
contain parts that we might want to work with separately.  We store
personal and family names in separate columns instead of putting the
entire name in one column so that we don't have to use substring
operations to get the name's components.  More importantly, we store
the two parts of the name separately because splitting on spaces is
unreliable: just think of a name like "Eloise St. Cyr" or "Jan Mikkel
Steubart".

The second rule is that every record should have a unique primary key.
This can be a serial number that has no intrinsic meaning,
one of the values in the record (like the `id` field in the `Person` table),
or even a combination of values:
the triple `(taken, person, quant)` from the `Survey` table uniquely identifies every measurement.

The third rule is that there should be no redundant information.
For example,
we could get rid of the `Site` table and rewrite the `Visited` table like this:

|id   |lat   |long   |dated      |
|-----|------|-------|-----------|
|619  |-49.85|-128.57| 1927-02-08|
|622  |-49.85|-128.57| 1927-02-10|
|734  |-47.15|-126.72| 1930-01-07|
|735  |-47.15|-126.72| 1930-01-12|
|751  |-47.15|-126.72| 1930-02-26|
|752  |-47.15|-126.72| None    |
|837  |-48.87|-123.40| 1932-01-14|
|844  |-49.85|-128.57| 1932-03-22|

In fact,
we could use a single table that recorded all the information about each reading in each row,
just as a spreadsheet would.
The problem is that it's very hard to keep data organized this way consistent:
if we realize that the date of a particular visit to a particular site is wrong,
we have to change multiple records in the database.
What's worse,
we may have to guess which records to change,
since other sites may also have been visited on that date.

The fourth rule is that the units for every value should be stored explicitly.
Our database doesn't do this,
and that's a problem:
Roerich's salinity measurements are several orders of magnitude larger than anyone else's,
but we don't know if that means she was using parts per million instead of parts per thousand,
or whether there actually was a saline anomaly at that site in 1932.

Stepping back,
data and the tools used to store it have a symbiotic relationship:
we use tables and joins because it's efficient,
provided our data is organized a certain way,
but organize our data that way because we have tools to manipulate it efficiently.
As anthropologists say,
the tool shapes the hand that shapes the tool.

> <question-title>Identifying Atomic Values</question-title>
>
> Which of the following are atomic values? Which are not? Why?
>
> *   New Zealand
> *   87 Turing Avenue
> *   January 25, 1971
> *   the XY coordinate (0.5, 3.3)
>
> > <solution-title></solution-title>
> > New Zealand is the only clear-cut atomic value.
> >
> > The address and the XY coordinate contain more than one piece of information
> > which should be stored separately:
> > - House number, street name
> > - X coordinate, Y coordinate
> >
> > The date entry is less clear cut, because it contains month, day, and year elements.
> > However, there is a `DATE` datatype in SQL, and dates should be stored using this format.
> > If we need to work with the month, day, or year separately, we can use the SQL functions available for our database software
> > (for example [`EXTRACT`](https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions050.htm) or [`STRFTIME`](http://www.sqlite.org/lang_datefunc.html) for SQLite).
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Identifying a Primary Key</question-title>
>
> What is the primary key in this table?
> I.e., what value or combination of values uniquely identifies a record?
>
> |latitude|longitude|date      |temperature|
> |--------|---------|----------|-----------|
> |57.3    |-22.5    |2015-01-09|-14.2      |
>
> > <solution-title></solution-title>
> > Latitude, longitude, and date are all required to uniquely identify the temperature record.
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

# Creating and Modifying Data

So far we have only looked at how to get information out of a database,
both because that is more frequent than adding information,
and because most other operations only make sense
once queries are understood.
If we want to create and modify data,
we need to know two other sets of commands.

The first pair are [`CREATE TABLE`][create-table] and [`DROP TABLE`][drop-table].
While they are written as two words,
they are actually single commands.
The first one creates a new table;
its arguments are the names and types of the table's columns.
For example,
the following statements create the four tables in our survey database:

```sql
CREATE TABLE Person(id text, personal text, family text);
CREATE TABLE Site(name text, lat real, long real);
CREATE TABLE Visited(id integer, site text, dated text);
CREATE TABLE Survey(taken integer, person text, quant text, reading real);
```

We can get rid of one of our tables using:

```sql
DROP TABLE Survey;
```

Be very careful when doing this:
if you drop the wrong table, hope that the person maintaining the database has a backup,
but it's better not to have to rely on it.

Different database systems support different data types for table columns,
but most provide the following:

|data type|  use                                       |
|---------|  ----------------------------------------- |
|INTEGER  |  a signed integer                          |
|REAL     |  a floating point number                   |
|TEXT     |  a character string                        |
|BLOB     |  a "binary large object", such as an image |

Most databases also support Booleans and date/time values;
SQLite uses the integers 0 and 1 for the former,
and represents the latter as text or numeric fields.

An increasing number of databases also support geographic data types,
such as latitude and longitude.
Keeping track of what particular systems do or do not offer,
and what names they give different data types,
is an unending portability headache.

> <tip-title>Which database should I use?</tip-title>
> SQLite is fantastic for small databases or embedded into applications where
> you want to be able to use SQL to query and process data.
>
> However for any real analysis PostgreSQL is usually the best choice, it
> scales incredibly well and can meet a wide range of use cases. It has good
> data type support.
{: .tip}

> <tip-title>Do you have geographic data?</tip-title>
> Use Postgres. The [PostGIS](https://postgis.net/) library is fantastic and industry standard for storing geographic data in a database.
{: .tip}

When we create a table,
we can specify several kinds of constraints on its columns.
For example,
a better definition for the `Survey` table would be:

```sql
CREATE TABLE Survey(
    taken   integer not null, -- where reading taken
    person  text,             -- may not know who took it
    quant   text not null,    -- the quantity measured
    reading real not null,    -- the actual reading
    primary key(taken, quant),
    foreign key(taken) references Visited(id),
    foreign key(person) references Person(id)
);
```

Once again,
exactly what constraints are available
and what they're called
depends on which database manager we are using.

Once tables have been created,
we can add, change, and remove records using our other set of commands,
`INSERT`, `UPDATE`, and `DELETE`.

Here is an example of inserting rows into the `Site` table:

```sql
INSERT INTO Site (name, lat, long) VALUES ('DR-1', -49.85, -128.57);
INSERT INTO Site (name, lat, long) VALUES ('DR-3', -47.15, -126.72);
INSERT INTO Site (name, lat, long) VALUES ('MSK-4', -48.87, -123.40);
```

We can also insert values into one table directly from another:

```sql
CREATE TABLE JustLatLong(lat real, long real);
INSERT INTO JustLatLong SELECT lat, long FROM Site;
```

Modifying existing records is done using the `UPDATE` statement.
To do this we tell the database which table we want to update,
what we want to change the values to for any or all of the fields,
and under what conditions we should update the values.

For example, if we made a mistake when entering the lat and long values
of the last `INSERT` statement above, we can correct it with an update:

```sql
UPDATE Site SET lat = -47.87, long = -122.40 WHERE name = 'MSK-4';
```

Be careful to not forget the `WHERE` clause or the update statement will
modify *all* of the records in the database.

Deleting records can be a bit trickier,
because we have to ensure that the database remains internally consistent.
If all we care about is a single table,
we can use the `DELETE` command with a `WHERE` clause
that matches the records we want to discard.
For example,
once we realize that Frank Danforth didn't take any measurements,
we can remove him from the `Person` table like this:

```sql
DELETE FROM Person WHERE id = 'danforth';
```

But what if we removed Anderson Lake instead?
Our `Survey` table would still contain seven records
of measurements he'd taken,
but that's never supposed to happen:
`Survey.person` is a foreign key into the `Person` table,
and all our queries assume there will be a row in the latter
matching every value in the former.

This problem is called referential integrity:
we need to ensure that all references between tables can always be resolved correctly.
One way to do this is to delete all the records
that use `'lake'` as a foreign key
before deleting the record that uses it as a primary key.
If our database manager supports it,
we can automate this
using cascading delete.
However,
this technique is outside the scope of this chapter.

> <tip-title>Hybrid Storage Models</tip-title>
>
> Many applications use a hybrid storage model
> instead of putting everything into a database:
> the actual data (such as astronomical images) is stored in files,
> while the database stores the files' names,
> their modification dates,
> the region of the sky they cover,
> their spectral characteristics,
> and so on.
> This is also how most music player software is built:
> the database inside the application keeps track of the MP3 files,
> but the files themselves live on disk.
{: .tip}

> <question-title>Replacing NULL</question-title>
>
> Write an SQL statement to replace all uses of `null` in
> `Survey.person` with the string `'unknown'`.
>
> > <solution-title></solution-title>
> > ```
> > UPDATE Survey SET person = 'unknown' WHERE person IS NULL;
> > ```
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Backing Up with SQL</question-title>
>
> SQLite has several administrative commands that aren't part of the
> SQL standard.  One of them is `.dump`, which prints the SQL commands
> needed to re-create the database.  Another is `.read`, which reads a
> file created by `.dump` and restores the database.  A colleague of
> yours thinks that storing dump files (which are text) in version
> control is a good way to track and manage changes to the database.
> What are the pros and cons of this approach?  (Hint: records aren't
> stored in any particular order.)
>
> > <solution-title></solution-title>
> > #### Advantages
> > - A version control system will be able to show differences between versions
> > of the dump file; something it can't do for binary files like databases
> > - A VCS only saves changes between versions, rather than a complete copy of
> > each version (save disk space)
> > - The version control log will explain the reason for the changes in each version
> > of the database
> >
> > #### Disadvantages
> > - Artificial differences between commits because records don't have a fixed order
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

{% include _includes/quiz.html id="recap.yml" %}

[create-table]: https://www.sqlite.org/lang_createtable.html
[drop-table]: https://www.sqlite.org/lang_droptable.html


topic: data-science
tutorial: git-cli
title: Basics of using Git from the Command Line
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 30b096c4bd0bb6d173b67c83443638ad
last_modified: 2026-02-26T21:12:14.849353
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/git-cli/tutorial.html

content:
Version control is a way of tracking the change history of a project. Even if you have never used a version control tool, you've probably already done it manually: copying and renaming project folders ("paper-1.doc", "paper-2.doc", etc.) is a form of version control. Within bioinformatics (from research, to development, to sysadmin) a lot of us are using `git` as our primary method of source control for everything we do: notes, slides, tutorials, code, notebooks, ansible, system configuration, and more.

> <comment-title>Source</comment-title>
> This tutorial contains text from [this tutorial](https://github.com/rdadolf/git-tutorial/) by Robert Adolf ([@rdadolf](https://github.com/rdadolf)), which is licensed CC-BY.
{: .comment}

Git is a tool that automates and enhances a lot of the tasks that arise when dealing with larger, longer-living, and collaborative projects. It's also become the common underpinning to many popular online code repositories, GitHub being the most popular.

While it can be used collaboratively, this tutorial focuses on a single-user git repository for the most basic operations.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Why should you use version control?

If you ask 10 people, you'll get 10 different answers, but one of the commonalities is that most people don't realize how integral it is to their development process until they've started using it. Still, for the sake of argument, here are some highlights:

- **You can undo anything**: Git provides a complete history of every change that has ever been made to your project, timestamped, commented, and attributed. If something breaks, you always have the choice of going back to a previous tate.
- **You won't need to keep undo-ing things**: One of the advantages of using git properly is that by keeping new changes separate from a stable base, you tend to avoid the massive rollbacks associated with constantly tinkering with a single code.
- **You can identify exactly when and where changes were made** (and by whom!): Git allows you to pinpoint when a particular piece of code was changed, so finding what other pieces of code a bug might affect or figuring out why a certain expression was added is easy.
- **Git forces teams to face conflicts directly**: On a team-based project, many people are often working with the same code. By having a tool which understands when and where files were changed, it's easy to see when changes might conflict with each other. While it might seem troublesome sometimes to have to deal with conflicts, the alternative—not knowing there's a conflict—is much more insidious.

## Pre-requisites

[You will need to install git, if you have not done so already.](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)

## Setting up a Repository

Let's create a new repository.

> <hands-on-title>Create a Repository</hands-on-title>
> 1. Make a new directory where you will store your files, and navigate into it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > mkdir git-tutorial; cd git-tutorial;
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Create or "initialise" the `git` repository with the `git init` command.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git init
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > Initialized empty Git repository in /tmp/project/.git/
>    > ```
>    {: .code-out}
>
{: .hands_on}

This has created a folder `.git` in your project directory, here is where `git` stores all of it's data that it needs to track repository changes over time. It's not terribly interesting yet though!

> <hands-on-title>What's the status</hands-on-title>
> 1. You can always check the status of a repository with `git status`
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git status
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > On branch main
>    >
>    > No commits yet
>    >
>    > nothing to commit (create/copy files and use "git add" to track)
>    >
>    > ```
>    {: .code-out}
>
{: .hands_on}

## Adding Files

Let's add our first file, often a (pretty empty) readme file.

> <hands-on-title>What's the status</hands-on-title>
> 1. Create a new file, `readme.md` with some basic content
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "My Project" > readme.md
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Add a file with `git add`. This adds it to git's staging area to be committed.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git add readme.md
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Commit the file! This will add it to git's log.
>
>    > <tip-title>What makes a good commit message?</tip-title>
>    > It depends a lot on the community, some have specific style guides they enforce, some don't, but in general
>    > - Keep the description short (<72 chars) and descriptive.
>    > - If you need, provide a long description as well, explaining your changes. (Use `git commit` without the `-m` flag!)
>    > [A lot has been written](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) about good commit messages, search the internet and find ideas for what you think makes a good commit message!
>    >
>    > And beware of the trap we all fall into sometimes, [unhelpful commit messages](https://xkcd.com/1296/)
>    > ![xkcd comic with commits in a table from 14h to 2h ago, starting with extremely useful commits like "created main loop & timing control", and becoming extremely unhelpful messages for the latest commits, like 'AAAAAA' or "asdfasdf".](../../images/xkcd/git_commit.png)
>    > Even your author is [very](https://github.com/galaxyproject/gxadmin/commit/568adb703189d98da701e35c3414be35ea106184), [very](https://github.com/galaxyproject/training-material/commit/611f82d267f1426aa09863c2ac9c9006f38e3c4c) guilty of this, but you can do better!
>    {: .tip}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git commit -m "Add readme"
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > [main (root-commit) f5ec14f] Add readme
>    >  1 file changed, 1 insertion(+)
>    >  create mode 100644 readme.md
>    > ```
>    {: .code-out}
>
> > <question-title>Is there anything left to do? Check the status</question-title>
> > Check `git status` to see if there's anything else left to resolve.
> > > <solution-title></solution-title>
> > > ```console
> > > $ git status
> > > On branch main
> > > nothing to commit, working tree clean
> > > ```
> > {: .solution}
> {: .question}
>
{: .hands_on}

Congratulations! You've made your first commit. The output of the commit command lists everything you've just done:

```
[main (root-commit) f5ec14f] Add readme
 1 file changed, 1 insertion(+)
 create mode 100644 readme.md
```

`f5ec14f` is the **commit id**, every commit you make is given a hash which uniquely refers to that specific commit. Next we see our commit message `Add readme`, a brief mention of how many files we've changed, and how many insertions or deletions we've made to the text, and lastly which files we've added.

### Exercise: Make some more commits

> <hands-on-title>Make some more commits</hands-on-title>
>
> 1. Add your name to the `readme.md` and commit your changes.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "Author: hexylena" >> readme.md
>    > git add readme.md
>    > git commit -m 'Add author name'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Make up a project description, add it to the readme, and commit.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "This project enables stakeholders to experience synergistic effects and increase link up opportunities to improve quarterly and YOY ROI.\n" >> readme.md
>    > git add readme.md
>    > git commit -m 'Add project description'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. [Pick a license](https://spdx.org/licenses/) for your project, and mention it in the `readme.md`, and commit.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "# License\nAGPL-3.0" >> readme.md
>    > git add readme.md
>    > git commit -m 'Add project license'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

After this step you should have ~3 commits to work with!

## Logs

One of the most helpful things about git is that, if you have written good commit messages, you can tell what you did and when!

> <hands-on-title>Check the Receipts</hands-on-title>
> 1. Check the `log` with `git log`. Notice that you can see each commit in reverse chronological order (newest at top), who made the commit, when, and what the commit message was.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git log
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```
>    > commit 5d05eb3ec22fd49282b585c60ef8f983d68c2fd7
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:13:21 2022 +0200
>    >
>    >     Add project license
>    >
>    > commit 62f974ec5f538232f65b016cf073815349364efa
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:13:16 2022 +0200
>    >
>    >     Add project description
>    >
>    > commit 10355c019c04052c15a95a817de04f9ea0ec336c
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:13:11 2022 +0200
>    >
>    >     Add author name
>    >
>    > commit f5ec14f05384d76812fc0576df5e4af79336f4e6
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 11:59:23 2022 +0200
>    >
>    >     Add readme
>    > ```
>    {: .code-in}
>
{: .hands_on}

The output of `git log` is a great way to help you remember what you were doing.

> <hands-on-title>`git log -p`</hands-on-title>
> 1. Use `git log -p` to see the log, along with which lines were changed in each commit.
{: .hands_on}

But currently this log is pretty boring, so let's replace a line and see how that looks.

> <hands-on-title>Replace a line</hands-on-title>
> 1. Update your project description in the `readme.md`, you've been told you need to support completely different features.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sed -i s'/enables.*ROI/creates baking recipes/g' readme.md
>    > git add readme.md
>    > git commit -m 'Update project description'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Check what happened with the `git log -p`:
>
>    > <code-out-title>Output</code-out-title>
>    > ```bash
>    > $ git log -p
>    > commit 416a121dfcda14de0c2cb181f298b2c08950475f (HEAD -> main)
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:18:00 2022 +0200
>    >
>    >     Update project description
>    >
>    > diff --git a/readme.md b/readme.md
>    > index befc0c9..3b8899e 100644
>    > --- a/readme.md
>    > +++ b/readme.md
>    > @@ -1,6 +1,6 @@
>    >  My Project
>    >  Author: hexylena
>    > -This project enables stakeholders to experience synergistic effects and increase link up opportunities to improve quarterly and YOY ROI.
>    > +This project creates baking recipes.
>    >
>    >  # License
>    >  AGPL-3.0
>    >
>    > ```
>    {: .code-out}
>
>    This is a diff, a comparison between two versions of a file.
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
{: .hands_on}

### Who did that? `git blame` to the rescue

If you want to know who changed a specific line of a file, you can use `git blame` to find out it was probably your fault (as most of us experience when we check the logs.)

> <code-in-title>Bash</code-in-title>
> ```bash
> git blame readme.md
> ```
> {: data-cmd="true"}
{: .code-in}

> <code-in-title>Output</code-in-title>
> ```bash
> ^f5ec14f (Helena Rasche 2022-06-13 11:59:23 +0200 1) My Project
> 10355c01 (Helena Rasche 2022-06-13 12:13:11 +0200 2) Author: hexylena
> 416a121d (Helena Rasche 2022-06-13 12:18:00 +0200 3) This project creates baking recipes.
> 62f974ec (Helena Rasche 2022-06-13 12:13:16 +0200 4)
> 5d05eb3e (Helena Rasche 2022-06-13 12:13:21 +0200 5) # License
> 5d05eb3e (Helena Rasche 2022-06-13 12:13:21 +0200 6) AGPL-3.0
> ```
> here we can see for every line: which commit last affected it, who made that commit, and when.
{: .code-in}

## Branching

Git has the concept of branches which are most often used to manage development over time, before it's considered final. Until now you've seen `main` in your commits and commit logs (or maybe `master` if your git installation is a bit older.)

Oftentimes you'll see this pattern:

1. There is a main branch with a lot of history
1. You want to test out a new option, new configuration, new script you're working on
	1. So you make a branch
	1. Work on that branch
1. And merge it back into the `main` branch, once it's done.

This is especially relevant for any project that is shared with others, has a public view, or a deployed version of the code. There you don't want to affect anyone else using the project, or you don't want to affect the production deployment, until you're done making your changes.

> <hands-on-title>Create a new branch</hands-on-title>
>
> 1. `git switch -c <branch>` is the command used to create a new branch and switch to it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch -c test
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    > Switched to a new branch 'test'
>    > ```
>    {: .code-in}
>
{: .hands_on}

If you look around, you'll notice everything looks exactly the same! But in fact we are now on a different branch:

> <hands-on-title>See available branches</hands-on-title>
>
> 1. `git branch` lists our available branches, and puts an asterisk next to the one we're currently on.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git branch
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    >   main
>    > * test
>    >	```
>    {: .code-in}
>
{: .hands_on}

We're now on the `test` branch, so let's make a commit.

> <hands-on-title>Add a new file</hands-on-title>
> 1. Add a new file, let's call it `docs.md`. Write something into it, it doesn't matter much what.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "# Project Documentation" > docs.md
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Add it, commit it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git add docs.md
>    > git commit -m "Added documentation"
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

This file now only exists on the testing branch.

> <hands-on-title>Try Switching Branches</hands-on-title>
> 1. Try switching back and forth between the `main` and `test` branches, and check what's available on each!
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git branch
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch main
>    > ls
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```console
>    > readme.md
>    > ```
>    {: .code-in}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch test
>    > ls
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```console
>    > docs.md   readme.md
>    > ```
>    {: .code-in}
>
{: .hands_on}

Each branch has a different view of the repository, and might have different changes on it. Branches are really useful to keep track of work in progress, until it's done. In a single user environment however, most people often don't use them, but once you're collaborating with other's they're incredibly important!

### Merging

Once you're done with a branch, you can merge it into the main branch. This will take all of the work you did on that branch, and make it part of the main branch.

First, let's compare the two branches, to see what changed.

> <hands-on-title>Replacing argv.</hands-on-title>
>
> 1. Compare your current branch against the `main` branch with `git diff main`
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git diff main
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    > diff --git a/docs.md b/docs.md
>    > new file mode 100644
>    > index 0000000..384aaaa
>    > --- /dev/null
>    > +++ b/docs.md
>    > @@ -0,0 +1 @@
>    > +# Project Documentation
>    > ```
>    {: .code-in}
>
{: .hands_on}

We can see the output shows all of our changes compared to the main branch and it looks like what we want, so, let's merge it in.

> <hands-on-title>Merge the `test` branch into `main`</hands-on-title>
> 1. Switch to the main branch
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch main
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Merge in the test branch
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git merge test
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    > Updating 416a121..9a3387d
>    > Fast-forward
>    >  docs.md | 1 +
>    >  1 file changed, 1 insertion(+)
>    >  create mode 100644 docs.md
>    > ```
>    {: .code-in}
>
{: .hands_on}

This has merged all of the changes you made on the `test` branch into the `main` branch.

> <hands-on-title>Check the history</hands-on-title>
> 1. Check `git log -p` again to see the history.
{: .hands_on}

## Undo! Revert!

Oh no, you've decided you liked your original project description better. Let's find that commit and revert it.

> <hands-on-title>Find and revert the bad commit</hands-on-title>
>
> 1. Find the commit you want to revert, e.g. with `git log`, find the one named "Update project description" (or similar.)
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git log
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. We can use the `git revert` command to undo this commit.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git revert 416a121dfcda14de0c2cb181f298b2c08950475f
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

This generates a *new* commit, which reverts the older commit (and probably puts you in a text editor to edit the commit message).
This is not the only way to undo mistakes, but probably the easiest.

If you check your `git log` you'll see the change was undone in a second commit, reverting the first. So if you just look at the current files it appears we never undid it, but within the logs we can see the undo step.

With that you've got enough skills to track your own data/code/etc with git!

# Further Reading

- [git - the simple guide](https://rogerdudler.github.io/git-guide/index.html)
- [giteveryday(7) manual page, the most common commands most folks use every day](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/giteveryday.html)
- [git flight rules, a convenient "how do I do X" guide](https://github.com/k88hudson/git-flight-rules)
- [do you prefer git tutorials as comics? (cw: language)](https://jvns.ca/blog/2018/10/27/new-zine--oh-shit--git-/)


topic: data-science
tutorial: cli-basics
title: CLI basics
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["[]"]
tags_json: ["bash"]
content_hash: b0fee311784b3865d53bdf932dad13b3
last_modified: 2026-02-26T21:12:14.737663
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-basics/tutorial.html

content:
This tutorial will walk you through the basics of how to use the Unix command line.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) ["The Unix Shell"](https://swcarpentry.github.io/shell-novice/) lesson, which is licensed CC-BY 4.0. Adaptations have been made to make this work better in a GTN/Galaxy environment.
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Background


Humans and computers commonly interact in many different ways, such as through a keyboard and mouse,
touch screen interfaces, or using speech recognition systems.
The most widely used way to interact with personal computers is called a
**graphical user interface** (GUI).
With a GUI, we give instructions by clicking a mouse and using menu-driven interactions.

While the visual aid of a GUI makes it intuitive to learn,
this way of delivering instructions to a computer scales very poorly.
Imagine the following task:
for a literature search, you have to copy the third line of one thousand text files in one thousand
different directories and paste it into a single file.
Using a GUI, you would not only be clicking at your desk for several hours,
but you could potentially also commit an error in the process of completing this repetitive task.
This is where we take advantage of the Unix shell.
The Unix shell is both a **command-line interface** (CLI) and a scripting language,
allowing such repetitive tasks to be done automatically and fast.
With the proper commands, the shell can repeat tasks with or without some modification
as many times as we want.
Using the shell, the task in the literature example can be accomplished in seconds.

## Setup

Before we do anything, we'll get you setup with some test data which can help
guide your exploration of the CLI.

```bash
cd ~/
mkdir -p Desktop/
cd Desktop/
wget -c https://github.com/swcarpentry/shell-novice/raw/2929ba2cbb1bcb5ff0d1b4100c6e58b96e155fd1/data/shell-lesson-data.zip
unzip -u shell-lesson-data.zip
```


## The Shell

The shell is a program where users can type commands.
With the shell, it's possible to invoke complicated programs like climate modeling software
or simple commands that create an empty directory with only one line of code.
The most popular Unix shell is Bash (the Bourne Again SHell ---
so-called because it's derived from a shell written by Stephen Bourne).
Bash is the default shell on most modern implementations of Unix and in most packages that provide
Unix-like tools for Windows.

Using the shell will take some effort and some time to learn.
While a GUI presents you with choices to select, CLI choices are not automatically presented to you,
so you must learn a few commands like new vocabulary in a language you're studying.
However, unlike a spoken language, a small number of "words" (i.e. commands) gets you a long way,
and we'll cover those essential few today.

The grammar of a shell allows you to combine existing tools into powerful
pipelines and handle large volumes of data automatically. Sequences of
commands can be written into a *script*, improving the reproducibility of
workflows.

In addition, the command line is often the easiest way to interact with remote machines
and supercomputers.
Familiarity with the shell is near essential to run a variety of specialized tools and resources
including high-performance computing systems.
As clusters and cloud computing systems become more popular for scientific data crunching,
being able to interact with the shell is becoming a necessary skill.
We can build on the command-line skills covered here
to tackle a wide range of scientific questions and computational challenges.

Let's get started.

When the shell is first opened, you are presented with a **prompt**,
indicating that the shell is waiting for input.

```
$
```

The shell typically uses `$ ` as the prompt, but may use a different symbol.
In the examples for this lesson, we will **not** include this prompt!

Most importantly:
when typing commands, either from these lessons or from other sources,
*do not type the prompt*, only the commands that follow it.
Also note that after you type a command, you have to press the <kbd>Enter</kbd> key to execute it.

The prompt is followed by a **text cursor**, a character that indicates the position where your
typing will appear.
The cursor is usually a flashing or solid block, but it can also be an underscore or a pipe.
You may have seen it in a text editor program, for example.

So let's try our first command, `ls` which is short for listing.


> <hands-on-title>Open a Jupyter Terminal</hands-on-title>
> This tutorial will let you accomplish almost everything from this view, running code in the cells below directly in the training material. You can choose between running the code here, or opening up a terminal tab in which to run it.
>
> Here are some instructions for how to do this on various environments.
>
> **Jupyter** on UseGalaxy.\* and **MyBinder.org**
>
> 1. Use the File → New → Terminal menu to launch a terminal.
>
>    ![screenshot of jupyterlab showing the File menu expanded to show new and terminal option.](../../images/carpentries-cli/mybinder-launch.png)
>
> 2. Disable "Simple" mode in the bottom left hand corner, if it activated.
>
>    ![screenshot of jupyterlab showing a toggle labelled simple](../../images/carpentries-cli/mybinder-simple.png)
>
> 3. Drag one of the terminal or notebook tabs to the side to have the training materials and terminal side-by-side
>
>    ![screenshot of jupyterlab with notebook and terminal side-by-side.](../../images/carpentries-cli/mybinder-final.png)
>
> **CoCalc**
>
> 1. Use the *Split View* functionality of cocalc to split your view into two portions.
>
>    ![screenshot of cocalc button to split views](../../images/carpentries-cli/cocalc-split.png)
>
> 2. Change the view of one panel to a terminal
>
>    ![screenshot of cocalc swapping view port to that of a terminal](../../images/carpentries-cli/cocalc-terminal.png)
>
{: .hands_on}

This command will list the contents of the current directory:

```bash
ls
```

> <tip-title>Command not found</tip-title>
> If the shell can't find a program whose name is the command you typed, it
> will print an error message such as:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ ks
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > ks: command not found
> > ```
> {: .code-out}
>
> This might happen if the command was mis-typed or if the program corresponding to that command
> is not installed.
{: .tip }


## Nelle's Pipeline: A Typical Problem

Nelle Nemo, a marine biologist,
has just returned from a six-month survey of the
[North Pacific Gyre](http://en.wikipedia.org/wiki/North_Pacific_Gyre),
where she has been sampling gelatinous marine life in the
[Great Pacific Garbage Patch](http://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch).
She has 1520 samples that she's run through an assay machine to measure the relative abundance
of 300 proteins.
She needs to run these 1520 files through an imaginary program called `goostats.sh` she inherited.
On top of this huge task, she has to write up results by the end of the month so her paper
can appear in a special issue of *Aquatic Goo Letters*.

The bad news is that if she has to run `goostats.sh` by hand using a GUI,
she'll have to select and open a file 1520 times.
If `goostats.sh` takes 30 seconds to run each file, the whole process will take more than 12 hours
of Nelle's attention.
With the shell, Nelle can instead assign her computer this mundane task while she focuses
her attention on writing her paper.

The next few lessons will explore the ways Nelle can achieve this.
More specifically,
they explain how she can use a command shell to run the `goostats.sh` program,
using loops to automate the repetitive steps of entering file names,
so that her computer can work while she writes her paper.

As a bonus,
once she has put a processing pipeline together,
she will be able to use it again whenever she collects more data.

In order to achieve her task, Nelle needs to know how to:
- navigate to a file/directory
- create a file/directory
- check the length of a file
- chain commands together
- retrieve a set of files
- iterate over files
- run a shell script containing her pipeline

# Navigating Files and Directories

The part of the operating system responsible for managing files and directories
is called the **file system**.
It organizes our data into files,
which hold information,
and directories (also called 'folders'),
which hold files or other directories.

Several commands are frequently used to create, inspect, rename, and delete files and directories.
To start exploring them, we'll go to our open shell window.

First, let's find out where we are by running a command called `pwd`
(which stands for 'print working directory'). Directories are like *places*, at any time
while we are using the shell, we are in exactly one place called
our **current working directory**. Commands mostly read and write files in the
current working directory, i.e. 'here', so knowing where you are before running
a command is important. `pwd` shows you where you are:

```bash
pwd
```

Try running it now.

> <tip-title>Home Directory Variation</tip-title>
>
> The home directory path will look different on different operating systems.
>
> Linux         | OSX            | Jupyter (UseGalaxy/Binder) | CoCalc
> ---           | --             | ---                        | ---
> `/home/nelle` | `/Users/nelle` | `/home/joyvan`             | `/projects/<id>`
>
> On Windows, it will be similar to `C:\Documents and Settings\nelle` or `C:\Users\nelle`.
> (Note that it may look slightly different for different versions of Windows.)
> In future examples, we've used Mac output as the default - Linux and Windows
> output may differ slightly but should be generally similar.
>
> We will also assume that your `pwd` command returns your user's home directory.
> If `pwd` returns something different, you may need to navigate there using `cd`
> or some commands in this lesson will not work as written.
> See [Exploring Other Directories](#exploring-other-directories) for more details
> on the `cd` command.
{: .tip}

To understand what a 'home directory' is,
let's have a look at how the file system as a whole is organized.  For the
sake of this example, we'll be
illustrating the filesystem on our scientist Nelle's computer.  After this
illustration, you'll be learning commands to explore your own filesystem,
which will be constructed in a similar way, but not be exactly identical.

On Nelle's computer, the filesystem looks like this:

![The file system is made up of a root directory that contains sub-directories titled bin, data, users, and tmp](../../images/carpentries-cli/filesystem.svg)

At the top is the **root directory**
that holds everything else.
We refer to it using a slash character, `/`, on its own;
this character is the leading slash in `/Users/nelle`.

Inside that directory are several other directories:
`bin` (which is where some built-in programs are stored),
`data` (for miscellaneous data files),
`Users` (where users' personal directories are located),
`tmp` (for temporary files that don't need to be stored long-term),
and so on.

We know that our current working directory `/Users/nelle` is stored inside `/Users`
because `/Users` is the first part of its name.
Similarly,
we know that `/Users` is stored inside the root directory `/`
because its name begins with `/`.

> <tip-title>Slashes</tip-title>
>
> Notice that there are two meanings for the `/` character.
> When it appears at the front of a file or directory name,
> it refers to the root directory. When it appears *inside* a path,
> it's just a separator.
{: .tip}

Underneath `/Users`,
we find one directory for each user with an account on Nelle's machine,
her colleagues *imhotep* and *larry*.

![Like other directories, home directories are sub-directories underneath '/Users' like '/Users/imhotep', '/Users/larry' or '/Users/nelle'](../../images/carpentries-cli/home-directories.svg)

The user *imhotep*'s files are stored in `/Users/imhotep`,
user *larry*'s in `/Users/larry`,
and Nelle's in `/Users/nelle`.  Because Nelle is the user in our
examples here, therefore we get `/Users/nelle` as our home directory.
Typically, when you open a new command prompt, you will be in
your home directory to start.

Now let's learn the command that will let us see the contents of our
own filesystem.  We can see what's in our home directory by running `ls`:

```bash
ls
```

`ls` prints the names of the files and directories in the current directory.
We can make its output more comprehensible by using the `-F` **option**
(also known as a **switch** or a **flag**),
which tells `ls` to classify the output
by adding a marker to file and directory names to indicate what they are:
- a trailing `/` indicates that this is a directory
- `@` indicates a link
- `*` indicates an executable

Depending on your default options,
the shell might also use colors to indicate whether each entry is a file or
directory.

```bash
ls -F
```

Here,
we can see that our home directory contains only **sub-directories**.
Any names in our output that don't have a classification symbol
are plain old **files**.

> <tip-title>Real Terminal Tip: Clearing your terminal</tip-title>
>
> If your screen gets too cluttered, you can clear your terminal using the
> `clear` command. You can still access previous commands using <kbd>↑</kbd>
> and <kbd>↓</kbd> to move line-by-line, or by scrolling in your terminal.
{: .tip}


## Getting help

`ls` has lots of other **options**. There are two common ways to find out how
to use a command and what options it accepts:

1. We can pass a `--help` option to the command,
2. We can read its manual with `man`

### The `--help` option

Many bash commands, and programs that people have written that can be
run from within bash, support a `--help` option to display more
information on how to use the command or program.

```bash
ls --help
```

> <tip-title>Unsupported command-line options</tip-title>
> If you try to use an option (flag) that is not supported, `ls` and other commands
> will usually print an error message similar to:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > ls -j
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > ls: invalid option -- 'j'
> > Try 'ls --help' for more information.
> > ```
> {: .code-out}
>
{: .tip}

### The `man` command

The other way to learn about `ls` is to type

> <code-in-title>Bash</code-in-title>
> ```
> $ man ls
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> LS(1)                                                          User Commands                                                          LS(1)
>
> NAME
>        ls - list directory contents
>
> SYNOPSIS
>        ls [OPTION]... [FILE]...
>
> DESCRIPTION
>        List information about the FILEs (the current directory by default).  Sort entries alphabetically if none of -cftuvSUX nor --sort is
>        specified.
>
>        Mandatory arguments to long options are mandatory for short options too.
>
>        -a, --all
>               do not ignore entries starting with .
>
>        -A, --almost-all
>               do not list implied . and ..
>
>        --author
>               with -l, print the author of each file
>
>        -b, --escape
>               print C-style escapes for nongraphic characters
>
>        --block-size=SIZE
>               with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below
>
>        -B, --ignore-backups
>               do not list implied entries ending with ~
>
>        -c     with -lt: sort by, and show, ctime (time of last modification of file status information); with -l: show ctime  and  sort  by
>               name; otherwise: sort by ctime, newest first
>
>        -C     list entries by columns
>
>        --color[=WHEN]
>               colorize the output; WHEN can be 'always' (default if omitted), 'auto', or 'never'; more info below
>
> ```
{: .code-out}

This command will turn your terminal into a page with a description
of the `ls` command and its options.

To navigate through the `man` pages,
you may use <kbd>↑</kbd> and <kbd>↓</kbd> to move line-by-line,
or try <kbd>B</kbd> and <kbd>Spacebar</kbd> to skip up and down by a full page.
To search for a character or word in the `man` pages,
use <kbd>/</kbd> followed by the character or word you are searching for.
Sometimes a search will result in multiple hits.
If so, you can move between hits using <kbd>N</kbd> (for moving forward) and
<kbd>Shift</kbd>+<kbd>N</kbd> (for moving backward).

To **quit** the `man` pages, press <kbd>Q</kbd>.

> <tip-title>Manual pages on the web</tip-title>
>
> Of course, there is a third way to access help for commands:
> searching the internet via your web browser.
> When using internet search, including the phrase `unix man page` in your search
> query will help to find relevant results.
>
> GNU provides links to its
> [manuals](http://www.gnu.org/manual/manual.html) including the
> [core GNU utilities](http://www.gnu.org/software/coreutils/manual/coreutils.html),
> which covers many commands introduced within this lesson.
{: .tip}

> <question-title>Exploring More `ls` Flags</question-title>
>
> You can also use two options at the same time. What does the command `ls` do when used
> with the `-l` option? What about if you use both the `-l` and the `-h` option?
>
> Some of its output is about properties that we do not cover in this lesson (such
> as file permissions and ownership), but the rest should be useful
> nevertheless.
>
> > <solution-title></solution-title>
> > The `-l` option makes `ls` use a **l**ong listing format, showing not only
> > the file/directory names but also additional information, such as the file size
> > and the time of its last modification. If you use both the `-h` option and the `-l` option,
> > this makes the file size '**h**uman readable', i.e. displaying something like `5.3K`
> > instead of `5369`.
> {: .solution}
{: .question}

> <question-title>Listing in Reverse Chronological Order</question-title>
>
> By default, `ls` lists the contents of a directory in alphabetical
> order by name. The command `ls -t` lists items by time of last
> change instead of alphabetically. The command `ls -r` lists the
> contents of a directory in reverse order.
> Which file is displayed last when you combine the `-t` and `-r` flags?
> Hint: You may need to use the `-l` flag to see the
> last changed dates.
>
> > <solution-title></solution-title>
> > The most recently changed file is listed last when using `-rt`. This
> > can be very useful for finding your most recent edits or checking to
> > see if a new output file was written.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

## Exploring Other Directories

Not only can we use `ls` on the current working directory,
but we can use it to list the contents of a different directory.
Let's take a look at our current directory by running `ls -F ~/`,
i.e.,
the command `ls` with the `-F` **option** and the [**argument**][Arguments]  `/`.
The argument `~/` tells `ls` that
we want a listing of something other than our current working directory:

```bash
ls -F ~/
```

As you may now see, using a bash shell is strongly dependent on the idea that
your files are organized in a hierarchical file system.
Organizing things hierarchically in this way helps us keep track of our work:
it's possible to put hundreds of files in our home directory,
just as it's possible to pile hundreds of printed papers on our desk,
but it's a self-defeating strategy.

Now that we know the `shell-lesson-data` directory is located in current working directory, we
can do two things.

First, we can look at its contents, using the same strategy as before, passing
a directory name to `ls`:

```bash
ls -F shell-lesson-data
```

Second, we can actually change our location to a different directory, so
we are no longer located in
our home directory.

The command to change locations is `cd` followed by a
directory name to change our working directory.
`cd` stands for 'change directory',
which is a bit misleading:
the command doesn't change the directory;
it changes the shell's idea of what directory we are in.
The `cd` command is akin to double clicking a folder in a graphical interface to get into a folder.

Let's say we want to move to the `data` directory we saw above. We can
use the following series of commands to get there:

```bash
cd shell-lesson-data
```

These commands will move us into
the `shell-lesson-data` directory, then into the `data` directory.
You will notice that `cd` doesn't print anything. This is normal.
Many shell commands will not output anything to the screen when successfully executed.
But if we run `pwd` after it, we can see that we are now
in `/Users/nelle/Desktop/shell-lesson-data/data`.
If we run `ls -F` without arguments now,
it lists the contents of `/Users/nelle/Desktop/shell-lesson-data/data`,
because that's where we now are:

```bash
pwd
```

```bash
ls -F data/
```

We now know how to go down the directory tree (i.e. how to go into a subdirectory),
but how do we go up (i.e. how do we leave a directory and go into its parent directory)?
We might try the following:

```bash
cd shell-lesson-data
```

But we get an error! Why is this?

With our methods so far,
`cd` can only see sub-directories inside your current directory. There are
different ways to see directories above your current location; we'll start
with the simplest.

There is a shortcut in the shell to move up one directory level
that looks like this:

```bash
cd ..
```

`..` is a special directory name meaning
"the directory containing this one",
or more succinctly,
the **parent** of the current directory.
Sure enough,
if we run `pwd` after running `cd ..`, we're back in `/Users/nelle/Desktop/shell-lesson-data`:

```bash
pwd
```

The special directory `..` doesn't usually show up when we run `ls`. If we want
to display it, we can add the `-a` option to `ls -F`:

```bash
ls -F -a
```

`-a` stands for 'show all';
it forces `ls` to show us file and directory names that begin with `.`,
such as `..` (which, if we're in `/Users/nelle`, refers to the `/Users` directory).
As you can see,
it also displays another special directory that's just called `.`,
which means 'the current working directory'.
It may seem redundant to have a name for it,
but we'll see some uses for it soon.

Note that in most command line tools, multiple options can be combined
with a single `-` and no spaces between the options: `ls -F -a` is
equivalent to `ls -Fa`.

> <tip-title>Other Hidden Files</tip-title>
>
> In addition to the hidden directories `..` and `.`, you may also see a file
> called `.bash_profile`. This file usually contains shell configuration
> settings. You may also see other files and directories beginning
> with `.`. These are usually files and directories that are used to configure
> different programs on your computer. The prefix `.` is used to prevent these
> configuration files from cluttering the terminal when a standard `ls` command
> is used.
{: .tip}

These three commands are the basic commands for navigating the filesystem on your computer:
`pwd`, `ls`, and `cd`. Let's explore some variations on those commands. What happens
if you type `cd` on its own, without giving
a directory?

```bash
cd
```

How can you check what happened? `pwd` gives us the answer!

```bash
pwd
```

It turns out that `cd` without an argument will return you to your home directory,
which is great if you've gotten lost in your own filesystem.

Let's try returning to the `data` directory from before. Last time, we used
three commands, but we can actually string together the list of directories
to move to `data` in one step:

```bash
cd ~/Desktop/shell-lesson-data/data
```

Check that we've moved to the right place by running `pwd` and `ls -F`.

If we want to move up one level from the data directory, we could use `cd ..`.  But
there is another way to move to any directory, regardless of your
current location.

So far, when specifying directory names, or even a directory path (as above),
we have been using **relative paths**.  When you use a relative path with a command
like `ls` or `cd`, it tries to find that location from where we are,
rather than from the root of the file system.

However, it is possible to specify the **absolute path** to a directory by
including its entire path from the root directory, which is indicated by a
leading slash. The leading `/` tells the computer to follow the path from
the root of the file system, so it always refers to exactly one directory,
no matter where we are when we run the command.

This allows us to move to our `shell-lesson-data` directory from anywhere on
the filesystem (including from inside `data`). To find the absolute path
we're looking for, we can use `pwd` and then extract the piece we need
to move to `shell-lesson-data`.

> <code-in-title>Bash</code-in-title>
> ```
> $ pwd
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```bash
> /Users/nelle/
> ```
{: .code-out}

Then we could run something like:

```
$ cd /Users/nelle/shell-lesson-data
```

Run `pwd` and `ls -F` to ensure that we're in the directory we expect. If we're not, edit the command block below to make sure we're in the correct location (`~/Desktop/shell-lesson-data/`)

```
pwd
ls -F
```

> <tip-title>Two More Shortcuts</tip-title>
>
> The shell interprets a tilde (`~`) character at the start of a path to
> mean "the current user's home directory". For example, if Nelle's home
> directory is `/Users/nelle`, then `~/data` is equivalent to
> `/Users/nelle/data`. This only works if it is the first character in the
> path: `here/there/~/elsewhere` is *not* `here/there/Users/nelle/elsewhere`.
>
> Another shortcut is the `-` (dash) character. `cd` will translate `-` into
> *the previous directory I was in*, which is faster than having to remember,
> then type, the full path.  This is a *very* efficient way of moving
> *back and forth between two directories* -- i.e. if you execute `cd -` twice,
> you end up back in the starting directory.
>
> The difference between `cd ..` and `cd -` is
> that the former brings you *up*, while the latter brings you *back*.
>
> ----
> Try it out below!
{: .tip}

First navigate to `~/Desktop/shell-lesson-data` (you should already be there).

```bash
cd ~/Desktop/shell-lesson-data
```

Then `cd` into the `creatures` directory
```bash
cd creatures
```

Now if you run
```bash
cd -
```
you'll see you're back in `~/Desktop/shell-lesson-data`.
Run `cd -` again and you're back in `~/Desktop/shell-lesson-data/creatures`

> <question-title>Absolute vs Relative Paths</question-title>
>
> Starting from `/Users/amanda/data`,
> which of the following commands could Amanda use to navigate to her home directory,
> which is `/Users/amanda`?
>
> 1. `cd .`
> 2. `cd /`
> 3. `cd /home/amanda`
> 4. `cd ../..`
> 5. `cd ~`
> 6. `cd home`
> 7. `cd ~/data/..`
> 8. `cd`
> 9. `cd ..`
>
> > <solution-title></solution-title>
> > 1. No: `.` stands for the current directory.
> > 2. No: `/` stands for the root directory.
> > 3. No: Amanda's home directory is `/Users/amanda`.
> > 4. No: this command goes up two levels, i.e. ends in `/Users`.
> > 5. Yes: `~` stands for the user's home directory, in this case `/Users/amanda`.
> > 6. No: this command would navigate into a directory `home` in the current directory, if it exists.
> > 7. Yes: unnecessarily complicated, but correct.
> > 8. Yes: shortcut to go back to the user's home directory.
> > 9. Yes: goes up one level.
> {: .solution}
{: .question}

> <question-title>Relative Path Resolution</question-title>
>
> Using the filesystem diagram below, if `pwd` displays `/Users/thing`,
> what will `ls -F ../backup` display?
>
> 1.  `../backup: No such file or directory`
> 2.  `2012-12-01 2013-01-08 2013-01-27`
> 3.  `2012-12-01/ 2013-01-08/ 2013-01-27/`
> 4.  `original/ pnas_final/ pnas_sub/`
>
> ![A directory tree below the Users directory where '/Users' contains the directories 'backup' and 'thing'; '/Users/backup' contains 'original', 'pnas_final' and 'pnas_sub'; '/Users/thing' contains 'backup'; and '/Users/thing/backup' contains '2012-12-01', '2013-01-08' and '2013-01-27'](../../images/carpentries-cli/filesystem-challenge.svg)
>
> > <solution-title></solution-title>
> > 1. No: there *is* a directory `backup` in `/Users`.
> > 2. No: this is the content of `Users/thing/backup`, but with `..`, we asked for one level further up.
> > 3. No: see previous explanation.
> > 4. Yes: `../backup/` refers to `/Users/backup/`.
> {: .solution}
{: .question}

> <question-title>`ls` Reading Comprehension</question-title>
>
> Using the filesystem diagram below,
> if `pwd` displays `/Users/backup`,
> and `-r` tells `ls` to display things in reverse order,
> what command(s) will result in the following output:
>
> ```
> pnas_sub/ pnas_final/ original/
> ```
> {: .output}
>
> ![A directory tree below the Users directory where '/Users' contains the directories 'backup' and 'thing'; '/Users/backup' contains 'original', 'pnas_final' and 'pnas_sub'; '/Users/thing' contains 'backup'; and'/Users/thing/backup' contains '2012-12-01', '2013-01-08' and '2013-01-27'](../../images/carpentries-cli/filesystem-challenge.svg)
>
> 1.  `ls pwd`
> 2.  `ls -r -F`
> 3.  `ls -r -F /Users/backup`
>
> > <solution-title></solution-title>
> >  1. No: `pwd` is not the name of a directory.
> >  2. Yes: `ls` without directory argument lists files and directories in the current directory.
> >  3. Yes: uses the absolute path explicitly.
> {: .solution}
{: .question}


## General Syntax of a Shell Command
We have now encountered commands, options, and arguments,
but it is perhaps useful to formalise some terminology.

Consider the command below as a general example of a command,
which we will dissect into its component parts:

```
$ ls -F ~/Desktop/shell-lesson-data/
```

`ls` is the **command**, with an **option** `-F` and an
**argument** `/`.
We've already encountered options (also called **switches** or **flags**) which
either start with a single dash (`-`) or two dashes (`--`),
and they change the behavior of a command.
[Arguments] tell the command what to operate on (e.g. files and directories).
Sometimes options and arguments are referred to as **parameters**.
A command can be called with more than one option and more than one argument, but a
command doesn't always require an argument or an option.

Each part is separated by spaces: if you omit the space
between `ls` and `-F` the shell will look for a command called `ls-F`, which
doesn't exist. Also, capitalization can be important.
For example, `ls -s` will display the size of files and directories alongside the names,
while `ls -S` will sort the files and directories by size, as shown below:

```
$ ls -s Desktop/shell-lesson-data/data
total 116
 4 amino-acids.txt   4 animals.txt   4 morse.txt  12 planets.txt  76 sunspot.txt
 4 animal-counts     4 elements      4 pdb         4 salmon.txt
$ ls -S Desktop/shell-lesson-data/data
sunspot.txt  animal-counts  pdb        amino-acids.txt  salmon.txt
planets.txt  elements       morse.txt  animals.txt
```

Putting all that together, our command above gives us a listing
of files and directories in our shell-lesson-data folder.


### Nelle's Pipeline: Organizing Files

Knowing this much about files and directories,
Nelle is ready to organize the files that the protein assay machine will create.
First,
she creates a directory called `north-pacific-gyre`
(to remind herself where the data came from).
Inside that,
she creates a directory called `2012-07-03`,
which is the date she started processing the samples.
She used to use names like `conference-paper` and `revised-results`,
but she found them hard to understand after a couple of years.
(The final straw was when she found herself creating
a directory called `revised-revised-results-3`.)

> <tip-title>Sorting Output</tip-title>
>
> Nelle names her directories 'year-month-day',
> with leading zeroes for months and days,
> because the shell displays file and directory names in alphabetical order.
> If she used month names,
> December would come before July;
> if she didn't use leading zeroes,
> November ('11') would come before July ('7'). Similarly, putting the year first
> means that June 2012 will come before June 2013.
{: .tip}

Each of her physical samples is labelled according to her lab's convention
with a unique ten-character ID,
such as 'NENE01729A'.
This ID is what she used in her collection log
to record the location, time, depth, and other characteristics of the sample,
so she decides to use it as part of each data file's name.
Since the assay machine's output is plain text,
she will call her files `NENE01729A.txt`, `NENE01812A.txt`, and so on.
All 1520 files will go into the same directory.

Now in her current directory `shell-lesson-data`,
Nelle can see what files she has using the command:

```bash
ls north-pacific-gyre/2012-07-03/
```

This command is a lot to type,
but she can let the shell do most of the work through what is called **tab completion**.
If she types:

```bash
ls nor
```

and then presses <kbd>Tab</kbd> (the tab key on her keyboard),
the shell automatically completes the directory name for her:

```
$ ls north-pacific-gyre/
```
{: .language-bash}

If she presses <kbd>Tab</kbd> again,
Bash will add `2012-07-03/` to the command,
since it's the only possible completion.
Pressing <kbd>Tab</kbd> again does nothing,
since there are 19 possibilities;
pressing <kbd>Tab</kbd> twice brings up a list of all the files,
and so on.
This is called **tab completion**,
and we will see it in many other tools as we go on.


# Working with Files and Directories

We now know how to explore files and directories,
but how do we create them in the first place?

## Step one: see where we are and what we already have

Let's go back to our `shell-lesson-data` directory on the Desktop
and use `ls -F` to see what it contains:

```bash
pwd
```

```bash
ls -F
```

You should see folders like the following, if not, you're in the wrong place!

```
creatures/  data/  molecules/  north-pacific-gyre/  notes.txt  pizza.cfg  solar.pdf  writing/
```

## Create a directory

Let's create a new directory called `thesis` using the command `mkdir thesis`
(which has no output):

```bash
mkdir thesis
```

As you might guess from its name,
`mkdir` means 'make directory'.
Since `thesis` is a relative path
(i.e., does not have a leading slash, like `/what/ever/thesis`),
the new directory is created in the current working directory:

```bash
ls -F
```

Since we've just created the `thesis` directory, there's nothing in it yet:

```bash
ls -F thesis
```

Note that `mkdir` is not limited to creating single directories one at a time.
The `-p` option allows `mkdir` to create a directory with nested subdirectories
in a single operation:

```bash
mkdir -p project/data project/results
```

The `-R` option to the `ls` command will list all nested subdirectories within a directory.
Let's use `ls -FR` to recursively list the new directory hierarchy we just created in the
`project` directory:

```bash
ls -FR project
```

It should look like:

```
project/:
data/  results/

project/data:

project/results:
```

> <tip-title>Two ways of doing the same thing</tip-title>
> Using the shell to create a directory is no different than using a file explorer.
> If you open the current directory using your operating system's graphical file explorer,
> the `thesis` directory will appear there too.
> While the shell and the file explorer are two different ways of interacting with the files,
> the files and directories themselves are the same.
{: .tip}

> <tip-title>Good names for files and directories</tip-title>
>
> Complicated names of files and directories can make your life painful
> when working on the command line. Here we provide a few useful
> tips for the names of your files and directories.
>
> 1. Don't use spaces.
>
>    Spaces can make a name more meaningful,
>    but since spaces are used to separate arguments on the command line
>    it is better to avoid them in names of files and directories.
>    You can use `-` or `_` instead (e.g. `north-pacific-gyre/` rather than `north pacific gyre/`).
>    To test this out, try typing `mkdir north pacific gyre`and see what directory (or directories!)
>    are made when you check with `ls -F`.
>
> 2. Don't begin the name with `-` (dash).
>
>    Commands treat names starting with `-` as options.
>
> 3. Stick with letters, numbers, `.` (period or 'full stop'), `-` (dash) and `_` (underscore).
>
>    Many other characters have special meanings on the command line.
>    We will learn about some of these during this lesson.
>    There are special characters that can cause your command to not work as
>    expected and can even result in data loss.
>
> If you need to refer to names of files or directories that have spaces
> or other special characters, you should surround the name in quotes (`""`).
{: .tip}

## Creating a text file

Let's change our working directory to `thesis` using `cd`,
then run a text editor called Nano to create a file called `draft.txt`:

```
$ cd thesis
$ nano draft.txt
```

> <warning-title>Nano won't work in a Jupyter Notebook!</warning-title>
> Nano is an interactive, full screen editor. You'll need to switch to a proper
> console to do this step, if you haven't already.
>
> In JupyterLab you can find this under the top menu by clicking **File -> New -> Terminal**
{: .warning}

> <tip-title>Which Editor?</tip-title>
>
> When we say, "`nano` is a text editor" we really do mean "text": it can
> only work with plain character data, not tables, images, or any other
> human-friendly media. We use it in examples because it is one of the
> least complex text editors. However, because of this trait, it may
> not be powerful enough or flexible enough for the work you need to do
> after this workshop. On Unix systems (such as Linux and macOS),
> many programmers use [Emacs](http://www.gnu.org/software/emacs/) or
> [Vim](http://www.vim.org/) (both of which require more time to learn),
> or a graphical editor such as
> [Gedit](http://projects.gnome.org/gedit/). On Windows, you may wish to
> use [Notepad++](http://notepad-plus-plus.org/).  Windows also has a built-in
> editor called `notepad` that can be run from the command line in the same
> way as `nano` for the purposes of this lesson.
>
> No matter what editor you use, you will need to know where it searches
> for and saves files. If you start it from the shell, it will (probably)
> use your current working directory as its default location. If you use
> your computer's start menu, it may want to save files in your desktop or
> documents directory instead. You can change this by navigating to
> another directory the first time you "Save As..."
{: .tip}

Let's type in a few lines of text.
Once we're happy with our text, we can press <kbd>Ctrl</kbd>+<kbd>O</kbd>
(press the <kbd>Ctrl</kbd> or <kbd>Control</kbd> key and, while
holding it down, press the <kbd>O</kbd> key) to write our data to disk
(we'll be asked what file we want to save this to:
press <kbd>Return</kbd> to accept the suggested default of `draft.txt`).

![screenshot of nano text editor in action](../../images/carpentries-cli/nano-screenshot.png)

Once our file is saved, we can use <kbd>Ctrl</kbd>+<kbd>X</kbd> to quit the editor and
return to the shell.

> <tip-title>Control, Ctrl, or ^ Key</tip-title>
>
> The Control key is also called the 'Ctrl' key. There are various ways
> in which using the Control key may be described. For example, you may
> see an instruction to press the <kbd>Control</kbd> key and, while holding it down,
> press the <kbd>X</kbd> key, described as any of:
>
> * `Control-X`
> * `Control+X`
> * `Ctrl-X`
> * `Ctrl+X`
> * `^X`
> * `C-x`
>
> In nano, along the bottom of the screen you'll see `^G Get Help ^O WriteOut`.
> This means that you can use `Control-G` to get help and `Control-O` to save your
> file.
{: .tip}

`nano` doesn't leave any output on the screen after it exits,
but `ls` now shows that we have created a file called `draft.txt`:

```bash
ls
```

> <question-title>Creating Files a Different Way</question-title>
>
> We have seen how to create text files using the `nano` editor.
> Now, try the following command:
>
> ```bash
> touch my_file.txt
> ```
>
> 1.  What did the `touch` command do?
>     When you look at your current directory using the GUI file explorer,
>     does the file show up?
>
> 2.  Use `ls -l` to inspect the files.  How large is `my_file.txt`?
>
> 3.  When might you want to create a file this way?
>
> > <solution-title></solution-title>
> > 1.  The `touch` command generates a new file called `my_file.txt` in
> >     your current directory.  You
> >     can observe this newly generated file by typing `ls` at the
> >     command line prompt.  `my_file.txt` can also be viewed in your
> >     GUI file explorer.
> >
> > 2.  When you inspect the file with `ls -l`, note that the size of
> >     `my_file.txt` is 0 bytes.  In other words, it contains no data.
> >     If you open `my_file.txt` using your text editor it is blank.
> >
> > 3.  Some programs do not generate output files themselves, but
> >     instead require that empty files have already been generated.
> >     When the program is run, it searches for an existing file to
> >     populate with its output.  The touch command allows you to
> >     efficiently generate a blank text file to be used by such
> >     programs.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <tip-title>What's In A Name?</tip-title>
>
> You may have noticed that all of Nelle's files are named 'something dot
> something', and in this part of the lesson, we always used the extension
> `.txt`.  This is just a convention: we can call a file `mythesis` or
> almost anything else we want. However, most people use two-part names
> most of the time to help them (and their programs) tell different kinds
> of files apart. The second part of such a name is called the
> **filename extension** and indicates
> what type of data the file holds: `.txt` signals a plain text file, `.pdf`
> indicates a PDF document, `.cfg` is a configuration file full of parameters
> for some program or other, `.png` is a PNG image, and so on.
>
> This is *just a convention*, albeit an important one. Files contain
> bytes: it's up to us and our programs to interpret those bytes
> according to the rules for plain text files, PDF documents, configuration
> files, images, and so on.
>
> Naming a PNG image of a whale as `whale.mp3` doesn't somehow
> magically turn it into a recording of whale song, though it *might*
> cause the operating system to try to open it with a music player
> when someone double-clicks it.
{: .tip}

## Moving files and directories
Returning to the `shell-lesson-data` directory,

```bash
cd ~/Desktop/shell-lesson-data/
```

In our `thesis` directory we have a file `draft.txt`
which isn't a particularly informative name,
so let's change the file's name using `mv`,
which is short for 'move':

```bash
mv thesis/draft.txt thesis/quotes.txt
```

The first argument tells `mv` what we're 'moving',
while the second is where it's to go.
In this case,
we're moving `thesis/draft.txt` to `thesis/quotes.txt`,
which has the same effect as renaming the file.
Sure enough,
`ls` shows us that `thesis` now contains one file called `quotes.txt`:

```bash
ls thesis
```

One must be careful when specifying the target file name, since `mv` will
silently overwrite any existing file with the same name, which could
lead to data loss. An additional option, `mv -i` (or `mv --interactive`),
can be used to make `mv` ask you for confirmation before overwriting.

Note that `mv` also works on directories.

Let's move `quotes.txt` into the current working directory.
We use `mv` once again,
but this time we'll use just the name of a directory as the second argument
to tell `mv` that we want to keep the filename
but put the file somewhere new.
(This is why the command is called 'move'.)
In this case,
the directory name we use is the special directory name `.` that we mentioned earlier.

```bash
mv thesis/quotes.txt .
```

The effect is to move the file from the directory it was in to the current working directory.
`ls` now shows us that `thesis` is empty:

```bash
ls thesis
```

Alternatively, we can confirm the file `quotes.txt` is no longer present in the `thesis` directory
by explicitly trying to list it:

```bash
ls thesis/quotes.txt
```

`ls` with a filename or directory as an argument only lists the requested file or directory.
If the file given as the argument doesn't exist, the shell returns an error as we saw above.
We can use this to see that `quotes.txt` is now present in our current directory:

```bash
ls quotes.txt
```

> <question-title>Moving Files to a new folder</question-title>
>
> After running the following commands,
> Jamie realizes that she put the files `sucrose.dat` and `maltose.dat` into the wrong folder.
> The files should have been placed in the `raw` folder.
>
> ```
> $ ls -F
>  analyzed/ raw/
> $ ls -F analyzed
> fructose.dat glucose.dat maltose.dat sucrose.dat
> $ cd analyzed
> ```
>
> Fill in the blanks to move these files to the `raw/` folder
> (i.e. the one she forgot to put them in)
>
> ```bash
> mv sucrose.dat maltose.dat ____/____
> ```
>
> > <solution-title></solution-title>
> > ```
> > $ mv sucrose.dat maltose.dat ../raw
> > ```
> > {: .language-bash}
> > Recall that `..` refers to the parent directory (i.e. one above the current directory)
> > and that `.` refers to the current directory.
> {: .solution}
{: .question}

## Copying files and directories

The `cp` command works very much like `mv`,
except it copies a file instead of moving it.
We can check that it did the right thing using `ls`
with two paths as arguments --- like most Unix commands,
`ls` can be given multiple paths at once:

```bash
cp quotes.txt thesis/quotations.txt
ls quotes.txt thesis/quotations.txt
```

We can also copy a directory and all its contents by using the
[recursive](https://en.wikipedia.org/wiki/Recursion) option `-r`,
e.g. to back up a directory:

```bash
cp -r thesis thesis_backup
```

We can check the result by listing the contents of both the `thesis` and `thesis_backup` directory:

```bash
ls thesis thesis_backup
```

> <question-title>Renaming Files</question-title>
>
> Suppose that you created a plain-text file in your current directory to contain a list of the
> statistical tests you will need to do to analyze your data, and named it: `statstics.txt`
>
> After creating and saving this file you realize you misspelled the filename! You want to
> correct the mistake, which of the following commands could you use to do so?
>
> 1. `cp statstics.txt statistics.txt`
> 2. `mv statstics.txt statistics.txt`
> 3. `mv statstics.txt .`
> 4. `cp statstics.txt .`
>
> > <solution-title></solution-title>
> > 1. No.  While this would create a file with the correct name,
> > the incorrectly named file still exists in the directory
> > and would need to be deleted.
> > 2. Yes, this would work to rename the file.
> > 3. No, the period(.) indicates where to move the file, but does not provide a new file name;
> > identical file names
> > cannot be created.
> > 4. No, the period(.) indicates where to copy the file, but does not provide a new file name;
> > identical file names cannot be created.
> {: .solution}
{: .question}

> <question-title>Moving and Copying</question-title>
>
> What is the output of the closing `ls` command in the sequence shown below?
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > pwd
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > /Users/jamie/data
> > ```
> {: .code-out}
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > ls
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > proteins.dat
> > ```
> {: .code-out}
>
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > mkdir recombined
> > mv proteins.dat recombined/
> > cp recombined/proteins.dat ../proteins-saved.dat
> > ls
> > ```
> {: .code-in}
>
> 1.   `proteins-saved.dat recombined`
> 2.   `recombined`
> 3.   `proteins.dat recombined`
> 4.   `proteins-saved.dat`
>
> > <solution-title></solution-title>
> > We start in the `/Users/jamie/data` directory, and create a new folder called `recombined`.
> > The second line moves (`mv`) the file `proteins.dat` to the new folder (`recombined`).
> > The third line makes a copy of the file we just moved.
> > The tricky part here is where the file was copied to.
> > Recall that `..` means 'go up a level', so the copied file is now in `/Users/jamie`.
> > Notice that `..` is interpreted with respect to the current working
> > directory, **not** with respect to the location of the file being copied.
> > So, the only thing that will show using ls (in `/Users/jamie/data`) is the recombined folder.
> >
> > 1. No, see explanation above. `proteins-saved.dat` is located at `/Users/jamie`
> > 2. Yes
> > 3. No, see explanation above. `proteins.dat` is located at `/Users/jamie/data/recombined`
> > 4. No, see explanation above. `proteins-saved.dat` is located at `/Users/jamie`
> >
> {: .solution}
{: .question}

## Removing files and directories

Returning to the `shell-lesson-data` directory,
let's tidy up this directory by removing the `quotes.txt` file we created.
The Unix command we'll use for this is `rm` (short for 'remove'):

```bash
rm quotes.txt
```

We can confirm the file has gone using `ls`:

```bash
ls quotes.txt
```

>


topic: data-science
tutorial: data-manipulation-olympics-sql
title: Data Manipulation Olympics - SQL
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa", "sql"]
content_hash: 6ace3676a678bd26e21ca46b29b0cfbb
last_modified: 2026-02-26T21:12:14.847485
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics-sql/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->


Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

| Operation                | Description                                                         | Galaxy Tool                                   |
| ------------------------ | ------------------------------------------------------------------- | --------------------------------------------- |
| Compute on rows          | to derive new column values from existing ones                      | `SELECT x * 2 FROM y`                         |
| Concatenate datasets     | one after the other                                                 | `SELECT * FROM x; union all; SELECT * FROM y` |
| Counting                 | Count occurrences of values in a column                             | `SELECT count(x) FROM y where x = 'value'`    |
| Cut Columns              | By header name                                                      | `SELECT x, y, z FROM a`                       |
| Filter                   | Remove rows based on values in one or more columns                  | `... WHERE x = 'value'`                       |
| Find and Replace         | in a specific column                                                | `REPLACE()`, `regexp_replace` in postgresql   |
| Group on a column        | And perform simple operations (count, mean, min, max etc)           | `... GROUP BY x ...`                          |
| Join two Datasets        | side by side on a specified field                                   | `SELECT * FROM x, y JOIN x.id = y.id`         |
| Select First lines       | Good for finding top 10s or saving header lines                     | `... LIMIT 10`                                |
| Sort on a column         | Change the order of the rows based on values in one or more columns | `... ORDER BY x ASC`                          |
| Unique                   | Remove duplicate rows                                               | `SELECT DISTINCT x FROM y`                    |


In this tutorial, these functions are explained in more detail, and we provide some exercises for you to practice.

# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"]({% link topics/introduction/tutorials/data-manipulation-olympics/images/cover.jpg %})


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!

# Preamble
```sql
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c {{page.zenodo_link}}/files/olympics.db
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///olympics.db")
%load_ext sql
%sql sqlite:///olympics.db
%config SqlMagic.displaycon=False
```

# Download Data

Before we can do any manipulation, we will need some data. Let's download our table with Olympics results now.

```sql
SELECT * FROM olympics LIMIT 10;
```

And now we can start querying the database:

```sql
SELECT
    name
FROM
    sqlite_schema
```

> <question-title></question-title>
>
> 1. What tables are available?
> 2. How are they structured?
>
> > <solution-title></solution-title>
> >
> > 1. `countries`, `olympics`, `olympics_2022`
> > 2. Each are tables with 10 or more columns.
> {: .solution}
{: .question}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The `olympics` table contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.db` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.

# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

In `SQL` we can use the `ORDER BY` clause. We'll start by limiting our results, as every table in this dataset is quite large.

```sql
SELECT NOC, `CLDR display name` FROM countries LIMIT 30;
```

You can use `ORDER BY column-name ASC` or `ORDER BY column-name DESC` to sort the data ascending or descending.

```sql
SELECT NOC, `CLDR display name` FROM countries ORDER BY NOC ASC LIMIT 30;
```

```sql
SELECT NOC, `CLDR display name` FROM countries ORDER BY NOC DESC LIMIT 30;
```

So let's sort our file in chronological order, based on the year of the Olympic games:

> <question-title></question-title>
>
> 1. Which column contains the year?
>
> > <solution-title></solution-title>
> >
> > 1. `year`
> >
> {: .solution}
{: .question}

```sql
SELECT * FROM olympics ORDER BY year LIMIT 30;
```

If we wanted to do it in reverse, we could just use `order by year desc`

```sql
SELECT * FROM olympics ORDER BY year DESC LIMIT 30;
```

> <question-title></question-title>
>
> 1. Write a query to access only the first entry.
> 2. Which athlete is listed at the top of the file now?
>
> > <solution-title></solution-title>
> > 1. We can use `LIMIT` for this.
> >    ```sql
> >    select * from olympics order by year limit 1;
> >    ```
> > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
> >
> {: .solution}
{: .question}

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, an then within each year, we sort again alphabetically by name.

We will sort the file in chronological order based on the year of the Olympic games

```sql
SELECT * FROM olympics ORDER BY year, name LIMIT 30;
```

> <question-title></question-title>
>
> Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>
> > <solution-title></solution-title>
> >
> > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
> >
> {: .solution}
{: .question}


## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> >
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <tip-title>Removing null values</tip-title>
> > This will be covered more during the Filtering section, but for now simply use this filter:
> > ```sql
> > SELECT * from olympics where height is not null ... ;
> > ```
> {: .tip}
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We can use `.height`, and because we want the tallest on top, we will need to sort in *descending* (decreasing) order. Unfortunately you might discover there are null values.
> > 2. Rerun the same query as step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Lyton Mphande from Seol is the shortest at 127 cm.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > 1. `select * from olympics  where height is not null order by height desc limit 1;`
> > 2. `select * from olympics  where height is not null order by height asc limit 1`
> > 3. `select * from olympics  where height is not null order by year desc, height desc limit 1`
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).

We will filter the file to show only winter Olympics
Look at the `olympics` table and answer the following questions

> <question-title></question-title>
>
> 1. Which key contains this information?
> 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>
> > <solution-title></solution-title>
> >
> > 1. `season`
> > 2. The values can be `Summer` or `Winter` (`select distinct season from olympics`)
> >
> {: .solution}
{: .question}

We'll be using the `WHERE` filter to select entries matching specific conditions:

> <question-title></question-title>
>
> 1. How would you write the expressions for the following conditions:
>    1. `enrolled` must be 'Yes'
>    2. `age` must be smaller than 75
>    3. `height` cannot be null
>    4. `birthplace` cannot be empty
>
> 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    How would you write expressions for the following filtering conditions:
>    1. `height` is larger than 200 or smaller than 160
>    2. `height` is larger than 200 and smaller than 210
>
> > <solution-title></solution-title>
> >
> > 1. The answers are:
> >    1. `select * from olympics where enrolled = 'Yes'`
> >    2. `select * from olympics where age < 75`
> >    3. `select * from olympics where height is not null``
> >    4. `select * from olympics where birthplace != ""`
> >
> > 2. The answers are:
> >    1. `select * from olympics where height > 200 or height < 160`
> >    2. `select * from olympics where height > 200 and height < 210`
> >
> {: .solution}
{: .question}



Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics. Make sure it is contained in an array, in case we want to do further sorting.

```sql
CREATE TABLE winter AS SELECT * FROM olympics WHERE season = 'Winter'
```

> <question-title></question-title>
>
> How many entries are in this file? (Hint: use `count(*)`)
>
> > <solution-title></solution-title>
> >
> > 44,680
> >
> {: .solution}
{: .question}


**Repeat** the step for the Summer Olympics

```sql
CREATE TABLE summer AS SELECT * FROM olympics WHERE season = 'Summer'
```

> <question-title></question-title>
>
> 1. How many lines do you expect in the this file?
> 2. How many lines are in this file? Were you right?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Use the `count(*)` select
> > 2. Be careful to consider whether these counts include the header line of the file or not
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > 1. The original file has 234,522 entries, and the Winter Olympics had 44,680 entries. So we would expect 234,522 - 44,680 = 189,842 rows of data.
> > It is always useful to take a moment to think about the expected outcome, this makes it easier to spot mistakes and will save you time in the long run.
> >
> {: .solution}
{: .question}

## Exercises

Ok, time to train! let's see if you can use the `select` filter to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm? (inclusive)
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `SELECT count(*) FROM olympics WHERE medal == "Gold"`)
> >  2. 24,633  (Expression: `SELECT count(*) FROM olympics WHERE medal == "Gold" or medal == "Silver" or medal == "Bronze")`, or `medal != null`)
> >  3. 131     (Expression: `SELECT count(*) FROM olympics WHERE medal == "Gold" and year == 2018` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `SELECT count(*) FROM olympics WHERE medal is not null and height >= 170 and height <=180`)
> >  5. 2,333   (Expression: `SELECT count(*) FROM olympics WHERE medal is not null and (height < 160 or height > 190)` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count.

Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)

We'll need to use the `group by` syntax which takes a key, and then groups by those values.

> <question-title></question-title>
>
> 1. How many different Olympic games are in our file?
> 2. Which Olympic games had the most participations? (Tip: use order by)
>
> > <solution-title></solution-title>
> >
> > 1. 52 games (`select count(*), games from olympics group by games`)
> >
> >    The resulting file looks something like:
> >
> >     ```
> >     615	1896 Summer Olympics
> >     2503	1900 Summer Olympics
> >     2643	1904 Summer Olympics
> >     3213	1908 Summer Olympics
> >     4610	1912 Summer Olympics
> >     3448	1920 Summer Olympics
> >     5242	1924 Summer Olympics
> >     358	1924 Winter Olympics
> >     4493	1928 Summer Olympics
> >     ...
> >     ```
> >
> > 2. 1996 Summer Olympics. (10501 participations)
> >
> {: .solution}
{: .question}

You may have guessed that like `order by`, that we could have selected multiple columns in the `group by` step. This lets us count on combinations of columns.

Let's try counting the number of men and women in each olympic games.

```sql
select count(*), games, sex from olympics group by games, sex
```

> <question-title></question-title>
>
> You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>
> 1. How many women were in the first Olympic games?
>
> 2. Which Olympic games had the most women participants?
>
> > <solution-title></solution-title>
> >
> > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice, in this query. Do you know any way we could query that? Try it out!)
> >    The results looks something like this:
> >    ```
> >    2	F	1896 Summer Olympics
> >    43	F	1900 Summer Olympics
> >    17	F	1904 Summer Olympics
> >    55	F	1908 Summer Olympics
> >    97	F	1912 Summer Olympics
> >    132	F	1920 Summer Olympics
> >    269	F	1924 Summer Olympics
> >    ```
> >
> > 2. 2020 Summer Olympics (4652)
> >
> {: .solution}
{: .question}

Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting query above, we would get resultsfor each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

We can use the `distinct` filter in our pipeline to discover this. First let's do our group by and iterate over each resulting group:

```sql
select games, sport from olympics group by games;
```

And let's count all of their appearances

```sql
select games, count(sport) as sports from olympics group by games;
```

But those results still aren't distinct, those numbers are far too high. So let's use distinct:

```sql
select games, count(distinct sport) as sports from olympics group by games;
```

We're almost there! Let's sort this

```sql
select games, count(distinct sport) as sports from olympics group by games order by sports asc;
```

> <question-title></question-title>
>
> 2. How many sport were in the first Olympics? How many in the latest?
> 3. Which Olympics had the most different sports?
>
> > <solution-title></solution-title>
> >
> > 2. 10 and 38.
> > 3. The 2020 Summer Olympics had the most different sports (38)
> >
> {: .solution}
{: .question}

Save the output as something descriptive.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting instances of a key, we can use `group by team` and then loop over that to print out the length, and the team name of each of those items.
> > 2. This is basically the same question as "how many women" participated, try modifying that query.
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations (`select team, count(team) as count from olympics group by team order by count desc;`)
> >  2. 15 and 250. (`select games, count(distinct team) as teams from olympics group by games;`)
> >
> {: .solution}
{: .question}


# Grouping

Often we may want to group rows based on a value in a column, and perform some operation on the resulting rows. For example we would like to group the olympics data by one value (e.g. year, country, sport), and determine some value for each group (e.g. number of medals won, average age of athletes, etc).

In the [counting](#counting) section of this tutorial we show how to get answers that require a count (e.g. number of medals won), but sometimes we want to do something more complex, like calculating the average height of athletes in a group, say per country or per sport. This section will show some example of these types of questions.

We can use continue to use group by for this, but now we'll need the max and min aggregate operations. Essentially every time we use `group by` we need to use an aggregation like finding the maximum, minimum, or counting the number of results.

> <hands-on-title>Tallest athlete per sport</hands-on-title>
>
> We would like to answer the following question: *How tall was the tallest athlete of each sport?*
>
> > <question-title></question-title>
> >
> > 1. How tall was the tallest athlete in basketball? And what about karate?
> > 2. Why do some sports have null values?
> >
> > > <solution-title></solution-title>
> > > ```sql
> > > select max(height), min(height),sport from olympics group by sport
> > > ```
> > >
> > > 1. Basketball's tallest athlete was 192cm. For Karate it is 163.
> > > 2. Our dataset had quite a number of `null` (unknown) values in the height column, especially for the earlier Olympics. These are preserved in the outputs.
> > {: .solution}
> {: .question}
{: .hands_on}


## Grouping on multiple columns

You may have noticed that we could also provide multiple columns to group on. If we do this, we can compute values for combinations of groups, such as sex and sport, to find e.g. the tallest woman in basketball or the shortest man per Olympics. There are also many more options for the computation we perform, so perhaps we are more interested not in the tallest athlete, but the average height. Let's perform some of these slightly more advanced queries now.


> <hands-on-title>Average height of men and women per sport</hands-on-title>
>
> The question we would like to answer here, is what is the average height for men and women per sport?
>
> ```sql
> select avg(height), sex, sport from olympics group by sex, sport;
> ```
>
> See if you can answer the following questions based on the output file.
>
> > <question-title></question-title>
> >
> > 1. What is the average height of women participating in archery?
> > 2. What is the average height of men participating in [ballooning](https://en.wikipedia.org/wiki/Ballooning_at_the_1900_Summer_Olympics)?
> > 3. Why do some values have `null` instead of a height?
> > 4. Why do some sports not have a value for one of the sexes?
> > 5. Can you find a sport where women were taller than the men? (Hint: it starts with the letter A)
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 167.25677031093 cm
> > > 2. 170 cm
> > > 3. If none of the rows in the group had height data available, it will output `nan` (not a number) instead. This is most common for sports that were only featured a few times in the early years of the Olympics.
> > > 4. Sports such as artistic swimming only exist for women, so no M appears in the data for that group, so there simply is no row for the mean height of men doing artistic swimming in our output.
> > > 5. [Art Competitions](https://en.wikipedia.org/wiki/Art_competitions_at_the_Summer_Olympics)
> > >
> > > If all went well, your output file should look something like:
> > >
> > > ```
> > > GroupBy(sport)	     GroupBy(sex)  mean(height)
> > > Aeronautics         M             nan
> > > Alpine Skiing       F             167.38324708926
> > > Alpine Skiing       M             178.18747142204
> > > Alpinism            M             nan
> > > Archery             F             167.25677031093
> > > Archery             M             178.5865470852
> > > Art Competitions    F             175.33333333333
> > > Art Competitions    M             173.97260273973
> > > Artistic Gymnastics F             156.15316901408
> > > ```
> > {: .solution}
> {: .question}
>
{: .hands_on}


## Exercises

> <question-title noprefix>Exercise: Grouping and computing</question-title>
>
> 1. How tall is the shortest woman Badminton player to win a gold medal?
> 2. What is the average height of athletes from team Denmark in the 1964 Olympics? (Note: 1964 has summer and winter olympics)
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We need to group on 3 columns: medal, sport and sex, and then select the `min`.
> > 2. We need to group on 2 columns: country (team) and year, then compute the average over height.
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 161 cm.
> >  2. mean height: 175.91304347826, standard deviation: 7.0335410308672`
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > ```sql
> > select min(height), medal, sport, sex from olympics group by medal, sport, sex;
> > select avg(height), team, games from olympics where team = "Denmark" group by team, games;
> > ```
> >
> {: .solution}
>
{: .question}


# Computing

Sometimes we want to use the data in our column to compute a new value, and add that to the table. For instance, for our dataset we could caluclate athtletes BMI (using height and weight columns), or their age at time of participation (from year of birth and year of the Olymics). By adding these computed values as a new colum to our datset, we can more easily query the dataset for these values. We can do these types of operations on the fly, and then if we like, store them as (temporary) tables.

As an example, let's calculate the age of each athlete at the time of participation, and add this as a new column to our dataset.

```sql
select year - birth_year as age, games from olympics LIMIT 30;
```

If we want to save that result to make it easier to query, then we have a couple options.

1. Don't store it, calculate on demand. Very storage efficient.

   ```sql
   select noc, name, ..., year - birth_year as age from olympics
   ```

1. Create a temporary table with this data, great if we only need it temporarily

   ```sql
   create temporary table olympics_ages as select *, year - birth_year as age from olympics
   ```

2. Create a new permanent table with this data

   ```sql
   create table olympics_ages as select *, year - birth_year as age from olympics
   ```

3. Update the existing table by adding it as a new column

   ```sql
   alter table olympics add column age int;
   update olympics set age = year - birth_year;
   ```

> <question-title></question-title>
>
> 1. How old was Arnaud Boetsch during his Olympic tennis participation?
>
> > <solution-title></solution-title>
> >
> > 1. Arnaud Boetsch is listed on the first two lines, who turned 27 the year of their Olympics.
> >
> {: .solution}
{: .question}

This was a simple computation, but much more complex mathematical expressions can be computed with this tool. In the exercise below, we will compute the BMI for each athlete as an example.


## Exercises

BMI stands for Body Mass Index, is a metric to provide a very crude measure of how healthy your weight is. The formula to compute BMI is:

$$ BMI = weight / (height^2) $$

(with weight in kilograms and height in meters).

> <tip-title>BMI is problematic</tip-title>
> [Adolphe Quetelet](https://en.wikipedia.org/wiki/Adolphe_Quetelet), who invented [BMI](https://en.wikipedia.org/wiki/Body_mass_index), was not a doctor of medicine, instead he was a Belgian astronomer, mathematician, statistician, and sociologist. His data consisted probably entirely of cisgender, white, european men and women {% cite Eknoyan_2007 %}.
> In his defense, he did argue it should not be used at individual levels, however it came to be used as such due to simplicity. (See [Wikipedia's History section](https://en.wikipedia.org/wiki/Body_mass_index#History)). BMI is a poor measure of health, especially for populations with high muscle content, and cannot be simply re-used as-is with anyone other than Anglo Saxons {% cite Caleyachetty_2021 %}, [Wikipedia:Limitations](https://en.wikipedia.org/wiki/Body_mass_index#Limitations).
>
> However as it is an easy to calculate metric, we include the calculation here.
{:.tip}

Let's compute this data for all athletes and add it as a new column!

> <question-title noprefix>Exercise: Calculating BMI</question-title>
>
> 1. How would you express this calculation in SQL?
>    - Remember that our height is in cm, and the formula expects height in meters
>    - And that we have null values, but we can ignore those as they will make the final calculation null as well
>
> 2. What is the BMI for Arnaud Boetsch?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - division is `/` and multiplication is ` * ` .
> > - Generally we cannot use `^`
> > - Parentheses may be required.
> > - use `select(.value != null)` to remove nulls.
> > - If that isn't eough, you can check that the type is a number with `(.value|type) == "number"` to ensure it really is a number and not e.g. a string.
> > - remember to wrap everything in `[...]` to retain the data shape
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> > 1. other variations are possible:
> >
> >    ```sql
> >    create temporary table olympics_bmi as select *, weight / (height / 100 * height / 100) as bmi from olympics;
> >    ```
> >
> > 2. 22.69
> >
> >    ```sql
> >    select * from olympics_bmi where name like 'Arnaud Boetsch';
> >    ```
> {: .solution}
{: .question}


# Find and Replace

Often you may need to change the contents of a file a bit to fit the expectations of an analysis tool. For instance, our database uses `null` for missing values, but other conventions included leaving the cell empty instead. Or, when working with chromosomal data, you may need to add or remove the `chr` prefix from a column before using it as input to a certain tool. In such situations, we can find all occurrences of a certain pattern in our file, and replace it with another value.

If we want to perform such a replacement on a single column in our data, we can use an update statement.

A few of the basics of regular expression, plus some links to further resources are given in the box below:

{% snippet faqs/galaxy/analysis_regular_expressions.md %}

Let's start with a simple example:
Our file uses a mix of `Athina` and `Athens` to indicate the Capital City of Greece in the `city` column.
Let's standardize this by replacing occurrences of `Athina` with `Athens`.

Let's start by filtering out the old spelling:

```sql
select * from olympics where city = 'Athina' limit 30;
```

Let's try replacing it:

```sql
update olympics
set city = 'Athens'
where city = 'Athina';
```

Look at the file before and after. Athlete 7 (Patrick Chila) near the top of the `olympics.tsv` file, had a value of Athina in the city column. Verify that it has been changed to Athens.

This was rather simple example, so let's try a few more examples with slightly more complex expressions.

## Exercises

You may have noticed that our file has a lot of missing data. Especially for the earlier years, things like height, weight and birthday of athletes was not registered, or simply not known. In some columns you see these missing values have been replaced with an `NA` (not available) value. In other columns (for example birth place), the cells have simply been left empty.

Different tools may expect different ways of handling missing data. So you may have to change your missing data from empty to `NA`, `NaN`, or something else, between analysis steps

> <hands-on-title>Fill empty cells</hands-on-title>
>
> We will now replace empty cells in the `birth_place` column, to use `null` instead.
>
> > <solution-title noprefix>Hints</solution-title>
> > Remember that comparison to nulls is done with `is` instead of `=`
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> > other variations are possible:
> > ```sql
> > update olympics set birth_place = null where birth_place = '';
> > ```
> {: .solution}
>
{: .hands_on}

Let's do another example, this one splitting and re-constructing strings.

Look at the `birth_day` column. It has values in a format like `12 December`. Suppose we have a tool that expects this data to be in the reverse format, `December 12`. We would not want to do this manually, but with sql we can make this replacement easily

We will now change the format in birthday column from `day month` to `month day`, as our boss is American and requested the silly format.

First we need to understand that sqlite does not ship a regex engine, thus we cannot use familiar regular expressions. Instead we can make use of `instr(string, search)` to find the location of a substring like ` `, identifying where the day stops and the month starts. Then we can use `substr(string, start)` and `substr(string, start, end)` to chop up our date string.

In SQL, concatenation is done with `||`.

> <question-title></question-title>
>
> 1. How do we captures both the day and the month?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We should use something like `substr(birth_day, instr(birth_day, ' '))`
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > ```sql
> > select
> >   birth_day,
> >   instr(birth_day, ' ') as idx,
> >   substr(birth_day, instr(birth_day, ' ')) as month,
> >   substr(birth_day, 0, instr(birth_day, ' ')) as day
> > from olympics
> > limit 10
> > ```
> > 
> > We can make our final query:
> > 
> > ```sql
> > select
> >   substr(birth_day, instr(birth_day, ' ')) || substr(birth_day, 0, instr(birth_day, ' ')) as birth_day_new
> > from olympics
> > limit 10
> > ```
> > And then store this as a new column (using `update`) or as a new table (using `create [temporary] table`)
> >
> {: .solution}
{: .question}


# Removing Columns

In sqlite you cannot remove columns, it is not supported. Proper databases like Postgres and MySQL support this operation.
Instead for sqlite you could select the columns you want to keep, create a new table from that, and then delete the original table.

Other databases:

```
alter table delete column NAME from olympics
```

# Unique

Sometimes, in the course of our data manipulations, we may end up with a file that has duplicate values. In order to filter these out, we can use the `distinct` filter.

Let's say we would like to create a list of all unique athletes (id and name).

First we will just select the `athlete_id` and `name` columns from our dataset

```sql
select name, athlete_id from olympics LIMIT 30
```

> <question-title></question-title>
>
> 1. Do you see duplication? Why is that?
>
> > <solution-title></solution-title>
> >
> > 1. Yes. For all athletes who participated more than once, the row will be identical.
> >
> {: .solution}
{: .question}

Now let's remove those duplicates.

```sql
select distinct name, athlete_id from olympics limit 30
```

> <question-title></question-title>
>
> How many unique athletes do we have? Note that you cannot count multiple columns, so choose one that is correct.
>
> > <solution-title></solution-title>
> > 94,733
> >
> > ```sql
> > select count(distinct athlete_id) from olympics;
> > ```
> >
> {: .solution}
{: .question}


# Joining Datasets

This database contains a lot of information, but we may want to add more information. For example if we had a file with information about each country (population, capital city, etc), we could join that information with our Olympics data, to get a single result with all information in every row.

For example, if we would like to be able to group by continent, to e.g. count athletes, medals etc per continent, we will have to add a `continent` column to our file. To do this we would need a second file that maps each country to the corresponding continent. This is what we will do in the next hands-on section.

We obtained country information data from [DataHub](https://datahub.io/core/country-codes). More information about this file can be found in the description there. It has 56 columns with a wide variety of data about each country (from country codes, to capital city, languages spoken, etc)

It is available in the countries table.

> <question-title></question-title>
>
> 2. Which keys(s) in this file are the same as in the `olympics.tsv` file?
>
> > <solution-title></solution-title>
> >
> > 2. Both files have a `NOC` column with the 3-letter country code (`NOC` stands for National Olympic Committee). However, one is lowercase.
> {: .solution}
{: .question}

We would now like to take our Olympics dataset as the basis, and add columns to every row of this file with some information about the country. In order to join, we will need to have one column that is shared between the two files, on which we can match. The `NOC` column is perfect for this because it is a defined standard. Both files also contain a column with the country name in it, which is also a possible candidate to use for joining, but because it is less standardised, it is safer to use the NOC column. For example, if one file uses "Netherlands", while the other uses "The Netherlands" to indicate the same country, the joining will fail for these rows. So always make sure the columns you join on are compatible!

We can use the join commands.

```sql
select * from olympics left join countries on olympics.noc = countries.NOC LIMIT 30
```

> <question-title></question-title>
>
> 1. What do you expect the output to look like? Were you right?
> 2. How many columns are in the resulting file? What about the NOC column?
> 3. What is a possible downside to this approach?
>
> > <solution-title></solution-title>
> >
> > 1. All the columns from the country information file are added to the end of each row of our olympics dataset
> > 2. Our olympics datset had 17 columns, the country information file has 56 columns. Therefore we have 17+56=73 columns columns in our resulting file. This also means the NOC column
> >    we joined on appears twice in our output.
> > 3. There is a lot of data duplication in the output now. The exact same country information is added to every line of every athlete from a certain country.
> >    This means much larger response size.
> >    If you do not need all these columns, it could save you a lot of space to select only specific columns that you require.
> >
> {: .solution}
{: .question}

# Concatenating

Concatenation of two files simple means putting the contents of the two files together, one after the other. Our dataset was created in 2021, but since then we've had another Olympic event, the 2022 Winter Olympics in Beijing. If we have the same data for this latest Olympics, we could simply add the rows from the 2022 games to our current file with data, in order to create a single file with all data from 1896 to 2022.

View the table `olympics_2022`, does it have the same structure as our original `olympics` table?

> <question-title></question-title>
>
> 1. Does the new table have the same structure?
> 2. Can we simply add the lines of the new table to the end of our existing olympics dataset?
>
> > <solution-title></solution-title>
> > 1. Yes, this file has all the same columns, in the same order, so concatenation should be relatively straightforward.
> > 2. Yes.
> >
> {: .solution}
{: .question}

Since this new dataset has the exact same structure (number and order of columns), we can simple add the lines from this file to the end of our existing `olympic` table.
For this, we'll need to use the `union all` which takes two separate sql queries and unifies the results.

```sql
select * from olympics
union all
select * from olympics_2022
LIMIT 30;
```

(Note: We are limiting the outputs to ensure your browser does not crash loading all of the data.)

Now this only works so simply because our two datasets had the same structure. If your data comes from different sources, you may have to do some additional data manipulation before you can union, e.g. to make sure the columns match, or how each file deals with missing data (empty cells, `NA`, `NaN` or something else).

# Conclusion

These operations covered in the tutorial are just a few examples of some of the most common operations. There are many more available. We encourage you to look around the documentation of sqlite or your database. The more comfortable you are performing these kinds of steps, the more you can get out of SQL!

# Exercises: Putting it all together!

This section provides a number of exercises that require you to combine two or more of the techniques you learned in this tutorial. This is a great way to practice your data manipulation skills. Full solutions are provided for every exercise (i.e. all tools and settings), but for many of these exercises there will be multiple solutions, so if you obtained the same results in a different way, that is correct too!

> <question-title noprefix>Exercise 1: Finding shortest/lightest athlete</question-title>
>
> If you have done exercises in the [sorting](#sorting) section, you noticed that finding the shortest athlete ever to compete was not easy,
> because all the rows with missing height data (`NA`) in the column were sorted to the top. We need to filter out these values first, then
> perform our sort, so that our answer is on top.
>
> 1. Find the shortest athlete ever to compete in the Olympics
> 2. Find the shortest athlete of the Winter Olympics
> 2. Find the lightest athlete of the *most recent* Summer Olympics
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. You will need to filter out the columns with (`NA`) in the height column first
> > 2. You will need to filter by season as well
> > 3. You will need to filter out missing data in the weight column, filter out Summer Olympics, then sort (by 2 columns)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Lyton Mphande and  Rosario Briones were both 127 cm tall, competing in boxing and gymnastics respectively
> >  2. Carolyn Krau was a 137 cm tall figure skater.
> >  3. Flávia Saraiva was the lightest athlete (31kg), she was a Artistic Gymnast from Brazil.
> >
> {: .solution}
>
> > <solution-title noprefix>Full solution</solution-title>
> >
> > 1. First we filter out the NA values from the height column:
> >
> >    ```sql
> >    select * from olympics where height is not null
> >    ```
> >
> >    Then we can sort by height, in ascending order to get the shortest athletes on top:
> >
> >    ```sql
> >    select * from olympics where height is not null order by height desc limit 10;
> >    ```
> >
> > 2. We can take the output from the first exercise, and filter for only Winter Olympics:
> >
> >    ```sql
> >    select * from olympics where height is not null and season = 'Winter' order by height desc limit 10;
> >    ```
> >
> > 3. First we filter out the NA values from the weight column:
> >
> >    ```sql
> >    select * from olympics where weight is not null and games = '2020 Summer Olympics' order by weight asc limit 10;
> >    ```
> >
> {: .solution}
>
{: .question}

Congratulations! You have now mastered the basics of data manipulation! There are a lot more data manipulation operations available that you may need. Please explore the tools for yourself, and check back with this tutorial often, we plan to add more sections and exercises over time!


topic: data-science
tutorial: gnmx-lecture4
title: Introduction to sequencing with Python (part three)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 9e181551cccbb124c8f0b94fef0ecb67
last_modified: 2026-02-26T21:12:14.881427
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture4/tutorial.html

content:
[![XKCD 2483](https://imgs.xkcd.com/comics/seven.png)](https://xkcd.com/2483)


Preclass prep: Chapters [5](https://greenteapress.com/thinkpython2/html/thinkpython2009.html) and [7](https://greenteapress.com/thinkpython2/html/thinkpython2011.html) from "Think Python"

> This material uses examples from notebooks developed by [Ben Langmead](https://langmead-lab.org/teaching-materials/)
{: .quote cite="https://langmead-lab.org/teaching-materials" }

# Prep

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Within JupyterLab start a new Python3 notebook
3. Open [this page](http://cs1110.cs.cornell.edu/tutor/#mode=edit) in a new browser tab

# Lists: Dynamic programming in sequence alignment

## Dynamic programming matrix as a 2D list

An excellent way to illustrate the utility of lists is to implement a dynamic programming algorithm for sequence alignment. Suppose we have two sequences that deliberately have different lengths:

$$
\texttt{G C T A T A C}$
$$

and 

$$
\texttt{G C G T A T G C}$
$$

Let's represent them as the following matrix where the first character $$\epsilon$$ represents an empty string:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon \\
\hline
G\\
\hline
 C\\
 \hline
G\\
\hline
T\\
\hline
A\\
 \hline
T\\
\hline
G\\
\hline
C
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

In this matrix, the cells are addressed as shown below. They filled in using the following logic:

$$
D[i,j] = min\begin{cases} 
          \color{green}{D[i-1,j] + 1} & \\
          \color{blue}{D[i,j-1] + 1} & \\
          \color{red}{D[i-1,j-1] + \delta(x[i-1],y[j-1])}
             \end{cases}
$$

where $$\color{green}{green}$$ is *upper* cell, $$\color{blue}{blue}$$ is *left* cell, and $$\color{red}{red}$$ is *upper-left* cell:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
            \hline
          \epsilon &  &  &  &  &  &  &  & \\
          \hline
          G & \\
          \hline
          C &  & & \color{red}{D[2,2]} & \color{green}{D[2,3]}\\
          \hline
          G &  & & \color{blue}{D[3,2]} & D[3,3]\\
          \hline
          T & \\
          \hline
          A & \\
          \hline
          T & \\
          \hline
          G & \\
          \hline
          C & 
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

## Initializing the matrix

Let's initialize the first column and first row of the matrix. Because the distance between a string and an empty string is equal to the length of the
string (e.g., a distance between, say, string $$\texttt{TCG}$$ and an empty string is 3) this resulting matrix will look like this:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
            \hline
          \epsilon & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7\\
          \hline
          G & 1\\
          \hline
          C & 2\\
          \hline
          G & 3\\
          \hline
          T & 4\\
          \hline
          A & 5\\
          \hline
          T & 6\\
          \hline
          G & 7\\
          \hline
          C & 8
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

This can be achieved with the following code:

```python
D = np.zeros((len(x)+1, len(y)+1), dtype=int)
D[0, 1:] = range(1, len(y)+1)
D[1:, 0] = range(1, len(x)+1)
```

## Filling the matrix

Now we can fill the entire matrix by using two nested loops: one iterating over $$i$$ coordinate (sequence $$x$$) and the other iterating over $$j$$ coordinate (sequence $$y$$):

```python
for i in range(1, len(x)+1):
    for j in range(1, len(y)+1):
        delt = 1 if x[i-1] != y[j-1] else 0
        D[i, j] = min(D[i-1, j-1]+delt, D[i-1, j]+1, D[i, j-1]+1)
```

Let's start with the cell between $$\texttt{G}$$ and $$\texttt{G}$$. Recall that:

$$
D[i,j] = min\begin{cases} 
          \color{green}{D[i-1,j] + 1} & \\
          \color{blue}{D[i,j-1] + 1} & \\
          \color{red}{D[i-1,j-1] + \delta(x[i-1],y[j-1])}
             \end{cases}
$$


where $$\delta(x[i-1],y[j-1]) = 0$$ if $$x[i-1] = y[j-1]$$ and $$1$$ otherwise. And now let's color each of the cells corresponding to each part of the above expression:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon & \color{red}0 & \color{green}1 & 2 & 3 & 4 & 5 & 6 & 7\\
\hline
G & \color{blue}1\\
          \hline
          C & 2\\
          \hline
          G & 3\\
          \hline
          T & 4\\
          \hline
          A & 5\\
          \hline
          T & 6\\
          \hline
          G & 7\\
          \hline
          C & 8
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

In this specific case:

$$
D[i,j] = min\begin{cases} 
          \color{green}{D[i-1,j] + 1}\ or\ 0+0=0 & \\
          \color{blue}{D[i,j-1] + 1}\ or\ 1+1=2 & \\
          \color{red}{D[i-1,j-1] + \delta(x[i-1],y[j-1])}\ or\ 1+1=2
             \end{cases}
$$


The minimum of 0, 2, and 2 will be 0, so we are putting zero into that cell:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon & \color{red}0 & \color{green}1 & 2 & 3 & 4 & 5 & 6 & 7\\
\hline
G & \color{blue}1 & \color{red}0\\
          \hline
          C & 2\\
          \hline
          G & 3\\
          \hline
          T & 4\\
          \hline
          A & 5\\
          \hline
          T & 6\\
          \hline
          G & 7\\
          \hline
          C & 8
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

Using this logic we can fill the entire matrix like this:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7\\
\hline
                 G & 1 & 0 & 1 & 2 & 3 & 4 & 5 & 6\\
          \hline
                 C & 2 & 1 & 0 & 1 & 2 & 3 & 4 & 5\\
          \hline
                 G & 3 & 2 & 1 & 1 & 2 & 3 & 4 & 5\\
          \hline
                 T & 4 & 3 & 2 & 1 & 2 & 2 & 3 & 4\\
          \hline
                 A & 5 & 4 & 3 & 2 & 1 & 2 & 2 & 3\\
          \hline
                 T & 6 & 5 & 4 & 3 & 2 & 1 & 2 & 3\\
          \hline
                 G & 7 & 6 & 5 & 4 & 3 & 2 & 2 & 3\\
          \hline
                 C & 8 & 7 & 6 & 5 & 4 & 3 & 3 & \color{red}2
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$


The lower rightmost cell highlighted in red is special. It contains the value for the edit distance between the two strings.
The following Python script implements this idea. You can see that it is essentially instantaneous:

```python
import numpy as np

def edDistDp(x, y):
    """ Calculate edit distance between sequences x and y using
        matrix dynamic programming.  Return matrix and distance. """
    D = np.zeros((len(x)+1, len(y)+1), dtype=int)
    D[0, 1:] = range(1, len(y)+1)
    D[1:, 0] = range(1, len(x)+1)
    for i in range(1, len(x)+1):
        for j in range(1, len(y)+1):
            delt = 1 if x[i-1] != y[j-1] else 0
            D[i, j] = min(D[i-1, j-1]+delt, D[i-1, j]+1, D[i, j-1]+1)
    return D,D[len(x),len(y)]
```
A graphical representation of the matrix between `GCGTATGCACGC` and `GCTATGCCACGC` looks like this:

![A DP matrix with Seaborn](https://i.imgur.com/veMfPFt.png)

This image is generated using [Seaborn](https://seaborn.pydata.org/index.html) package using matrix directly:

```python
sns.heatmap(D,annot=True,cmap="crest")
```


------

# Dictionaries: Translating sequences

Perhaps the best way to demonstrate the utility of dictionaries is by using DNA-to-Protein translation as an example. 

## Using dictionaries to translate DNA

The following dictionary maps codons to corresponding amino acid translations. In this case, codon is the *key* and amino acid is the *value*:


```python
table = {
        'ATA':'I', 'ATC':'I', 'ATT':'I', 'ATG':'M',
        'ACA':'T', 'ACC':'T', 'ACG':'T', 'ACT':'T',
        'AAC':'N', 'AAT':'N', 'AAA':'K', 'AAG':'K',
        'AGC':'S', 'AGT':'S', 'AGA':'R', 'AGG':'R',                
        'CTA':'L', 'CTC':'L', 'CTG':'L', 'CTT':'L',
        'CCA':'P', 'CCC':'P', 'CCG':'P', 'CCT':'P',
        'CAC':'H', 'CAT':'H', 'CAA':'Q', 'CAG':'Q',
        'CGA':'R', 'CGC':'R', 'CGG':'R', 'CGT':'R',
        'GTA':'V', 'GTC':'V', 'GTG':'V', 'GTT':'V',
        'GCA':'A', 'GCC':'A', 'GCG':'A', 'GCT':'A',
        'GAC':'D', 'GAT':'D', 'GAA':'E', 'GAG':'E',
        'GGA':'G', 'GGC':'G', 'GGG':'G', 'GGT':'G',
        'TCA':'S', 'TCC':'S', 'TCG':'S', 'TCT':'S',
        'TTC':'F', 'TTT':'F', 'TTA':'L', 'TTG':'L',
        'TAC':'Y', 'TAT':'Y', 'TAA':'_', 'TAG':'_',
        'TGC':'C', 'TGT':'C', 'TGA':'_', 'TGG':'W',
    }
```

Let's generate a random DNA sequence:


```python
import random
seq = "".join([random.choice('atcg') for x in range(100)])
```


```python
seq
```




    'agaccgtagcccaagtgcgtttgaatgtggctacttgggaggatttcattgcggtctgtctccgtacttgttattggtcttctttctgcattatgacgca'



To translate this sequence we write a code that uses a `for` loop that iterates over the DNA sequence in steps of 3, creating a codon at each iteration. If the codon is less than 3 letters long, the loop is broken. The resulting amino acid is then added to the `translation` string:


```python
translation = ""
for i in range(0, len(seq), 3):
    codon = seq[i:i+3].upper()
    if len(codon) < 3: break
    if codon in table:
        translation += table[codon]
    else:
        translation += "X"

print("Translation:", translation)
```

    Translation: RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R


Note that the code uses the `upper()` method to ensure the codon is in uppercase since the table dictionary is case-sensitive. Additionally, the code checks if the codon is in the table dictionary and if not, it adds the letter "X" to the translation. This is a common way to represent unknown or stop codons in a protein sequence.


```python
translation
```




    'RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R'



Now we define a function that would perform translation so that we can reuse it later:


```python
def translate(seq):
    translation = ''
    table = {
        'ATA':'I', 'ATC':'I', 'ATT':'I', 'ATG':'M',
        'ACA':'T', 'ACC':'T', 'ACG':'T', 'ACT':'T',
        'AAC':'N', 'AAT':'N', 'AAA':'K', 'AAG':'K',
        'AGC':'S', 'AGT':'S', 'AGA':'R', 'AGG':'R',                
        'CTA':'L', 'CTC':'L', 'CTG':'L', 'CTT':'L',
        'CCA':'P', 'CCC':'P', 'CCG':'P', 'CCT':'P',
        'CAC':'H', 'CAT':'H', 'CAA':'Q', 'CAG':'Q',
        'CGA':'R', 'CGC':'R', 'CGG':'R', 'CGT':'R',
        'GTA':'V', 'GTC':'V', 'GTG':'V', 'GTT':'V',
        'GCA':'A', 'GCC':'A', 'GCG':'A', 'GCT':'A',
        'GAC':'D', 'GAT':'D', 'GAA':'E', 'GAG':'E',
        'GGA':'G', 'GGC':'G', 'GGG':'G', 'GGT':'G',
        'TCA':'S', 'TCC':'S', 'TCG':'S', 'TCT':'S',
        'TTC':'F', 'TTT':'F', 'TTA':'L', 'TTG':'L',
        'TAC':'Y', 'TAT':'Y', 'TAA':'_', 'TAG':'_',
        'TGC':'C', 'TGT':'C', 'TGA':'_', 'TGG':'W',
    }
    for i in range(0, len(seq), 3):
        codon = seq[i:i+3].upper()
        if len(codon) < 3: break
        if codon in table:
            translation += table[codon]
        else:
            translation += "X"
    return(translation)
```


```python
translate(seq)
```




    'RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R'

## Expanding to multiple phases (frames)

We can further modify the function by adding a `phase` parameter that would allow translating in any of the three phases:


```python
def translate_phase(seq,phase):
    translation = ''
    table = {
        'ATA':'I', 'ATC':'I', 'ATT':'I', 'ATG':'M',
        'ACA':'T', 'ACC':'T', 'ACG':'T', 'ACT':'T',
        'AAC':'N', 'AAT':'N', 'AAA':'K', 'AAG':'K',
        'AGC':'S', 'AGT':'S', 'AGA':'R', 'AGG':'R',                
        'CTA':'L', 'CTC':'L', 'CTG':'L', 'CTT':'L',
        'CCA':'P', 'CCC':'P', 'CCG':'P', 'CCT':'P',
        'CAC':'H', 'CAT':'H', 'CAA':'Q', 'CAG':'Q',
        'CGA':'R', 'CGC':'R', 'CGG':'R', 'CGT':'R',
        'GTA':'V', 'GTC':'V', 'GTG':'V', 'GTT':'V',
        'GCA':'A', 'GCC':'A', 'GCG':'A', 'GCT':'A',
        'GAC':'D', 'GAT':'D', 'GAA':'E', 'GAG':'E',
        'GGA':'G', 'GGC':'G', 'GGG':'G', 'GGT':'G',
        'TCA':'S', 'TCC':'S', 'TCG':'S', 'TCT':'S',
        'TTC':'F', 'TTT':'F', 'TTA':'L', 'TTG':'L',
        'TAC':'Y', 'TAT':'Y', 'TAA':'_', 'TAG':'_',
        'TGC':'C', 'TGT':'C', 'TGA':'_', 'TGG':'W',
    }
    assert phase >= 0 and phase <= 3, "Phase parameter can only be set to 0, 1, or 2! You specified {}".format(phase)
    for i in range(phase, len(seq), 3):
        codon = seq[i:i+3].upper()
        if len(codon) < 3: break
        if codon in table:
            translation += table[codon]
        else:
            translation += "X"
    return(translation)
```


```python
translate_phase(seq,2)
```




    'TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT'



To translate in all six reading frames (three of the "+" strand and three of the "-" strand) we need to be able to create a reverse complement of the sequence. Let's write a simple function for that.
The cell below implements a function `revcomp` that takes a DNA sequence as input and returns its reverse complement. It works by first reversing the sequence using the slice
notation `seq[::-1]`, which returns the sequence in reverse order. Then, the `translate` method is used with the `str.maketrans` function to replace each occurrence
of 'a', 't', 'g', 'c', 'A', 'T', 'G', and 'C' in the reversed sequence with 't', 'a', 'c', 'g', 'T', 'A', 'C', and 'G', respectively:


```python
import string

def revcomp(seq):
    return seq[::-1].translate(str.maketrans('atgcATCG','tagcTACG'))
```

Now let's use this function to create translation in all six reading frames. The cell below uses a `for` loop that iterates over the range `(0, 3)`, representing the different phases
(or starting positions) of the translation. At each iteration, the `translate_phase` function is called with the DNA sequence and the current phase, and the resulting protein sequence
is appended to the `translations` list along with the `phase` and the `strand` orientation (+ or -):


```python
translations = []

for i in range(0,3):
    translations.append((translate_phase(seq,i),str(i),'+'))
    translations.append((translate_phase(revcomp(seq),i),str(i),'-'))
```


```python
translations
```




    [('RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R', '0', '+'),
     ('SLIIDKQQE_ELATDRRIKWWELRRLKASSRSL', '0', '-'),
     ('DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA', '1', '+'),
     ('R__STNNRNKN_PQTGESNGGNYGD_KRVPVAC', '1', '-'),
     ('TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT', '2', '+'),
     ('ADNRQTTGIRTSHRQANQMVGTTEIESEFP_P', '2', '-')]



## Finding coordinates of continuous translations

The translation we've generated above contains stops (e.g., `_` symbols). The actual biologically relevant protein sequences are between stops.
We now need to split translation strings into meaningful peptide sequences and compute their coordinates. Let's begin by splitting a string on `_` and computing the start and end positions of each peptide:


```python
string = "aadsds_dsds_dsds"

split_indices = []
for i,char in enumerate(string):
    if char == "_":
        split_indices.append(i)
print(split_indices)
```

    [6, 11]


The code above generates a list of split indices for a string. The list contains the indices of the characters in the string that match a specified character (in this case, the underscore `_` character).

The `enumerate` function is used to loop over the characters in the string, and at each iteration, the current index and character are stored in the variables `i` and `char`, respectively.
If the current character matches the specified character, the index `i` is appended to the `split_indices` list.

After the loop, the `split_indices` list is printed to the console. For the input string `"aadsds_dsds_dsds"`, the output would be `[6, 11]`, indicating that the dashes are located at indices 6 and 11.

But we actually need coordinates of peptides bound by `_` characters. To get to this let's first modify `split_indices` by adding the beginning and end:


```python
string = "aadsds_dsds_dsds"

split_indices = []
for i,char in enumerate(string):
    if char == "_":
        split_indices.append(i)
        
split_indices.insert(0,-1)
split_indices.append(len(string))
print(split_indices)
```

    [-1, 6, 11, 16]


Now, let's convert these to ranges and also stick the peptide sequence in:


```python
string = "aadsds_dsds_dsds"

split_indices = []
for i, char in enumerate(string):
    if char == "_":
        split_indices.append(i)
        
split_indices.insert(0, -1)
split_indices.append(len(string))

orfs = string.split('_')

parts = []
for i in range(len(split_indices)-1):
    parts.append((orfs[i],split_indices[i]+1, split_indices[i+1]))

print(parts)
```

    [('aadsds', 0, 6), ('dsds', 7, 11), ('dsds', 12, 16)]


Now let's convert this to function:


```python
def extract_coords(translation):
    split_indices = []
    for i, char in enumerate(translation):
        if char == "_":
            split_indices.append(i)
        
    split_indices.insert(0, -1)
    split_indices.append(len(translation))

    parts = []
    for i in range(len(split_indices)-1):
        parts.append((translation.split('_')[i], split_indices[i] + 1, split_indices[i + 1]))

    return(parts)
```


```python
extract_coords(string)
```




    [('aadsds', 0, 6), ('dsds', 7, 11), ('dsds', 12, 16)]



And specify the right parameters to make it truly generic:


```python
def extract_coords_with_annotation(separator,translation,phase,strand):
    split_indices = []
    for i,char in enumerate(translation):
        if char == separator:
            split_indices.append(i)
        
    split_indices.insert(0,-1)
    split_indices.append(len(translation))

    parts = []
    for i in range(len(split_indices)-1):
        parts.append((translation.split(separator)[i], phase, strand, split_indices[i]+1, split_indices[i+1]))

    return(parts)
```


```python
extract_coords_with_annotation('_', string, '0', '+')
```




    [('aadsds', '0', '+', 0, 6),
     ('dsds', '0', '+', 7, 11),
     ('dsds', '0', '+', 12, 16)]



## Analyzing all translations of a given sequence

We begin by parsing the `translations` list we defined above:

```python
all_translations = []
for item in translations:
    all_translations.append(extract_coords_with_annotation('_',item[0],item[1],item[2]))
```


```python
all_translations
```




    [[('RP', '0', '+', 0, 2),
      ('PKCV', '0', '+', 3, 7),
      ('MWLLGRISLRSVSVLVIGLLSAL', '0', '+', 8, 31),
      ('R', '0', '+', 32, 33)],
     [('SLIIDKQQE', '0', '-', 0, 9),
      ('ELATDRRIKWWELRRLKASSRSL', '0', '-', 10, 33)],
     [('DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA', '1', '+', 0, 33)],
     [('R', '1', '-', 0, 1),
      ('', '1', '-', 2, 2),
      ('STNNRNKN', '1', '-', 3, 11),
      ('PQTGESNGGNYGD', '1', '-', 12, 25),
      ('KRVPVAC', '1', '-', 26, 33)],
     [('TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT', '2', '+', 0, 32)],
     [('ADNRQTTGIRTSHRQANQMVGTTEIESEFP', '2', '-', 0, 30),
      ('P', '2', '-', 31, 32)]]



Now the problem with this list is that it is nested. However, we need to make it flat:


```python
flat_list = []
for sublist in all_translations:
    for item in sublist:
        flat_list.append(item)
```


```python
flat_list
```




    [('RP', '0', '+', 0, 2),
     ('PKCV', '0', '+', 3, 7),
     ('MWLLGRISLRSVSVLVIGLLSAL', '0', '+', 8, 31),
     ('R', '0', '+', 32, 33),
     ('SLIIDKQQE', '0', '-', 0, 9),
     ('ELATDRRIKWWELRRLKASSRSL', '0', '-', 10, 33),
     ('DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA', '1', '+', 0, 33),
     ('R', '1', '-', 0, 1),
     ('', '1', '-', 2, 2),
     ('STNNRNKN', '1', '-', 3, 11),
     ('PQTGESNGGNYGD', '1', '-', 12, 25),
     ('KRVPVAC', '1', '-', 26, 33),
     ('TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT', '2', '+', 0, 32),
     ('ADNRQTTGIRTSHRQANQMVGTTEIESEFP', '2', '-', 0, 30),
     ('P', '2', '-', 31, 32)]



Now we can load the list into `Pandas` and plot away:


```python
import pandas as pd
df = pd.DataFrame(flat_list,columns=['aa','frame','phase','start','end'])
```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>aa</th>
      <th>frame</th>
      <th>phase</th>
      <th>start</th>
      <th>end</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RP</td>
      <td>0</td>
      <td>+</td>
      <td>0</td>
      <td>2</td>
    </tr>
    <tr>
      <th>1</th>
      <td>PKCV</td>
      <td>0</td>
      <td>+</td>
      <td>3</td>
      <td>7</td>
    </tr>
    <tr>
      <th>2</th>
      <td>MWLLGRISLRSVSVLVIGLLSAL</td>
      <td>0</td>
      <td>+</td>
      <td>8</td>
      <td>31</td>
    </tr>
    <tr>
      <th>3</th>
      <td>R</td>
      <td>0</td>
      <td>+</td>
      <td>32</td>
      <td>33</td>
    </tr>
    <tr>
      <th>4</th>
      <td>SLIIDKQQE</td>
      <td>0</td>
      <td>-</td>
      <td>0</td>
      <td>9</td>
    </tr>
    <tr>
      <th>5</th>
      <td>ELATDRRIKWWELRRLKASSRSL</td>
      <td>0</td>
      <td>-</td>
      <td>10</td>
      <td>33</td>
    </tr>
    <tr>
      <th>6</th>
      <td>DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA</td>
      <td>1</td>
      <td>+</td>
      <td>0</td>
      <td>33</td>
    </tr>
    <tr>
      <th>7</th>
      <td>R</td>
      <td>1</td>
      <td>-</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td></td>
      <td>1</td>
      <td>-</td>
      <td>2</td>
      <td>2</td>
    </tr>
    <tr>
      <th>9</th>
      <td>STNNRNKN</td>
      <td>1</td>
      <td>-</td>
      <td>3</td>
      <td>11</td>
    </tr>
    <tr>
      <th>10</th>
      <td>PQTGESNGGNYGD</td>
      <td>1</td>
      <td>-</td>
      <td>12</td>
      <td>25</td>
    </tr>
    <tr>
      <th>11</th>
      <td>KRVPVAC</td>
      <td>1</td>
      <td>-</td>
      <td>26</td>
      <td>33</td>
    </tr>
    <tr>
      <th>12</th>
      <td>TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT</td>
      <td>2</td>
      <td>+</td>
      <td>0</td>
      <td>32</td>
    </tr>
    <tr>
      <th>13</th>
      <td>ADNRQTTGIRTSHRQANQMVGTTEIESEFP</td>
      <td>2</td>
      <td>-</td>
      <td>0</td>
      <td>30</td>
    </tr>
    <tr>
      <th>14</th>
      <td>P</td>
      <td>2</td>
      <td>-</td>
      <td>31</td>
      <td>32</td>
    </tr>
  </tbody>
</table>
</div>

Now let's plot it:


```python
import altair as alt

plus = alt.Chart(df[df['phase']=='+']).mark_rect().encode(
    x = alt.X('start:Q'),
    x2 = alt.X2('end:Q'),
    y = alt.Y('frame:N'),
    color='frame',
    tooltip='aa:N'
             ).properties(
    width=600,
    height=100)

minus = alt.Chart(df[df['phase']=='-']).mark_rect().encode(
    x = alt.X('start:Q',sort=alt.EncodingSortField('start:Q', order='descending')),
    x2 = alt.X2('end:Q'),
    y = alt.Y('frame:N'),
    color='frame',
    tooltip='aa:N'
             ).properties(
    width=600,
    height=100)

plus & minus
```

![ORFS](./images/orfs.svg "Translation on plus (top) and minus (bottom) strands.")


topic: data-science
tutorial: gnmx-lecture3
title: Introduction to sequencing with Python (part two)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: fec0d64c92b85cd1cc45e28d17185367
last_modified: 2026-02-26T21:12:14.880949
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture3/tutorial.html

content:
[![XKCD353](https://imgs.xkcd.com/comics/python.png)](https://xkcd.com/353/)

Preclass prep: Chapters [8](https://greenteapress.com/thinkpython2/html/thinkpython2009.html) and [10](https://greenteapress.com/thinkpython2/html/thinkpython2011.html) from "Think Python"

> This material uses examples from notebooks developed by [Ben Langmead](https://langmead-lab.org/teaching-materials/)
{: .quote cite="https://langmead-lab.org/teaching-materials" }

# Prep

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Within JupyterLab start a new Python3 notebook
3. Open [this page](http://cs1110.cs.cornell.edu/tutor/#mode=edit) in a new browser tab

# Strings in Python

In Python, strings are sequences of characters enclosed in quotation marks (either single or double quotes). They can be assigned to a variable and manipulated using various string methods. For example:

string1 = "Hello World!"
string2 = 'Hello World!'

print(string1)  # Output: Hello World!
print(string2)  # Output: Hello World!

You can also use the + operator to concatenate strings, and the * operator to repeat a string a certain number of times:


```python
string3 = "Hello " + "World!"
print(string3)  # Output: Hello World!

string4 = "Hello " * 3
print(string4)  # Output: Hello Hello Hello
```

    Hello World!
    Hello Hello Hello 


There are also many built-in string methods such as `upper()`, `lower()`, `replace()`, `split()`, `find()`, `len()`, etc. These can be used to manipulate and extract information from strings:


```python
string5 = "Hello World!"
print(string5.upper())   # Output: HELLO WORLD!
print(string5.lower())   # Output: hello world!
print(string5.replace("H", "J"))  # Output: Jello World!
print(string5.split(" "))  # Output: ['Hello', 'World!']
```

    HELLO WORLD!
    hello world!
    Jello World!
    ['Hello', 'World!']


You can also use indexing, slicing and string formatting to access or manipulate substrings or insert dynamic data into a string.


```python
name = "John"
age = 30

print("My name is {} and I am {} years old.".format(name, age))
```

    My name is John and I am 30 years old.

You can also use f-strings, or formatted string literals, to embed expressions inside string literals.

```python
name = "John"
age = 30

print(f"My name is {name} and I am {age} years old.")
```

    My name is John and I am 30 years old.

## String indexing

In Python, strings are sequences of characters, and each character has a corresponding index, starting from 0.
This means that we can access individual characters in a string using their index. This is called "string indexing".

Here is an example:

```python
string = "Hello World!"
print(string[0])  # Output: H
print(string[1])  # Output: e
print(string[-1])  # Output: !
```

    H
    e
    !

You can also use negative indexing to access characters from the end of the string, with -1 being the last character, -2 the second to last and so on.

You can also use slicing to extract a substring from a string. The syntax is `string[start:end:step]`, where `start` is the starting index, `end` is the ending index (not included), and step is the number of characters to skip between each index.

```python
string = "Hello World!"
print(string[0:5])  # Output: Hello
print(string[6:11])  # Output: World
print(string[::2])  # Output: HloWrd
```

    Hello
    World
    HloWrd

You can also use the string formatting method to get the string at the specific index.

```python
string = "Hello World!"
index = 3
print(f"The character at index {index} is: {string[index]}")
```

    The character at index 3 is: l


## String functions

In Python, many built-in string methods can be used to manipulate and extract information from strings. Here are some of the most commonly used ones:

- `upper()`: Converts the string to uppercase
- `lower()`: Converts the string to lowercase
- `replace(old, new)`: Replaces all occurrences of the old substring with the new substring
- `split(separator)`: Splits the string into a list of substrings using the specified separator
- `find(substring)`: Returns the index of the first occurrence of the substring, or -1 if the substring is not found
- `index(substring)`: Returns the index of the first occurrence of the substring or raises a ValueError if the substring is not found
- `count(substring)`: Returns the number of occurrences of the substring
- `join(iterable)`: Concatenates the elements of an iterable (such as a list or tuple) with the string as the separator
- `strip()`: Removes leading and trailing whitespaces from the string
- `lstrip()`: Removes leading whitespaces from the string
- `rstrip()`: Removes trailing whitespaces from the string
- `startswith(substring)`: Returns True if the string starts with the specified substring, False otherwise
- `endswith(substring)`: Returns True if the string ends with the specified substring, False otherwise
- `isalpha()`: Returns True if the string contains only alphabetic characters, False otherwise
- `isdigit()`: Returns True if the string contains only digits, False otherwise
- `isalnum()`: Returns True if the string contains only alphanumeric characters, False otherwise
- `format()`: Formats the string by replacing placeholders with specified values
- `len()`: Returns the length of the string

Here is an example of some of these methods:


```python
string = "Hello World!"

print(string.upper())  # Output: HELLO WORLD!
print(string.lower())  # Output: hello world!
print(string.replace("H", "J"))  # Output: Jello World!
print(string.split(" "))  # Output: ['Hello', 'World!']
print(string.find("World"))  # Output: 6
print(string.count("l"))  # Output: 3
print(string.strip())  # Output: "Hello World!"
print(string.startswith("Hello"))  # Output: True
print(string.endswith("World!"))  # Output: True
print(string.isalpha())  # Output: False
print(string.isalnum())  # Output: False
print(string.format())  # Output: Hello World!
print(len(string))  # Output: 12
```

    HELLO WORLD!
    hello world!
    Jello World!
    ['Hello', 'World!']
    6
    3
    Hello World!
    True
    True
    False
    False
    Hello World!
    12


## Playing with strings

```python
st = 'ACGT'
```


```python
len(st) # getting the length of a string
```

    4

```python
'' # empty string (epsilon)
```
    ''

```python
len('')
```

    0

```python
import random
random.choice('ACGT') # generating a random nucleotide
```
    'G'

```python
# now I'll make a random nucleotide string by concatenating random nucleotides
st = ''.join([random.choice('ACGT') for _ in range(40)])
st
```

    'GCTATATCAATGTTATCCGTTTTCTGATGTCGCGAGGACA'


```python
st[1:3] # substring, starting at position 1 and extending up to but not including position 3
# note that the first position is numbered 0
```

    'CT'


```python
st[0:3] # prefix of length 3
```

    'GCT'

```python
st[:3] # another way of getting the prefix of length 3
```

    'GCT'

```python
st[len(st)-3:len(st)] # suffix of length 3
```

    'ACA'

```python
st[-3:] # another way of getting the suffix of length 3
```

    'ACA'

```python
st1, st2 = 'CAT', 'ATAC'
```


```python
st1
```


    'CAT'


```python
st2
```

    'ATAC'


```python
st1 + st2 # concatenation of 2 strings
```

    'CATATAC'


# FASTQ

This notebook explores [FASTQ], the most common format for storing sequencing reads.

FASTA and FASTQ are rather similar, but FASTQ is almost always used for storing *sequencing reads* (with associated quality values), whereas FASTA is used for storing all kinds of DNA,
RNA or protein sequences (without associated quality values).

## Basic format
Here's a single sequencing read in FASTQ format:

    @ERR294379.100739024 HS24_09441:8:2203:17450:94030#42/1
    AGGGAGTCCACAGCACAGTCCAGACTCCCACCAGTTCTGACGAAATGATGAGAGCTCAGAAGTAACAGTTGCTTTCAGTCCCATAAAAACAGTCCTACAA
    +
    BDDEEF?FGFFFHGFFHHGHGGHCH@GHHHGFAHEGFEHGEFGHCCGGGFEGFGFFDFFHBGDGFHGEFGHFGHGFGFFFEHGGFGGDGHGFEEHFFHGE

It's spread across four lines.  The four lines are:

1. "`@`" followed by a read name
2. Nucleotide sequence
3. "`+`", possibly followed by some info, but ignored by virtually all tools
4. Quality sequence (explained below)

## Reading FASTQ with python

Download a sample small fastq file:

```
!wget https://zenodo.org/records/10602772/files/fastq_single_end_short.fq
```

Now we will use a very simple Python function to read this file and load fastq data into a list:

```python
def parse_fastq(fh):
    """ Parse reads from a FASTQ filehandle.  For each read, we
        return a name, nucleotide-string, quality-string triple. """
    reads = []
    while True:
        first_line = fh.readline()
        if len(first_line) == 0:
            break  # end of file
        name = first_line[1:].rstrip()
        seq = fh.readline().rstrip()
        fh.readline()  # ignore line starting with +
        qual = fh.readline().rstrip()
        reads.append((name, seq, qual))
    return reads

with open('fastq_single_end_short.fq','r') as fq:
    reads = parse_fastq(fq)

for read in reads:
    print(read)
```


```
('ERR294379.100739024 HS24_09441:8:2203:17450:94030#42/1', 'AGGGAGTCCACAGCACAGTCCAGACTCCCACCAGTTCTGACGAAATGATG', 'BDDEEF?FGFFFHGFFHHGHGGHCH@GHHHGFAHEGFEHGEFGHCCGGGF')
('ERR294379.136275489 HS24_09441:8:2311:1917:99340#42/1',  'CTTAAGTATTTTGAAAGTTAACATAAGTTATTCTCAGAGAGACTGCTTTT', '@@AHFF?EEDEAF?FEEGEFD?GGFEFGECGE?9H?EEABFAG9@CDGGF')
('ERR294379.97291341 HS24_09441:8:2201:10397:52549#42/1',  'GGCTGCCATCAGTGAGCAAGTAAGAATTTGCAGAAATTTATTAGCACACT', 'CDAF<FFDEHEFDDFEEFDGDFCHD=GHG<GEDHDGJFHEFFGEFEE@GH')
```


The nucleotide string can sometimes contain the character "`N`".  `N` essentially means "no confidence." The sequencer knows there's a nucleotide there but doesn't know whether it's an A, C, G or T.


> <comment-title>A note on <tt>while True</tt></comment-title>
>
>In Python, the while loop is used to repeatedly execute a block of code as long as a certain condition is true. The while True statement is a special case where the loop will run indefinitely until a break statement is encountered inside the loop.
>
>Here is an example of a while True loop:
>
>
>```python
>while True:
>    user_input = input("Enter 'q' to quit: ")
>    if user_input == 'q':
>        break
>    print("You entered:", user_input)
>
>print("Exited the loop")
>```
>```
>Enter 'q' to quit:  q
>Exited the loop
>```
>
>
>In this example, the loop will keep asking for user input until the user enters the 'q' character, which triggers the break statement, and the loop is exited.
>
>It is important to be careful when using while True loops, as they will run indefinitely if a break statement is not included. This can cause the program to crash or hang, if not handled properly.
>
>Also, It is recommended to use `while True` loop with a `break` statement, in case you want to execute the loop until some specific condition is met, otherwise, it's not a good practice to use `while True`.
>
>It's a good practice to include a way for the user to exit the loop, such as the break statement in the example above, or a counter variable to keep track of the number of iterations.
{: .comment}

## Read name

Read names often contain information about:

1. The scientific study for which the read was sequenced.  E.g. the string `ERR294379` (an [SRA accession number](http://www.ebi.ac.uk/ena/about/sra_format)) in the read names correspond to [this study](http://www.ncbi.nlm.nih.gov/sra/?term=ERR294379).
2. The sequencing instrument, and the exact *part* of the sequencing instrument, where the DNA was sequenced.  See the [FASTQ format](http://en.wikipedia.org/wiki/FASTQ_format#Illumina_sequence_identifiers) Wikipedia article for specifics on how the Illumina software encodes this information.
3. Whether the read is part of a *paired-end read* and, if so, which end it is.  Paired-end reads will be discussed further below.  The `/1` you see at the end of the read names above indicates the read is the first end from a paired-end read.

## Quality values

Quality values are probabilities.  Each nucleotide in each sequencing read has an associated quality value.  A nucleotide quality value encodes the probability that the nucleotide was *incorrectly called* by the sequencing instrument and its software.  If the nucleotide is `A`, the corresponding quality value encodes the probability that the nucleotide at that position is actually *not* an `A`.

Quality values are encoded in two senses: first, the relevant probabilities are re-scaled using the Phread scale, which is a negative log scale.
In other words if *p* is the probability that the nucleotide was incorrectly called, we encode this as *Q* where *Q* = -10 \* log10(*p*).

For example, if *Q* = 30, then *p* = 0.001, a 1-in-1000 chance that the nucleotide is wrong.  If *Q* = 20, then *p* = 0.01, a 1-in-100 chance.  If *Q* = 10, then *p* = 0.1, a 1-in-10 chance.  And so on.

Second, scaled quality values are *rounded* to the nearest integer and encoded using [ASCII printable characters](http://en.wikipedia.org/wiki/ASCII#ASCII_printable_characters).  For example, using the Phred33 encoding (which is by far the most common), a *Q* of 30 is encoded as the ASCII character with code 33 + 30 = 63, which is "`?`".  A *Q* of 20 is encoded as the ASCII character with code 33 + 20 = 53, which is "`5`".  And so on.

Let's define some relevant Python functions:

```python
def phred33_to_q(qual):
  """ Turn Phred+33 ASCII-encoded quality into Phred-scaled integer """
  return ord(qual)-33

def q_to_phred33(Q):
  """ Turn Phred-scaled integer into Phred+33 ASCII-encoded quality """
  return chr(Q + 33)

def q_to_p(Q):
  """ Turn Phred-scaled integer into error probability """
  return 10.0 ** (-0.1 * Q)

def p_to_q(p):
  """ Turn error probability into Phred-scaled integer """
  import math
  return int(round(-10.0 * math.log10(p)))
```


```python
# Here are the examples I discussed above

# Convert Qs into ps
q_to_p(30), q_to_p(20), q_to_p(10)
```

    (0.001, 0.01, 0.1)

```python=
p_to_q(0.00011) # note that the result is rounded
```

    40

```python=
q_to_phred33(30), q_to_phred33(20)
```

    ('?', '5')

To convert an entire string Phred33-encoded quality values into the corresponding *Q* or *p* values, I can do the following:


```python
# Take the first read from the small example above
name, seq, qual = parse_fastq(StringIO(fastq_string))[0]
q_string = list(map(phred33_to_q, qual))
p_string = list(map(q_to_p, q_string))
print(q_string)
print(p_string)
```

    [33, 35, 35, 36, 36, 37, 30, 37, 38, 37, 37, 37, 39, 38, 37, 37, 39, 39, 38, 39, 38, 38, 39, 34, 39, 31, 38, 39, 39, 39, 38, 37, 32, 39, 36, 38, 37, 36, 39, 38, 36, 37, 38, 39, 34, 34, 38, 38, 38, 37]
    [0.000501187233627272, 0.00031622776601683794, 0.00031622776601683794, 0.00025118864315095795, 0.00025118864315095795, 0.00019952623149688788, 0.001, 0.00019952623149688788, 0.00015848931924611126, 0.00019952623149688788, 0.00019952623149688788, 0.00019952623149688788, 0.0001258925411794166, 0.00015848931924611126, 0.00019952623149688788, 0.00019952623149688788, 0.0001258925411794166, 0.0001258925411794166, 0.00015848931924611126, 0.0001258925411794166, 0.00015848931924611126, 0.00015848931924611126, 0.0001258925411794166, 0.0003981071705534969, 0.0001258925411794166, 0.0007943282347242813, 0.00015848931924611126, 0.0001258925411794166, 0.0001258925411794166, 0.0001258925411794166, 0.00015848931924611126, 0.00019952623149688788, 0.000630957344480193, 0.0001258925411794166, 0.00025118864315095795, 0.00015848931924611126, 0.00019952623149688788, 0.00025118864315095795, 0.0001258925411794166, 0.00015848931924611126, 0.00025118864315095795, 0.00019952623149688788, 0.00015848931924611126, 0.0001258925411794166, 0.0003981071705534969, 0.0003981071705534969, 0.00015848931924611126, 0.00015848931924611126, 0.00015848931924611126, 0.00019952623149688788]


You might wonder how the sequencer and its software can *know* the probability that a nucleotide is incorrectly called.  It can't; this number is just an estimate.  To describe exactly how it's estimated is beyond the scope of this notebook; if you're interested, search for academic papers with "base calling" in the title.  Here's a helpful [video by Rafa Irizarry](http://www.youtube.com/watch?v=eXkjlopwIH4).

A final note: other ways of encoding quality values were proposed and used in the past.  For example, Phred64 uses an ASCII offset of 64 instead of 33, and Solexa64 uses "odds" instead of the probability *p*.  But Phred33 is by far the most common today and you will likely never have to worry about this.

><comment-title>A note in <tt>map()</tt></comment-title>
>
>In Python, the `map()` function is used to apply a given function to all elements of an iterable (such as a list, tuple, or string) and return an iterator (an object that can be iterated, e.g. in a `for`-loop) that yields the results.
>
>The `map()` function takes two arguments:
>
>A function that is to be applied to each element of the iterable
>An iterable on which the function is to be applied
>
>Here is an example:
>
>
>```python
># Using a function to square each element of a list
>numbers = [1, 2, 3, 4, 5]
>squared_numbers = map(lambda x: x**2, numbers)
>print(list(squared_numbers))  # Output: [1, 4, 9, 16, 25]
>```
>
>```
>[1, 4, 9, 16, 25]
>```
>
>In the example above, the `map()` function applies the lambda function `lambda x: x**2` to each element of the numbers list, and returns an iterator of the squared numbers. The `list()` function is used to convert the iterator to a list, so that the result can be printed.
>
>Another example is,
>
>
>```python
># Using the map() function to convert a list of strings to uppercase
>words = ["hello", "world"]
>uppercase_words = map(lambda word: word.upper(), words)
>print(list(uppercase_words))  # Output: ['HELLO', 'WORLD']
>```
>
>```
>['HELLO', 'WORLD']
>```
>
>It's important to note that the `map()` function returns an iterator, which can be used in a for loop, but is not a list, tuple, or any other iterable. If you want to create a list, tuple, or other iterable from the result of the `map()` function, you can use the `list()`, `tuple()`, or any other built-in function that creates an iterable.
>
>In Python 3, the `map()` function returns an iterator, which can be used in a for loop, but it's not iterable. If you want to create a list, tuple, or other iterable from the result of the `map()` function, you can use the `list()`, `tuple()`, or any other built-in function that creates an iterable.
>
>In Python 2, `map()` function returns a list, which can be used in a for loop, and it's iterable.
>
>In python 3.x, there is an alternative way to use map() function is `list(map(...))` or `tuple(map(...))` etc.
{: .comment}

## Paired-end reads

Sequencing reads can come in *pairs*.  Basically instead of reporting a single snippet of nucleotides from the genome, the sequencer might report a *pair* of
snippets that appear *close to each other* in the genome.  To accomplish this, the sequencer sequences *both ends* of a longer *fragment* of DNA.

Here is simple Python code that mimics how the sequencer obtains one paired-end read:

```python
# Let's just make a random genome of length 1K
import random
random.seed(637485)
genome = ''.join([random.choice('ACGT') for _ in range(1000)])
genome
```


```python
# The sequencer draws a fragment from the genome of length, say, 250
offset = random.randint(0, len(genome) - 250)
fragment = genome[offset:offset+250]
fragment
```


```python
# Then it reads sequences from either end of the fragment
end1, end2 = fragment[:75], fragment[-75:]
end1, end2
```

    ('GTATTGATTGTGTTACATAAACCCTCCCCCCCTACACGTGCTTGTCGACGCGGCACTGGACACTGATACGAGGAG',
     'CAATATATTAGCTCTCGAGGACAACGCACTACGTTTTCCAATTCAGCGGACTGGCGCCATTACAGTAAGTTGATT')

```python
# And because of how Illumina sequencing works, the
# second end is always from the opposite strand from the first
# (this is not the case for 454 and SOLiD data)

import string

# function for reverse-complementing
revcomp_trans = str.maketrans("ACGTacgt", "TGCAtgca")
def reverse_complement(s):
    return s[::-1].translate(revcomp_trans)

end2 = reverse_complement(end2)
end1, end2
```

    ('GTATTGATTGTGTTACATAAACCCTCCCCCCCTACACGTGCTTGTCGACGCGGCACTGGACACTGATACGAGGAG',
     'AATCAACTTACTGTAATGGCGCCAGTCCGCTGAATTGGAAAACGTAGTGCGTTGTCCTCGAGAGCTAATATATTG')

FASTQ can be used to store paired-end reads.  Say we have 1000 paired-end reads.  We should store them in a *pair* of FASTQ files.  The first FASTQ file (say, `reads_1.fq`) would contain all of the first ends and the second FASTQ file (say, `reads_2.fq`) would contain all of the second ends.  In both files, the ends would appear in corresponding order.  That is, the first entry in `reads_1.fq` is paired with the first entry in `reads_2.fq` and so on.

Here is a Python function that parses a pair of files containing paired-end reads.

```python
def parse_paired_fastq(fh1, fh2):
    """ Parse paired-end reads from a pair of FASTQ filehandles
        For each pair, we return a name, the nucleotide string
        for the first end, the quality string for the first end,
        the nucleotide string for the second end, and the
        quality string for the second end. """
    reads = []
    while True:
        first_line_1, first_line_2 = fh1.readline(), fh2.readline()
        if len(first_line_1) == 0:
            break  # end of file
        name_1, name_2 = first_line_1[1:].rstrip(), first_line_2[1:].rstrip()
        seq_1, seq_2 = fh1.readline().rstrip(), fh2.readline().rstrip()
        fh1.readline()  # ignore line starting with +
        fh2.readline()  # ignore line starting with +
        qual_1, qual_2 = fh1.readline().rstrip(), fh2.readline().rstrip()
        reads.append(((name_1, seq_1, qual_1), (name_2, seq_2, qual_2)))
    return reads

fastq_string1 = '''@509.6.64.20524.149722/1
AGCTCTGGTGACCCATGGGCAGCTGCTAGGGAGCCTTCTCTCCACCCTGA
+
HHHHHHHGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHIIHHIHFHHF
@509.4.62.19231.2763/1
GTTGATAAGCAAGCATCTCATTTTGTGCATATACCTGGTCTTTCGTATTC
+
HHHHHHHHHHHHHHEHHHHHHHHHHHHHHHHHHHHHHHDHHHHHHGHGHH'''

fastq_string2 = '''@509.6.64.20524.149722/2
TAAGTCAGGATACTTTCCCATATCCCAGCCCTGCTCCNTCTTTAAATAAT
+
HHHHHHHHHHHHHHHHHHHH@HHFHHHEFHHHHHHFF#FFFFFFFHHHHH
@509.4.62.19231.2763/2
CTCTGCTGGTATGGTTGACGCCGGATTTGAGAATCAANAAGAGCTTACTA
+
HHHHHHHHHHHHHHHHHHEHEHHHFHGHHHHHHHH>@#@=44465HHHHH'''

parse_paired_fastq(StringIO(fastq_string1), StringIO(fastq_string2))
```

    [(('509.6.64.20524.149722/1',
       'AGCTCTGGTGACCCATGGGCAGCTGCTAGGGAGCCTTCTCTCCACCCTGA',
       'HHHHHHHGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHIIHHIHFHHF'),
      ('509.6.64.20524.149722/2',
       'TAAGTCAGGATACTTTCCCATATCCCAGCCCTGCTCCNTCTTTAAATAAT',
       'HHHHHHHHHHHHHHHHHHHH@HHFHHHEFHHHHHHFF#FFFFFFFHHHHH')),
     (('509.4.62.19231.2763/1',
       'GTTGATAAGCAAGCATCTCATTTTGTGCATATACCTGGTCTTTCGTATTC',
       'HHHHHHHHHHHHHHEHHHHHHHHHHHHHHHHHHHHHHHDHHHHHHGHGHH'),
      ('509.4.62.19231.2763/2',
       'CTCTGCTGGTATGGTTGACGCCGGATTTGAGAATCAANAAGAGCTTACTA',
       'HHHHHHHHHHHHHHHHHHEHEHHHFHGHHHHHHHH>@#@=44465HHHHH'))]

><comment-title>A note on triple quotes</comment-title>
>
>In Python, triple quotes (either single or double) are used to create multi-line strings. They can also be used to create doc-strings, which are used to document a function, class, or module.
>
>For example:
>
>
>```python
>string1 = """This is a
>multiline string"""
>
>string2 = '''This is also
>a multiline string'''
>
>print(string1)
># Output:
># This is a
># multiline string
>
>print(string2)
># Output:
># This is also
># a multiline string
>```
>
>    This is a
>    multiline string
>    This is also
>    a multiline string
>
>
>Triple quotes can also be used to create docstrings, which are used to document a function, class, or module. The first line of a docstring is a brief summary of what the function, class, or module does, and the following lines provide more detailed information.
>
>
>```python
>def my_function():
>    """
>    This is a docstring for the my_function.
>    This function does not perform any operation.
>    """
>    pass
>
>print(my_function.__doc__)
># Output:
># This is a docstring for the my_function.
># This function does not perform any operation.
>```
>
>    
>        This is a docstring for the my_function.
>        This function does not perform any operation.
>        
{: .comment}

## How good are my reads?

Let's reuse the `parse_fastq` function from above:

```python
# Read the file into a list
with open('fastq_single_end_short.fq','r') as fq:
    reads = parse_fastq(fq)
```

☝️ This will load data into a list called `reads` containing individual reads (name, nucleotides, and quality values) represented as a [tuple](https://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences). 


```python
# Extract qualities and convert them into numerical values
run_qualities = []
for read in reads:
    read_qualities = []
    for quality in read[2]:
        read_qualities.append(phred33_to_q(quality))
    run_qualities.append(read_qualities)
```

☝️ This will take quality string only for each read from `reads` and convert it into a list of values. The end result, a list of lists called `run_qualities`, will contain as many elements as there as reads, where each element (also a list) contains all quality values for a read. We need to [transpose](https://en.wikipedia.org/wiki/Transpose) this list, so that instead of representing every reads as a list of quality values we represent every position as a list of quality values for all reads at that position. This can be easily done with [numpy](https://numpy.org/)'s [`T`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.T.html):

```python
# Transpose the quality values matrix

import numpy as np
base_qualities = np.array(run_qualities).T
```

Now we need to extract some key per-position statistics from the `base_qualities` list generated above. We, again, use numpy for this:

```python
# Prep data for plotting
plotting_data = {
    'base':[],
    'mean':[],
    'median':[],
    'q25':[],
    'q75':[],
    'min':[],
    'max':[]
}
for i,base in enumerate(base_qualities):
    plotting_data['base'].append(i)
    plotting_data['mean'].append(np.mean(base))
    plotting_data['median'].append(np.median(base))
    plotting_data['q25'].append(np.quantile(base,.25))
    plotting_data['q75'].append(np.quantile(base,.75))
    plotting_data['min'].append(np.min(base))
    plotting_data['max'].append(np.max(base))
```

☝️ The result of this, a dictionary called `plotting_data`, contains statistics types as *keys* and corresponding values as *lists*. Now we load these into Pandas:

```python
# Load into Pandas
import pandas as pd
plotting_data_df = pd.DataFrame(plotting_data)
```

And plot:

```python
# Plot!
import altair as alt

base = alt.Chart(plotting_data_df).encode(
    alt.X('base:Q', title="Position in the read")
).properties(
    width=800,
    height=200)

median = base.mark_tick(color='red',orient='horizontal').encode(
    alt.Y('median:Q', title="Phred quality score"),
)

q = base.mark_rule(color='green',opacity=.5,strokeWidth=10).encode(
    alt.Y('q25:Q'),
    alt.Y2('q75:Q')
)

min_max = base.mark_rule(color='black').encode(
        alt.Y('min:Q'),
        alt.Y2('max:Q')
)

median + q + min_max
```

Below is the same thing as just one code blob:

```python
# Read the file into a list
with open('fastq_single_end_short.fq','r') as fq:
    reads = parse_fastq(fq)

# Extract qualities and convert them into numerical values
run_qualities = []
for read in reads:
    read_qualities = []
    for quality in read[2]:
        read_qualities.append(phred33_to_q(quality))
    run_qualities.append(read_qualities)

# Transpose the quality values matrix

import numpy as np
base_qualities = np.array(run_qualities).T

# Prep data for plotting
plotting_data = {
    'base':[],
    'mean':[],
    'median':[],
    'q25':[],
    'q75':[],
    'min':[],
    'max':[]
}
for i,base in enumerate(base_qualities):
    plotting_data['base'].append(i)
    plotting_data['mean'].append(np.mean(base))
    plotting_data['median'].append(np.median(base))
    plotting_data['q25'].append(np.quantile(base,.25))
    plotting_data['q75'].append(np.quantile(base,.75))
    plotting_data['min'].append(np.min(base))
    plotting_data['max'].append(np.max(base))

# Load into Pandas
import pandas as pd
plotting_data_df = pd.DataFrame(plotting_data)

# Plot!
import altair as alt

base = alt.Chart(plotting_data_df).encode(
    alt.X('base:Q', title="Position in the read")
).properties(
    width=800,
    height=200)

median = base.mark_tick(color='red',orient='horizontal').encode(
    alt.Y('median:Q', title="Phred quality score"),
)

q = base.mark_rule(color='green',opacity=.5,strokeWidth=10).encode(
    alt.Y('q25:Q'),
    alt.Y2('q75:Q')
)

min_max = base.mark_rule(color='black').encode(
        alt.Y('min:Q'),
        alt.Y2('max:Q')
)

median + q + min_max
```

![Qualities](./images/qualities.svg "Box plot of quality values across read positions")

## Other comments

In all the examples above, the reads in the FASTQ file are all the same length.  This is not necessarily the case though it is usually true for datasets generated by sequencing-by-synthesis instruments.  FASTQ files can contain reads of various lengths.

FASTQ files often have the extension `.fastq` or `.fq`.

## Other resources

- [BioPython](http://biopython.org/wiki/Main_Page)
- [SeqIO](http://biopython.org/wiki/SeqIO)
- [SAMtools](http://samtools.sourceforge.net/)
- [FASTX](http://hannonlab.cshl.edu/fastx_toolkit/)
- [FASTQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
- [seqtk](https://github.com/lh3/seqtk)


topic: data-science
tutorial: python-files
title: Python - Files & CSV
difficulty: introductory
time_estimation: 1H30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 8d7f3c70407f89d6b905b37748542d04
last_modified: 2026-02-26T21:12:14.901425
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-files/tutorial.html

content:
Here we'll give a quick tutorial on how to read and write files within Python.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Setup

For this tutorial, we assume you're working in a notebook (Jupyter, CoCalc, etc) so we'll run a quick "setup" step to download some CSV data:

```python
import urllib.request
# Download a copy of Hamlet.
urllib.request.urlretrieve("https://gutenberg.org/cache/epub/1524/pg1524.txt", "hamlet.txt")
# Download some COVID data for Europe
urllib.request.urlretrieve("https://opendata.ecdc.europa.eu/covid19/vaccine_tracker/csv/data.csv", "vaccinations.csv")
# And a fastq file
urllib.request.urlretrieve("https://gist.github.com/hexylena/7d249607f8f763301f06c78a48c3bf6f/raw/a100e278cee1c94035a3a644b16863deee0ba2c0/example.fastq", "example.fastq")
```

And now we're ready to get started learning about files!

## Reading from a file

Reading from and writing to files in Python is very straightforward, we use `open()` to open a file to read from it. Files are accessed through something called a **file [handle](https://en.wikipedia.org/wiki/Handle_(computing))**, you're not accessing the file itself, you're opening a connection to the file, and then you can read lines from through that file handle. When you open a file handle, you must specify it's mode:

Mode | Purpose
--- | ---
`r` | We will **read** from this file handle
`w` | We will **write** to this file handle
`a` | We will **append** to this file handle (we cannot access earlier contents!)

You will mostly use `r` and `w`, `a` is especially useful for writing to program logs where you don't really care what was written before, you just want to add your new logs to the end of the file.

```python
with open('hamlet.txt', 'r') as handle:
    # readlines reads every line of a file into a giant list!
    lines = handle.readlines()
```

Here we introduce a new bit of syntax, the `with` block. Technically `with` begins a "context manager" which allows python to setup some things before the block, run some contents in the block, and automatically handle cleanup of this block. When you open a file, you *must* close it when you're done with it (otherwise bad things can happen!) and `with` prevents most of those issues.

In the above code snippet after the second line, the file (referred to by `handle`) is automatically closed.

> > <code-in-title>Using `with`</code-in-title>
> > ```
> > with open('file.txt', 'r') as handle:
> >     print(handle.readlines())
> > ```
> {: .code-in}
> > <code-out-title>Not using `with`</code-out-title>
> > ```
> > handle = open('file.txt', 'r')
> > print(handle.readlines())
> > handle.close() # Important!
> > ```
> {: .code-out}
{: .code-2col}

Let's see what's in our file. `

```python
print(lines[0])
print(lines[1])
print(lines[2])
```

Notice how it prints out a blank line afterwards! This is due to a `\n`, a newline. A newline just tells the computer "please put content on the next line". We can see it by using the `repr()` function:

```python
print(repr(lines[0]))
```

Every line that's read in ends in a newline currently. This is done because if we wanted to write it back out, we would need to preserve those newlines, or all of the content would be on one giant line. Let's try writing out a file, it's *just* like reading in a file!

```python
with open('hamlet-copy.txt', 'w') as handle:
    # readlines reads every line of a file into a giant list!
    for line in lines:
        handle.write(line)
```

Check this file in your folder, does it look right? Is it identical in size to the original?

### Use Case: Summarisation

Let's use the file's contents for something useful. This file specifically is the play Hamlet, by Shakespeare. The contents are formatted with a speaker indicated with all capital letters, followed by their lines (potentially spread over multiple lines of the file.)

```
HAMLET. Madam, how like you this play?

QUEEN. The lady protests too much, methinks.

HAMLET. O, but she’ll keep her word.
```

So let's count up how many times each character speaks! (Roughly)

```python
with open('hamlet.txt', 'r') as handle:
    # readlines reads every line of a file into a giant list!
    lines = handle.readlines()

speakers = {}
```

Here we've initialised the `lines` variable with the contents of the text, and setup `speakers` as a dictionary that will let us track how many times each character speaks. Next let's define a function to check if a character is speaking on that line. It is if it meets two conditions: the first word is all caps, and it ends with a `.`.

```python
def is_speaker(word):
    return word == word.upper() and word[-1] == '.'
```

We can use that function later to check if a line starts with a speaker

```python
# Loop over every line we read in
for line in lines:
    # Split by default splits on whitespace.
    words = line.split()

    # Are there words on this line? We can't access the first word if we
    # haven't any words.
    if len(words) == 0:
        continue

    # Check if the first word is uppercase, and the last character is a `.`,
    # then it's a character speaking.
    if is_speaker(words[0]):
        # Give this an easier to remember and understand name.
        speaker = words[0]

        # Have we seen this speaker before? If not, we should add them to the
        # speakers dictionary. Hint: Try removing this to see why we do this.
        if speaker not in speakers:
            speakers[speaker] = 0

        # Increment the number of times we've seen them speak.
        speakers[speaker] = speakers[speaker] + 1
```

Ok! We've done a couple things here that all fall into the category of **defensive programming**. As programmers, we often accept input from users or from unknown sources. That input may be wrong, it may have bad data, it may be trying to attack us. So we respond by checking very carefully if things match our expectations, and rejecting the input otherwise. We did a couple things here for that:

- Checking if the input matches our expectations exactly (capitals, `.`)
- Checking if the line is empty, using `continue` to skip it if it was
- Checking if the speaker is already known in the dictionary, adding it otherwise.

Let's see who was the most chatty:

```python
for key, value in speakers.items():
    print(key, value)
```

We've clearly caught a number of values that aren't expected, some section headers (the numeric values, and some rare values we don't expect.)

```python
for key, value in speakers.items():
    if value > 1:
        print(key, value)
```

Hamlet, the titular character, has the vast majority of turns speaking throughout the play:

Character | Turns speaking
---       | ---
HAMLET    | 358
HORATIO   | 107
KING      | 102
POLONIUS  | 86
QUEEN     | 69


## Writing files

Writing a file out is exactly like reading a file, we just use the different file mode `w` to indicate we wish to write to a file:

```python
with open('hello.txt', 'w') as handle:
    # Let's write it out a few times
    handle.write("Hello, world!")
    handle.write("Hello, world!")
    handle.write("Hello, world!")
    handle.write("Hello, world!")
```

Check the file's contents in your folder, does it look like you expect? Remember your `\n`s!

### Use Case: Transformation

A common use case is transforming one file's contents into another file format or file type. Let's do a very simple example of that, taking a FASTQ file and transforming it into a FASTA file. Remember first that a FASTQ file looks like:

```
@M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT
+
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGFGGGGGGAFFGGFGGGGGGGGFGGGGGGGGGGGGGGFGGG+38+35*311*6,,31=******441+++0+0++0+*1*2++2++0*+*2*02*/***1*+++0+0++38++00++++++++++0+0+2++*+*+*+*+*****+0**+0**+***+)*.***1**//*)***)/)*)))*)))*),)0(((-((((-.(4(,,))).,(())))))).)))))))-))-(
```

Line | Contents
-- | --
1 | Identifier, prefixed with `@`
2 | Sequence
3 | `+`
4 | Quality scores

And a fasta file looks like this, where `>` indicates a sequence identifier, and it is followed by one or more lines of ACTGs.

```
>M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT
```

In the setup portion we downloaded a FASTQ file, now let's extract all of the sequences from this file, and write them out as a FASTA file. Why would you want to do this? Sometimes after sequencing a sample (especially metagenomics), you want to blast the sequences to figure out which organisms they belong to. A common way to do that is BLAST which accepts fasta formatted sequences. So we'll write something to convert these formats, removing the `+` and quality score lines.

```python
with open('example.fastq', 'r') as handle:
    data = handle.readlines()

i = 0
with open('example-converted.fasta', 'w') as handle:
    for line in data:
        # Since fastq files have groups of 4 lines, we can use that to extract
        # specific lines of every read:
        if i % 4 == 0:
            handle.write(">" + line[1:])
        if i % 4 == 1:
            handle.write(line)
        i = i + 1
```

That's it! Check if your `fasta` file looks correct.

> <tip-title>`enumerate`</tip-title>
> If you want to know which item number you're on while you're looping over a list, you can use the function `enumerate()`
> Try out this code to see how it works:
>
> ```
> for index, item in enumerate(['a', 'b', 'c']):
>     print(index, item)
> ```
>
> Try using it to clean up the above code.
{: .tip}

## Reading CSV data

If you're reading data from a comma separated value (CSV) or tab separated value (TSV) file, you should use the built in `csv` module to do this. You might ask yourself "Why, csv parsing is easy" and that is a common thought! It would be so simple to do something like

```python
# Please don't do this :)
with open('vaccinations.csv', 'r') as handle:
    first_lines = handle.readlines()[0:10]
    for line in first_lines:
        print(line.split(','))
```

But you would be wrong! This code has a subtle bug that you might not see until someone generates data that specifically affects it, with "quoted" columns. If you have a table like

Patient | Location | Disease Indications
--- | --- | ---
Helena | Den Haag, the Netherlands | Z87.890
Bob | Little Rock, Arkansas, USA | Z72.53
Jane | London, UK | Z86.16

This would probably be exported as a CSV file from Excel that looks like:

```
Patient,Location,Disease Indications
Helena,"Den Haag, the Netherlands",Z87.890
Bob,"Little Rock, Arkansas, USA",Z72.53
Jane,"London, UK",Z86.16
```

Note that some columns are quoted. What do you think will happen with the following code?

```python
csv_data = """
Patient,Location,Disease Indications
Helena,"Den Haag, the Netherlands",Z87.890
Bob,"Little Rock, Arkansas, USA",Z72.53
Jane,"London, UK",Z86.16
""".strip().split('\n')

# Please don't do this :)
for line in csv_data:
    print(line.split(','))
```

Does that look right? Maybe not. Instead we can use the `csv` module to work around this and properly process CSV files:

```python
import csv

csv_data = """
Patient,Location,Disease Indications
Helena,"Den Haag, the Netherlands",Z87.890
Bob,"Little Rock, Arkansas, USA",Z72.53
Jane,"London, UK",Z86.16
""".strip().split('\n')

# Please DO this :)
csv_reader = csv.reader(csv_data, delimiter=",", quotechar='"')
for row in csv_reader:
    print(row)
```

That looks a lot better! Now we've properly handled the quoted columns that contain one or more `,` in the middle of our file. This is actually one of the motivating factors in using the TSV format, the <kbd>tab</kbd> character is much more rare in data than <kbd>,</kbd>. There is less chance for confusion with poorly written software.

Let's read in some statistics about vaccinations:

```python
import csv

vax = []
with open('vaccinations.csv', 'r') as handle:
    csv_reader = csv.reader(handle, delimiter=",", quotechar='"')
    for row in csv_reader:
        # Skip our header row
        if row[0] == 'YearWeekISO':
            continue
        # Otherwise load in the data
        vax.append(row)

print(vax[0:10])
```

Here we have a 2 dimensional array, a list of lists. Each row is an entry in the main list, and each column is an entry in each of those children.

Our columns are:

Column | Value
-- | --
0   | YearWeekISO
1   | ReportingCountry
2   | Denominator
3   | NumberDosesReceived
4   | NumberDosesExported
5   | FirstDose
6   | FirstDoseRefused
7   | SecondDose
8   | DoseAdditional1
9   | UnknownDose
10  | Region
11  | TargetGroup
12  | Vaccine
13  | Population

Let's subset the data to make it a bit easier to work with, maybe we'll just use the Dutch data (please feel free to choose another column though!)

```python
country = 'NL'

subset = []
for row in vax:
    # Here we select for a country
    if row[1] == country:
        subset.append(row)
print(subset[0:10])
```

That should be easier to work with, now we only have one country's data. Let's do some exercises with this data:


> <question-title>Which vaccines were given?</question-title>
>
> Which vaccines were given? Use the `subset` to examine which vaccines were given in the Netherlands. *Tip*: if `x` is a list, `set(x)` will return the unique values in that list.
>
> > <solution-title></solution-title>
> > To figure out which vaccines were given, we can look at column 12:
> >
> > ```python
> > vaccines = []
> > for row in subset:
> >     vaccines.append(row[12])
> > print(set(vaccines))
> > ```
> >
> > We can use the `set` function to convert the list to a set, and show only the unique values.
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>How many of each vaccine were given?</question-title>
>
> How many of each were given?
>
> *Tip*: use the accumulator pattern.
> *Tip*: Columns 5, 7, and 8 have doses being given out to patients.
>
> > <solution-title></solution-title>
> > ```python
> > doses = {}
> > for row in subset:
> >     brand = row[12]
> >     if brand not in doses:
> >         doses[brand] = 0
> >     doses[brand] = doses[brand] + int(row[5]) + int(row[7]) + int(row[8])
> > print(doses)
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>How many of each vaccine were exported? received?</question-title>
>
> How many of each were exported? received?
>
> *Tip*: you only need to loop once.
> *Tip*: you will need to handle an edge case here. Try and find it out!
>
> > <solution-title></solution-title>
> > ```python
> > export = {}
> > received = {}
> > for row in subset:
> >     brand = row[12]
> >     if brand not in export:
> >         export[brand] = 0
> >     if brand not in received:
> >         received[brand] = 0
> >     if row[4]:
> >         export[brand] = export[brand] + int(row[4])
> >     if row[3]:
> >         received[brand] = received[brand] + int(row[3])
> > print(export)
> > print(received)
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>When was the first dose received?</question-title>
>
> *Tip*: use `break`, and check how many doses were given!
>
> > <solution-title></solution-title>
> > ```python
> > for row in subset:
> >     if row[5] and int(row[5]) > 0:
> >         print(f"On {row[0]}, {row[5]} doses of {row[12]} were given")
> >         break
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>Transform the data for plotting</question-title>
>
> Let's say you want to plot the fraction of the population that has been vaccinated by the various points in time.
>
> - Subset the data further for TargetGroup (column 11) set to 'ALL'
> - Create an accumulator, to count how many doses have been given their FirstDose at each week
> - Use column 13 (population) to calculate the fraction of the population that has been given one of those doses at each week
>
> The output should be a list of percentages ranging from [0.0 to 1.0].
>
> > <solution-title></solution-title>
> > ```python
> > total_doses = 0
> > percent_vaccinated_per_week = []
> > for row in subset:
> >     if row[11] != 'ALL':
> >         continue
> >     total_doses = total_doses + int(row[5])
> >     percent_vaccinated_per_week.append(total_doses / int(row[13]))
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
percent_vaccinated_per_week = []
# Write code here!




# When you're done, you should have a 'results' variable
# You may need to `pip install matplotlib`
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(percent_vaccinated_per_week)
plt.xlabel('Week')
plt.ylabel('Percent Vaccinated')
```


> <question-title>Which vaccines were given?</question-title>
>
> Write this out to a new file, with two columns. The index and the Percent Vaccinated. It should be a comma separated file, and should have a header. Save this as a csv file named `weekly-percent-vax.csv`
>
> *Tip*: use a `csvwriter`, it works exactly like a `csvreader`. You can use the `writerow()` function to write out a row.
> *Tip*: Use `enumerate()` to get a list of items with indexes.
>
> > <solution-title></solution-title>
> >
> > ```
> > with open('weekly-percent-vax.csv', 'w') as handle:
> >     writer = csv.writer(csv_data, delimiter=",", quotechar='"')
> >     for row in enumerate(percent_vaccinated_per_week):
> >         writer.writerow(row)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try things out here!
```

Congratulations on getting this far! Hopefully you feel more comfortable working with files.


topic: data-science
tutorial: python-loops
title: Python - Loops
difficulty: introductory
time_estimation: 40M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 7a86289d7fda6343135a1605c16ae684
last_modified: 2026-02-26T21:12:14.903260
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-loops/tutorial.html

content:
A *for loop* tells Python to execute some statements once for each value in a list, a character string, or some other collection: "for each thing in this group, do these operations"

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/), [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/), and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which are licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# For Loops

Which of these would you rather write

> > <code-in-title>Manually</code-in-title>
> > ```
> > print(2)
> > print(3)
> > print(5)
> > print(7)
> > print(11)
> > ```
> {: .code-in}
>
> > <code-out-title>With Loops</code-out-title>
> > ```python
> > for number in [2, 3, 5, 7, 11]:
> >     print(number)
> > ```
> {: .code-out}
{: .code-2col}

It may be less clear here, since you just need to do one operation (`print`) but if you had to do two operations, three, more?

## Structure

## A `for` loop is made up of a collection, a loop variable, and a body.

```python
for number in [2, 3, 5]:
    doubled = number * 2
    print(f"{number} doubled is {doubled}")
```

- `number` - this is the loop variable. It's a new variable, that's assigned to the values from the collection. It does not need to be defined before the loop.
- the collection, `[2, 3, 5]` is a `list` of numbers which we can tell from the square brackets used: `[`, `]`
- the loop body, where we double a number and the print out a message. The loop body is what gets executed for every iteration of the loop.

> > <code-in-title>The loop</code-in-title>
> > ```python
> > for number in [2, 3, 5]:
> >     doubled = number * 2
> >     print(f"{number} doubled is {doubled}")
> > ```
> {: .code-in}
>
> > <code-out-title>What's really happening internally</code-out-title>
> > ```python
> > # First iteration, number = 2
> > doubled = number * 2
> > print(f"{number} doubled is {doubled}")
> > # Second iteration, number = 3
> > doubled = number * 3
> > print(f"{number} doubled is {doubled}")
> > # Third iteration, number = 5
> > doubled = number * 5
> > print(f"{number} doubled is {doubled}")
> > ```
> {: .code-out}
{: .code-2col}

Writing loops saves us time and makes sure our code is accurate, that we don't accidentally introduce a typo somewhere in the loop body.

## Things You Can Loop Over

You can loop over characters in a string

```python
dna_string = 'ACTGGTCATCG'
for base in dna_string:
    print(base)
```

You can loop over lists:

```python
cast = ['Elphaba', 'Glinda', 'Fiyero', 'Nessarose']
for character in cast:
    print(character)
```

## Indentation

The first line of the `for` loop must end with a colon, and the body must be indented with *four spaces*. Many editors do this automatically for you and even convert <kbd>Tab</kbd>s into 4 spaces.

> <tip-title>Blocks in Python</tip-title>
>
> The colon at the end of the first line signals the start of a *block* of statements.
>
> ```
> for x in y:
>     print(x)
> ```
>
> or
>
> ```
> if x > 10:
>     print(x)
> ```
>
> or even further nesting is possible:
>
> ```
> for x in y:
>     if x > 10:
>         print(x)
> ```
>
{: .tip}

The indentation is in fact, quite necessary. Notice how this fails:

```python
#Fix me!
for number in [2, 3, 5]:
print(number)
```

And, likewise, this:

```python
patient1 = "z2910"
  patient2 = "y9583"
```

## Variable Naming

Loop variables can be called anything, `i`, `j`, and `k` are very commong defaults due to their long history of use in other programing languages.
As with all variables, loop variables are: Created on demand, and Meaningless; their names can be anything at all.

```python
for kitten in [2, 3, 5]:
    print(kitten)
```

But *meaningless* is bad for variable names, and whenever possible, we should strive to pick useful, accurate variable names that help use remember what's going on:

```
for sequence in sequences:
    print()
for patient in clinic_patients:
    print()
for nucleotide in dna_sequence:
    print()
```

## Range

You can use `range` to iterate over a sequence of numbers. This is a built in function (check `help(range)`!) so it's always available even if you don't `import` anything. The range produced is non-inclusive: `range(N)` is the numbers `0` to `N-1`, so the result will be exactly the length you requested.

```python
for number in range(10):
    print(number)
```

> <tip-title>Iterables can be weird</tip-title>
> In python `range` is a special type of iterable: none of the numbers are created until we need them.
>
> ```python
> print(range(5))
> print(range(-3, 8)[0:4])
> ```
>
> The easiest way to see what numbers are actually in there is to convert it to a `list`:
>
> ```python
> print(list(range(5)))
> print(list(range(-3, 8)))
> print(list(range(0, 10, 2)))
> ```
{: .tip}

## Accumulation

In programming you'll often want to accumulate some values: counting things (or "accumulating"). The pattern consists of creating a variable to store your result, running a loop over some data, and in that loop, adding to the variable for your result.

```python
# Sum the first 10 integers.
total = 0
for number in range(1, 11):
    total = total + (number)
print(f" final: {{ total }}")
```

But how did we get that result? We can add some "debugging" lines to the above code to figure out how we got to that result. Try adding the following line in the above loop

```
print(f'Currently {number}, our total is {total}')
```

You can add it before you update `total`, after it, or both! Compare the outputs to understand what's happening on each line.

> <tip-title>Controlling your loop!</tip-title>
>
> There are multiple ways to efficiently control your loop if you need it.
> these are the inbuilt python functions: continue & break
>
> when python encounters *continue* in your loop it will stop working and goes to the next iteration of the loop.
> ```
> for letter in 'Galaxy':
>     if letter == 'l':
>         continue
>     print(f'The letters are: {letter}')
>```
> with *break* python stops the loop and continues with the next part of the code like nothing happened
> ```
> for letter in 'Galaxy':
>     if letter == 'l':
>         break
>     print(f'The letters are: {letter}')
> print('Done')
> ```
{: .tip}

```python
# Test break and continue here
```

## Exercises

> <question-title>Tracing Execution</question-title>
>
> Create a table showing the numbers of the lines that are executed when this program runs,
> and the values of the variables after each line is executed.
>
> ```
> total = 0
> for char in "tin":
>     total = total + 1
> ```
> > <solution-title></solution-title>
> >
> > | Line | Variables            |
> > |------|----------------------|
> > | 1    | total = 0            |
> > | 2    | total = 0 char = 't' |
> > | 3    | total = 1 char = 't' |
> > | 2    | total = 1 char = 'i' |
> > | 3    | total = 2 char = 'i' |
> > | 2    | total = 2 char = 'n' |
> > | 3    | total = 3 char = 'n' |
> {: .solution}
{: .question}

```python
#Test your code here!
```

> <question-title>Reversing a String</question-title>
>
> Fill in the blanks in the program below so that it prints "stressed"
> (the reverse of the original character string "desserts").
>
> ```
> original = "stressed"
> result = ____
> for char in original:
>     result = ____
> print(result)
> ```
> > <solution-title></solution-title>
> > ```
> > original = "stressed"
> > result = ""
> > for char in original:
> >     result = char + result
> > print(result)
> > ```
> {: .solution}
{: .question}

```python
# Test your code here!
original = "stressed"
result = ____
for char in original:
    result = ____
print(result)
```


> <question-title>Practice Accumulating</question-title>
>
> Fill in the blanks in each of the programs below
> to produce the indicated result.
>
> ```
> # Total length of the strings in the list: ["red", "green", "blue"] => 12
> total = 0
> for word in ["red", "green", "blue"]:
>     ____ = ____ + len(word)
> print(total)
> ```
>
> > <solution-title></solution-title>
> > ```
> > total = 0
> > for word in ["red", "green", "blue"]:
> >     total = total + len(word)
> > print(total)
> > ```
> {: .solution}
>
> ```
> # List of word lengths: ["red", "green", "blue"] => [3, 5, 4]
> lengths = ____
> for word in ["red", "green", "blue"]:
>     lengths.____(____)
> print(lengths)
> ```
> > <solution-title></solution-title>
> > ```
> > lengths = []
> > for word in ["red", "green", "blue"]:
> >     lengths.append(len(word))
> > print(lengths)
> > ```
> {: .solution}
>
> ```
> # Concatenate all words: ["red", "green", "blue"] => "redgreenblue"
> words = ["red", "green", "blue"]
> result = ____
> for ____ in ____:
>     ____
> print(result)
> ```
> > <solution-title></solution-title>
> > ```
> > words = ["red", "green", "blue"]
> > result = ""
> > for word in words:
> >     result = result + word
> > print(result)
> > ```
> {: .solution}
>
> __Create an acronym:__ Starting from the list `["red", "green", "blue"]`, create the acronym `"RGB"` using
> a for loop.
>
> __Hint:__ You may need to use a string method to properly format the acronym.
> > <solution-title></solution-title>
> > ```
> > acronym = ""
> > for word in ["red", "green", "blue"]:
> >     acronym = acronym + word[0].upper()
> > print(acronym)
> > ```
> {: .solution}
{: .question}

```python
#Test your code here!
```

> ## Cumulative Sum
>
> Reorder and properly indent the lines of code below
> so that they print a list with the cumulative sum of data.
> The result should be `[1, 3, 5, 10]`.
>
> ```
> cumulative.append(total)
> for number in data:
> cumulative = []
> total += number
> total = 0
> print(cumulative)
> data = [1,2,2,5]
> ```
> > <solution-title></solution-title>
> > ```
> > total = 0
> > data = [1,2,2,5]
> > cumulative = []
> > for number in data:
> >     total += number
> >     cumulative.append(total)
> > print(cumulative)
> > ```
> {: .solution}
{: .question}

```python
# Test your code here!
```

> <question-title>A classic programmer test: Fizz Buzz</question-title>
> [FizzBuzz](https://en.wikipedia.org/wiki/Fizz_buzz) is a classic "test" question that is used in some job interviews to remove candidates who really do not understand programming. Your task is this:
>
> Write a for loop that loops over the numbers 1 to 50.
>
> - If the number is divisible by 3, write Fizz instead of the number
> - If the number is divisible by 5, write Buzz instead of the number
> - If the number is divisible by 3 and 5 both, write FizzBuzz instead of the number
> - Otherwise, write the number itself.
>
> > <solution-title></solution-title>
> > ```python
> > for i in range(1, 50):
> >     if i % 3 == 0 and i % 5 == 0:
> >         print("FizzBuzz")
> >     elif i % 3 == 0:
> >         print("Fizz")
> >     elif i % 5 == 0:
> >         print("Buzz")
> >     else:
> >         print(i)
> > ```
> {: .solution}
{: .question}

```python
# Do a FizzBuzz
```

> <question-title>Identifying Item Errors</question-title>
>
> 1. Read the code below and try to identify what the errors are
>    **without** running it.
> 2. Run the code, and read the error message. What type of error is it?
> 3. Fix the error.
>
> ```
> seasons = ['Spring', 'Summer', 'Fall', 'Winter']
> print(f'My favorite season is {seasons[4]}')
> ```
>
> > <solution-title></solution-title>
> > This list has 4 elements and the index to access the last element in the list is `3`.
> > ```
> > seasons = ['Spring', 'Summer', 'Fall', 'Winter']
> > print(f'My favorite season is {seasons[3]}')
> > ```
> {: .solution}
{: .question}

```python
# Fix me!
seasons = ['Spring', 'Summer', 'Fall', 'Winter']
print(f'My favorite season is {seasons[4]}')
```

> <question-title>Correct the errors</question-title>
>
> This code is completely missing indentation, it needs to be fixed. Can you make some guesses at how indented each line should be?
> ```
> data = [1, 3, 5, 9]
> acc = 0
> for i in data:
> if i < 4:
> acc = acc + i * 2
> else:
> acc = acc + i
> print(f'The value at {i} is {acc}')
> print(f'The answer is {acc}')
> ```
>
> > <solution-title></solution-title>
> >
> >
> > ```python
> > data = [1, 3, 5, 9]
> > acc = 0
> > # There is a : character at the end of this line, so you KNOW the next line
> > # must be indented.
> > for i in data:
> >     # Same here, another :
> >     if i < 4:
> >         acc = acc + i * 2
> >     # And again! Another :
> >     else:
> >         acc = acc + i
> > # But what about these lines?
> > print(f'The value at {i} is {acc}')
> > print(f'The answer is {acc}')
> > ```
> >
> > Here this code is actually ambiguous, we don't know how indented the two prints should be. This very synthetic example lacks good context, but there are three places it could be, with three different effects.
> >
> > There are two bits of knowledge we can use, however:
> > - the first print uses `i`, so it must be within the loop
> > - the second print cannot be indented more than the first print (Why? It would require a block like `for ... :` or `if .. :` to indent further.)
> >
> > The first option, no indentation, prints out the value once per loop, that seems good
> >
> > ```python
> > [...]
> >     else:
> >         acc = acc + i
> >     print(f'The value at {i} is {acc}')
> > ```
> >
> > The second, prints out the value only during the else case, not otherwise.
> >
> > ```python
> >     else:
> >         acc = acc + i
> >         print(f'The value at {i} is {acc}')
> > ```
> >
> > So that's probably wrong, and we should take the first option. That leaves two options for the final print, no indentation, or at the same level as our first print statement. We can guess that we probably want to print out the final result of the loop, and that it should not be indented.
> >
> > ```python
> > data = [1, 3, 5, 9]
> > acc = 0
> > for i in data:
> >     if i < 4:
> >         acc = acc + i * 2
> >     else:
> >         acc = acc + i
> >     print(f'The value at {i} is {acc}')
> > print(f'The answer is {acc}')
> > ```
> >
> {: .solution}
{: .question}

```python
# This code accidentally lost it's indentation! Can you fix it?
data = [1, 3, 5, 9]
acc = 0
for i in data:
if i < 4:
acc = acc + i * 2
else:
acc = acc + i
print(f'The value at {i} is {acc}')
print(f'The answer is {acc}')
```

> <question-title>Trimming a FASTQ string</question-title>
> Given a FASTQ string, and a list with quality scores, use `break` to print out just the good bit of DNA and it's quality score.
>
> ```
> # We've got a Read
> read = """
> @SEQ_ID
> GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
> +
> 55CCF>>>>>>CCCCCCC65!''*((((***+))%%%++)(%%%%).1***-+*''))**
> """.strip().split('\n')
>
> def quality_to_percent(q):
>     return 100 * (1 - (10 ** (q / -10)))
>
> sequence = read[1]
> quality_scores = [ord(x) - 33 for x in read[3]]
>
> for i in ... # TODO
> ```
>
> > <solution-title>Hint: Looping over two variables</solution-title>
> > There are two ways to do this, one you might be able to guess, and one that might be new:
> > 1. Loop over a `range()` using `len(sequence)`. Since `len(sequence) == len(quality_scores)`, when we access the Nth position of either, they match up.
> > 2. `zip(sequence, quality_scores)` will loop over both of these lists together. It produces a new list that looks like `[['G', 20], ['A', 20], ['T', 34]]`.
> {: .solution}
>
> > <solution-title></solution-title>
> > The naïve solution is quite easy and readable:
> > ```
> > for i in range(len(sequence)):
> >     if quality_scores[i] < 15:
> >         break
> >     print(f'Base {i} = {sequence[i]} with {quality_to_percent(quality_scores[i])}% accuracy')
> > ```
> > But we can make this a bit prettier using the `zip()` function:
> > ```
> > for base, score in zip(sequence, quality_scores):
> >     if score < 15:
> >         break
> >     print(f'Base = {base} with {quality_to_percent(score)}% accuracy')
> > ```
> > But note that we don't have the position in the list anymore, so we remove it from the print statement.
> {: .solution}
{: .question}

```python
# We've got a Read
read = """
@SEQ_ID
GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
+
55CCF>>>>>>CCCCCCC65!''*((((***+))%%%++)(%%%%).1***-+*''))**
""".strip().split('\n')

def quality_to_percent(q):
    return 100 * (1 - (10 ** (q / -10)))

# Extract the sequence
sequence = read[1]
# And the quality scores, and map those to the correct values.
quality_scores = [ord(x) - 33 for x in read[3]]

# Write something here
# That loops over BOTH the sequence and Quality Scores.
# And prints them out
# If the quality scores are `<15`, then break and quit printing.
for i in ...
```


topic: data-science
tutorial: gnmx-lecture2
title: Introduction to sequencing with Python (part one)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 1893085f9863a7feb9546dffbf060294
last_modified: 2026-02-26T21:12:14.880220
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture2/tutorial.html

content:
![Imgur image](https://i.imgur.com/1PCleoW.png)

# The problem

The difficulty with sequencing nucleic acids is nicely summarized by [Hutchinson 2007](http://dx.doi.org/10.1093/nar/gkm688):

1. The chemical properties of different DNA molecules were so similar that it appeared difficult to separate them.
2. The chain length of naturally occurring DNA molecules was much greater than for proteins and made complete sequencing seem unapproachable.
3. The 20 amino acid residues found in proteins have widely varying properties that had proven useful in the separation of peptides. The existence of only four bases in DNA therefore seemed to make sequencing a more difficult problem for DNA than for protein.
4. No base-specific DNAases were known. Protein sequencing had depended upon proteases that cleave adjacent to certain amino acids.

It is therefore not surprising that protein-sequencing was developed before DNA sequencing by [Sanger and Tuppy 1951](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1197535/). 

tRNA was the first complete nucleic acid sequenced (see pioneering work of [Robert Holley and colleagues](http://www.jstor.org/stable/1715055) and also [Holley's Nobel Lecture](https://www.nobelprize.org/nobel_prizes/medicine/laureates/1968/holley-lecture.pdf)). Conceptually, Holley's approach was similar to Sanger's protein sequencing: break molecule into small pieces with RNases, determine sequences of small fragments, and use overlaps between fragments to reconstruct (assemble) the final nucleotide sequence. 

The work on finding approaches to sequencing DNA molecules began in the late 60s and early 70s. One of the earliest contributions has been made by Ray Wu (Cornell) and Dave Kaiser (Stanford), who used _E. coli_ DNA polymerase to [incorporate radioactively labelled nucleotides into protruding ends of bacteriphage lambda](http://www.sciencedirect.com/science/article/pii/S0022283668800129?via%3Dihub). It took several more years for the development of more "high throughput" technologies by Sanger and Maxam/Gilbert. The Sanger technique has ultimately won over Maxam/Gilbert's protocol due to its relative simplicity (once dideoxynucleotides has become commercially available) and the fact that it required a smaller amount of starting material as the polymerase was used to generate fragments necessary for sequence determination. 

# Sanger/Coulson plus/minus method

><comment-title>Two Nobel prizes</comment-title>
>[Fred Sanger](https://www.nature.com/articles/505027a) is one of only [four people](https://en.wikipedia.org/wiki/Category:Nobel_laureates_with_multiple_Nobel_awards), who received two Nobel Prizes in their original form (for scientific, not societal, breakthroughs).
{: .comment}

This method builds on the idea of Wu and Kaiser (for *minus* part) and on the special property of DNA polymerase isolated from phage T4 (for *plus* part). The schematics of the method is given in the following figure (from [Sanger & Coulson: 1975](http://www.sciencedirect.com/science/article/pii/0022283675902132?via%3Dihub)):

![Plus/minus method](./images/sanger_plus_minus.png "Plus/minus method (from Sanger & Coulson 1975)")

In this method, a primer and DNA polymerase is used to synthesize DNA in the presence of P<sup>32</sup>-labeled nucleotides (only one of four is labeled). This generates P<sup>32</sup>-labeled copies of DNA being sequenced. These are then purified and (without denaturing) separated into two groups: *minus* and *plus*. Each group is further divided into four equal parts. 

In the case of *minus* polymerase a mix of nucleotides minus one is added to each of the four aliquotes: ACG (-T), ACT (-G), CGT (-A), AGT (-C). As a result in each case DNA strand is extended up to a missing nucleotide. 

In the case of plus only one nucleotide is added to each of the four aliquotes (+A, +C, +G, and +T) and T4 DNA polymerase is used. T4 DNA polymerase acts as an exonuclease that would degrade DNA
from 3'-end up to a nucleotide that is supplied in the reaction. 

The products of these are loaded into a denaturing polyacrylamide gel as eight tracks (four for minus and four for plus; from [Sanger & Coulson: 1975](http://www.sciencedirect.com/science/article/pii/0022283675902132?via%3Dihub)):

![Plus/minus gel](./images/sanger_plus_minus_gel.png "Plus/minus method gel radiograph (from Sanger & Coulson 1975)")

# Maxam/Gilbert chemical cleavage method

In this method DNA is terminally labeled with P<sup>32</sup>, separated into four equal aliquotes.  Two of these are treated with [Dimethyl sulfate (DMSO)](https://en.wikipedia.org/wiki/Dimethyl_sulfate) and the remaining two are treated with [hydrazine](https://en.wikipedia.org/wiki/Hydrazine). 

DMSO methylates G and A residues. Treatment of DMSO-incubated DNA with alkali at high temperature will break DNA chains at G and A with Gs being preferentially broken, while treatment of DMSO-incubated DNA with acid will preferentially break DNA at As. Likewise treating hydrazine-incubated DNA with [piperidine](https://en.wikipedia.org/wiki/Piperidine) breaks DNA at C and T, while DNA treated with hydrazine in the presence of NaCl preferentially breaks at Cs. The four reactions are then loaded on a gel generating the following picture (from [Maxam & Gilbert: 1977](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC392330/pdf/pnas00024-0174.pdf)):

![Maxam/Gilbert gel](./images/maxam_gilbert.jpeg "Radiograph of Maxam/Gilbert gel (from Maxam & Gilbert 1977)")

# Sanger dideoxy method

The original Sanger +/- method was not popular and had a number of technical limitations. In a new approach, Sanger took advantage of inhibitors that stop the extension of a DNA strand at particular nucleotides.
These inhibitors are dideoxy analogs of normal nucleotide triphosphates (from [Sanger et al. 1977](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC431765/pdf/pnas00043-0271.pdf)):

![Sanger ddN](./images/sanger_ddN.png "Sanger ddNTP gel (from Sanger et al. 1977)")

# Original approaches were laborious

In the original [Sanger paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC431765/pdf/pnas00043-0271.pdf) the authors sequenced bacteriophage phiX174 by using its own restriction fragments as primers. This was an ideal set up to show the proof of principle for the new method. This is because phiX174 DNA is homogeneous and can be isolated in large quantities. Now suppose that you would like to sequence a larger genome (say _E. coli_). Remember that the original version of Sanger method can only sequence fragments up to 200 nucleotides at a time. So to sequence the entire _E. coli_ genome (which by-the-way was not sequenced until [1997](http://science.sciencemag.org/content/277/5331/1453)) you would need to split the genome into multiple pieces and sequence each of them individually. This is hard because to produce a readable Sanger sequencing gel each sequence must be amplified to a suitable amount (around 1 nanogram) and be homogeneous (you cannot mix multiple DNA fragments in a single reaction as it will be impossible to interpret the gel). Molecular cloning enabled by the availability of commercially available restriction enzymes and cloning vectors simplified this process. Until the onset of next generation sequencing in 2005 the process for sequencing looked something like this:

* (**1**) - Generate a collection of fragments you want to sequence. It can be a collection of fragments from a genome that was mechanically sheared or just a single fragment generated by PCR.
* (**2**) - These fragment(s) are then cloned into a plasmid vector (we will talk about other types of vectors such as BACs later in the course).
* (**3**) - Vectors are transformed into bacterial cells and positive colonies (containing vectors with insert) are picked from an agar plate.
* (**4**) - Each colony now represents a unique piece of DNA. 
* (**5**) - An individual colony is used to seed a bacterial culture that is grown overnight.
* (**6**) - Plasmid DNA is isolated from this culture and now can be used for sequencing because it is (1) homogeneous and (2) we now have a sufficient amount.
* (**7**) - It is sequenced using universal primers. For example, the image below shows a map for pGEM-3Z plasmid (a pUC18 derivative). Its multiple cloning site is enlarged and sites for **T7** and **SP6** sequencing primers are shown.
* These are the **pads** I'm referring to in the lecture. These provide universal sites that can be used to sequence any insert in between. 


![pGEM3z](./images/pgem3z.png "Map of pGEM-3Z cloning vector (from Promega Inc.)")

Until the invention of NGS the above protocol was followed with some degree of automation. But you can see that it was quite laborious if the large number of fragments needed to be sequenced. This is because each of them needed to be subcloned and handled separately. This is in part why the Human Genome Project, which will be discussed in future lectures in detail, took so much time to complete. 

# Evolution of sequencing machines

The simplest possible sequencing machine is a [gel rig with polyacrylamide gel](https://en.wikipedia.org/wiki/Polyacrylamide_gel_electrophoresis). Sanger used it is his protocol obtaining the following results (from [Sanger et al. 1977](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC431765/pdf/pnas00043-0271.pdf)):

![Sanger gel](./images/sanger_gel.png "Typical polyacrylamide gel separating DNA fragments generate with Sanger's dideoxy method (from Sanger et al. 1977)")

Here for sequencing each fragment four separate reactions are performed (with ddA, ddT, ggC, and ddG) and four lanes on the gel are used. [One simplification of this process](http://www.jstor.org/stable/pdf/2879539.pdf) that
came in the 90s was to use fluorescently labeled dideoxy nucleotides. This is easier because everything can be performed in a single tube and uses a single lane on a gel
(from Applied Biosystems [support site](https://www3.appliedbiosystems.com/cms/groups/mcb_support/documents/generaldocuments/cms_041003.pdf)):

![PA versus chromotogram](./images/gel_versus_chromotogram.png "Comparison between a chromatogram and a polyacrylamide gel readouts (from Applied Biosystems)")

However, there is still substantial labor involved in pouring the gels, loading them, running machines, and cleaning everything post-run. A significant improvement was offered by the development of capillary electrophoresis allowing automation of liquid handling and sample loading. Although several manufacturers have been developing and selling such machines a _de facto_ standard in this area was (and still is) the Applied Biosystems (ABI) Genetics and DNA Analyzer systems. The highest throughput ABI system, 3730_xl_, had 96 capillaries and could automatically process 384 samples. 

# NGS!

384 samples may sound like a lot, but it is nothing if we are sequencing an entire genome. The beauty of NGS is that these technologies are not bound by sample handling logistics. They still require the preparation of libraries,
but once a library is made (which can be automated) it is processed more or less automatically to generate multiple copies of each fragment
(in the case of 454, Illumina, Ion Torrent, PacBio, Oxford Nanopore, Element, Complete Genomics etc...) and loaded onto the machine, where millions of individual fragments are sequenced simultaneously.
We will learn about these technologies later in this course.

# Reading

* 2001 [Overview of pyrosequencing methodology - Ronaghi](http://genome.cshlp.org/content/11/1/3)
* 2005 [Description of 454 process - Margulies et al.](http://www.nature.com/nature/journal/v437/n7057/pdf/nature03959.pdf)
* 2007 [History of pyrosequencing - Pål Nyrén](http://link.springer.com/protocol/10.1385/1-59745-377-3:1)
* 2007 [Errors in 454 data - Huse et al. ](http://genomebiology.com/content/pdf/gb-2007-8-7-r143.pdf)
* 2010 [Properties of 454 data - Balzer et al.](http://bioinformatics.oxfordjournals.org/content/26/18/i420.full.pdf+html)

## A few classical papers

In a series of now classical papers ([Paper 1](http://genome.cshlp.org/content/8/3/175.full), [Paper2](http://genome.cshlp.org/content/8/3/186.full)) Philip Green and co-workers have developed a quantitative framework for the analysis of data generated by automated DNA sequencers: 

![phred1](./images/phred1.png "The first paper")
![phred2](./images/phred2.png "The second paper")

In particular, they developed a standard metric for describing the reliability of base calls:

An important technical aspect of our work is the use of log-transformed error probabilities rather than untransformed ones, which facilitates working with error rates in the range of most importance (very close to 0).
Specifically, we define the quality value $$q$$ assigned to a base-call to be:

$$q = -10\times log_{10}(p)$$

where $$p$$ is the estimated error probability for that base-call. Thus a base-call having a probability of 1/1000 of being incorrect is assigned a quality value of 30. Note that high-quality values correspond to low error probabilities, and conversely.

We will be using the concept of "*quality score*" or "*phred-scaled quality score*" repeatedly in this course. 

## Myers - Green debate

Can we sequence a genome using the shotgun approach?

![Myers/Green debate](./images/myers_versus_green.png "Gene Myers versus Phil Green")

# Simulating Sanger sequencing with Python

[![xkcd1306](https://imgs.xkcd.com/comics/sigil_cycle.png)](https://xkcd.com/1306)

In this lesson we will cover some of the fundamental Python basics including variables, expressions, statements, and functions.

## Prep

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Within JupyterLab start a new Python3 notebook

## Preclass prep: Chapters [1](https://greenteapress.com/thinkpython2/html/thinkpython2002.html), [2](https://greenteapress.com/thinkpython2/html/thinkpython2003.html), [3](https://greenteapress.com/thinkpython2/html/thinkpython2004.html) from "Think Python"

## [Indentation](https://peps.python.org/pep-0008/#indentation) is everything!

><warning-title>Indentation or bust!</warning-title>
>Python is an indented language: code blocks are defined using indentation with [spaces](https://peps.python.org/pep-0008/#tabs-or-spaces)!
{: .warning}

In Python, indentation is used to indicate the scope of control structures such as `for` loops, `if` statements, and function and class definitions. The amount of indentation is not fixed, but it must be consistent within a block of code.
The recommended amount of indentation is 4 spaces, although some developers prefer to use 2 spaces. Indenting is important in Python because it is used to indicate the level of nesting and structure of the code,
which makes it easier to read and understand. Additionally, indentation is also used to indicate which lines of code are executed together as a block.

## The storyline

In this lecture, we will re-implement our Sanger sequencing simulator from the previous lecture and generate realistic gel images. 

## Generate a random sequence

First, we import a module called [`random`](https://docs.python.org/3/library/random.html) which contains a number of functions for generating and working with random numbers 

```python
import random
```
Next, we will write a simple loop that would generate a sequence of pre-set lengths:

```python
seq = ''
for _ in range(100):
    seq += random.choice('ATCG')
```

```python
seq
```

```
CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC
```

## Simulate one polymerase molecule

The code below iterates through each element of a sequence `seq` (assumed to be a string containing nucleotides) and it checks if the current nucleotide is equal to 'A'. If it is, it generates a random number between 0 and 1 using the `random.random()` function.

It then checks if the random number is greater than 0.5. If it is, the code does nothing and proceeds to the next iteration. If the random number is less than or equal to 0.5, the code adds the
lowercase version of the nucleotide ('a') to a string called `synthesized_strand` and then breaks out of the loop.

In every iteration of the loop, regardless of whether the nucleotide is 'A' or not, the code then adds the current nucleotide to the `synthesized_strand` string.

This means that when the current nucleotide is 'A', then the generated random number will decide whether the code will add the nucleotide 'A' or 'a' to the `synthesized_strand`, and it will break out of the loop
after adding the nucleotide to the `synthesized_strand`. To get a good idea of what is going on let's visualize the code execution in 

```python
synthesized_strand = ''

for nucleotide in seq:
    if nucleotide == 'A':
        d_or_dd = random.random()
        if d_or_dd > 0.5:
            None
        else:
            synthesized_strand += 'a'
            break
    synthesized_strand += nucleotide
```

This can be simplified by first removing `d_or_dd` variable:

```python
synthesized_strand = ''
for nucleotide in seq:
    if nucleotide == 'A':
        if random.random() > 0.5:
            None
        else:
            synthesized_strand += 'a'
            break
    synthesized_strand += nucleotide
print(synthesized_strand)
```

```
CTTGCGGCTATa
```

and removing an unnecessary group of `if ... else` statements:

```python
synthesized_strand = ''
for nucleotide in seq:
    if nucleotide == 'A' and random.random() > 0.5:
        synthesized_strand += 'a'
        break
    synthesized_strand += nucleotide
print(synthesized_strand)
```

```
CTTGCGGCTATAGGAATa
```

finally let's make `synthesized_strand += 'a'` a bit more generic:

```python
synthesized_strand = ''
for nucleotide in seq:
    if nucleotide == 'A' and random.random() > 0.5:
        synthesized_strand += nucleotide.lower()
        break
    synthesized_strand += nucleotide
print(synthesized_strand)
```

    CTTGCGGCTATAGGa


## Simulating multiple molecules

To simulate 10 polymerase molecules we simply wrap the code from above into a `for` loop:

```python
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'A' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    print(synthesized_strand)
```

    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTATa
    CTTGCGGCTATAGGa
    CTTGCGGCTa
    CTTGCGGCTa


One problem with this code is that does not actually save the newly synthesized strand: it simply prints it. To fix this we will create a [list](https://greenteapress.com/thinkpython2/html/thinkpython2011.html)
(or an array) called `new_strands` and initialize it by assigning an empty array to it:


```python
new_strands = []
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'A' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
```


```python
new_strands
```




    ['CTTGCGGCTa',
     'CTTGCGGCTa',
     'CTTGCGGCTATAGGAATAa',
     'CTTGCGGCTATAGGAATa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTa']



## Simulating multiple molecules and all nucleotides

And to repeat this for the remaining three nucleotides we will do the following crazy thing:


```python
new_strands = []
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'A' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
    
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'C' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
    
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'G' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
    
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'T' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
```


```python
len(new_strands)
```

    40

Repeating the same code four times is just plain stupid so instead we will write a function called `polymerase`. Here we need to worry about the scope of variables.  The scope of a variable refers to the regions
of the code where the variable can be accessed or modified. Variables that are defined within a certain block of code (such as a function or a loop) are said to have a *local* scope, meaning that they can
only be accessed within that block of code. Variables that are defined outside of any block of code are said to have a *global* scope, meaning that they can be accessed from anywhere in the code.

In most programming languages, a variable defined within a function has a local scope, and it can only be accessed within that function. If a variable with the same name is defined outside the function,
it will have a global scope and can be accessed from anywhere in the code. However, if a variable with the same name is defined within the function, it will take precedence over the global variable and will be used within the function.

There are also some languages that have block scope, where a variable defined within a block (such as an if statement or a for loop) can only be accessed within that block and not outside of it.

In Python, variables defined in the main module have a global scope and can be accessed from any function or module. Variables defined within a function have local scope, and they can only be
accessed within that function. Variables defined within a loop or a block can be accessed only within the scope of the loop or block.


```python
def ddN(number_of_iterations, template, base, ddN_ratio):
    new_strands = []
    for _ in range(number_of_iterations):
        synthesized_strand = ''
        for nucleotide in template:
            if nucleotide == base and random.random() > ddN_ratio:
                synthesized_strand += nucleotide.lower()
                break
            synthesized_strand += nucleotide
        new_strands.append(synthesized_strand)
    return(new_strands)
```


```python
ddN(10,seq,'A',0.5)
```

    ['CTTGCGGCTATAGGa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGAATa',
     'CTTGCGGCTa',
     'CTTGCGGCTa',
     'CTTGCGGCTa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATa']



To execute this function on all four types of ddNTPs with need to wrap it in a `for` loop iterating over the four possibilities:


```python
for nt in 'ATCG':
    ddN(10,seq,nt,0.5)
```

## A bit about [lists](https://greenteapress.com/thinkpython2/html/thinkpython2011.html)

To store the sequences being generated in the previous loop we will create and initialize a list called `seq_run`:


```python
seq_run = []
for nt in 'ATCG':
    seq_run.append(ddN(10,seq,nt,0.5))
```

you will see that the seq run is a two-dimensional list:


```python
seq_run
```




    [['CTTGCGGCTa',
      'CTTGCGGCTATAGGa',
      'CTTGCGGCTa',
      'CTTGCGGCTATa',
      'CTTGCGGCTATAGGa',
      'CTTGCGGCTATa',
      'CTTGCGGCTa',
      'CTTGCGGCTATAGGAATAa',
      'CTTGCGGCTa',
      'CTTGCGGCTATa'],
     ['CTTGCGGCt', 'Ct', 'Ct', 'Ct', 'Ct', 'CTt', 'Ct', 'Ct', 'CTTGCGGCt', 'Ct'],
     ['CTTGc',
      'c',
      'c',
      'c',
      'c',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGc',
      'c',
      'CTTGCGGCTATAGGAATAAAAGGc',
      'c',
      'CTTGc'],
     ['CTTg',
      'CTTGCg',
      'CTTg',
      'CTTg',
      'CTTGCg',
      'CTTGCGGCTATAGGAATAAAAg',
      'CTTGCGGCTATAGGAATAAAAg',
      'CTTg',
      'CTTg',
      'CTTg']]



as you will read in your next home assignment list elements can be addressed by "index". The first element has the number 0:


```python
seq_run[0]
```




    ['CTTGCGGCTa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATa',
     'CTTGCGGCTa',
     'CTTGCGGCTATAGGAATAa',
     'CTTGCGGCTa',
     'CTTGCGGCTATa']



## A bit about [dictionaries](https://greenteapress.com/thinkpython2/html/thinkpython2012.html)

Another way to store these data is in a dictionary, which is a collection of `key:value` pairs where a key and value can be anything:


```python
seq_run = {}
for nt in 'ATCG':
    seq_run[nt] = ddN(10,seq,nt,0.90)
```


```python
seq_run
```




    {'A': ['CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTa',
      'CTTGCGGCTATAGGAATAa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTa',
      'CTTGCGGCTa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCa',
      'CTTGCGGCTATAGGa',
      'CTTGCGGCTATAGGAATAAAa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAa',
      'CTTGCGGCTATAGGAa'],
     'T': ['CTTGCGGCTAt',
      'CTTGCGGCTATAGGAATAAAAGGCTTt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTAt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCAt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCAt',
      'CTt'],
     'C': ['CTTGCGGCTATAGGAATAAAAGGc',
      'CTTGCGGCTATAGGAATAAAAGGc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGAc',
      'c',
      'CTTGCGGCTATAGGAATAAAAGGc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTc',
      'CTTGCGGc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGAc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGc'],
     'G': ['CTTGCGGCTATAGg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTg',
      'CTTGCg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC',
      'CTTg',
      'CTTGCGGCTATAGGAATAAAAGg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGg',
      'CTTGCGGCTATAGGAATAAAAg']}



dictionary elements can be retrieved using a key:


```python
seq_run['A']
```




    ['CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTa',
     'CTTGCGGCTATAGGAATAa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTa',
     'CTTGCGGCTa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATAGGAATAAAa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAa',
     'CTTGCGGCTATAGGAa']



## Drawing a sequencing gel

Now that we can simulate and store newly synthesized sequencing strands terminated with ddNTPs let us try to draw a realistic representation of the sequencing gel. For this, we will use
several components that will be discussed in much greater detail in the upcoming lectures. These components are:

- [`pandas`](https://pandas.pydata.org/) - a dataframe manipulation tool
- [`altair`](https://altair-viz.github.io/) - a statistical visualization library built on top of `vega-light` visualization grammar

These two libraries will be used in almost all lectures concerning Python in this class.

[Gel electophoresis](https://en.wikipedia.org/wiki/Gel_electrophoresis) separates molecules based on mass, shape, or charge. In the case of DNA all molecules are universally negatively charged and thus will always migrate
to (+) electrode. All our molecules are linear single-stranded pieces (our gel is *denaturing*) and so the only physical/chemical characteristic that distinguishes them is *length*.
Therefore the first thing we will do is to convert our sequences into their lengths. For this, we will initialize a new dictionary called `seq_lengths`:


```python
seq_lengths = {'base':[],'length':[]}
for key in seq_run.keys():
    for sequence in seq_run[key]:
        seq_lengths['base'].append(key)
        seq_lengths['length'].append(len(sequence))
```


```python
seq_lengths
```




    {'base': ['A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G'],
     'length': [81,
      54,
      19,
      34,
      10,
      87,
      15,
      21,
      88,
      16,
      11,
      27,
      84,
      100,
      57,
      51,
      60,
      100,
      60,
      3,
      24,
      24,
      67,
      39,
      1,
      24,
      67,
      8,
      39,
      47,
      14,
      65,
      6,
      100,
      4,
      23,
      28,
      43,
      31,
      22]}



now let's import `pandas`:


```python
import pandas as pd
```

and inject `seq_lengths` into a pandas *dataframe*:


```python
sequences = pd.DataFrame(seq_lengths)
```

it looks pretty:


```python
sequences
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>base</th>
      <th>length</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>A</td>
      <td>81</td>
    </tr>
    <tr>
      <th>1</th>
      <td>A</td>
      <td>54</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A</td>
      <td>19</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A</td>
      <td>34</td>
    </tr>
    <tr>
      <th>4</th>
      <td>A</td>
      <td>10</td>
    </tr>
    <tr>
      <th>5</th>
      <td>A</td>
      <td>87</td>
    </tr>
    <tr>
      <th>6</th>
      <td>A</td>
      <td>15</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A</td>
      <td>21</td>
    </tr>
    <tr>
      <th>8</th>
      <td>A</td>
      <td>88</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A</td>
      <td>16</td>
    </tr>
    <tr>
      <th>10</th>
      <td>T</td>
      <td>11</td>
    </tr>
    <tr>
      <th>11</th>
      <td>T</td>
      <td>27</td>
    </tr>
    <tr>
      <th>12</th>
      <td>T</td>
      <td>84</td>
    </tr>
    <tr>
      <th>13</th>
      <td>T</td>
      <td>100</td>
    </tr>
    <tr>
      <th>14</th>
      <td>T</td>
      <td>57</td>
    </tr>
    <tr>
      <th>15</th>
      <td>T</td>
      <td>51</td>
    </tr>
    <tr>
      <th>16</th>
      <td>T</td>
      <td>60</td>
    </tr>
    <tr>
      <th>17</th>
      <td>T</td>
      <td>100</td>
    </tr>
    <tr>
      <th>18</th>
      <td>T</td>
      <td>60</td>
    </tr>
    <tr>
      <th>19</th>
      <td>T</td>
      <td>3</td>
    </tr>
    <tr>
      <th>20</th>
      <td>C</td>
      <td>24</td>
    </tr>
    <tr>
      <th>21</th>
      <td>C</td>
      <td>24</td>
    </tr>
    <tr>
      <th>22</th>
      <td>C</td>
      <td>67</td>
    </tr>
    <tr>
      <th>23</th>
      <td>C</td>
      <td>39</td>
    </tr>
    <tr>
      <th>24</th>
      <td>C</td>
      <td>1</td>
    </tr>
    <tr>
      <th>25</th>
      <td>C</td>
      <td>24</td>
    </tr>
    <tr>
      <th>26</th>
      <td>C</td>
      <td>67</td>
    </tr>
    <tr>
      <th>27</th>
      <td>C</td>
      <td>8</td>
    </tr>
    <tr>
      <th>28</th>
      <td>C</td>
      <td>39</td>
    </tr>
    <tr>
      <th>29</th>
      <td>C</td>
      <td>47</td>
    </tr>
    <tr>
      <th>30</th>
      <td>G</td>
      <td>14</td>
    </tr>
    <tr>
      <th>31</th>
      <td>G</td>
      <td>65</td>
    </tr>
    <tr>
      <th>32</th>
      <td>G</td>
      <td>6</td>
    </tr>
    <tr>
      <th>33</th>
      <td>G</td>
      <td>100</td>
    </tr>
    <tr>
      <th>34</th>
      <td>G</td>
      <td>4</td>
    </tr>
    <tr>
      <th>35</th>
      <td>G</td>
      <td>23</td>
    </tr>
    <tr>
      <th>36</th>
      <td>G</td>
      <td>28</td>
    </tr>
    <tr>
      <th>37</th>
      <td>G</td>
      <td>43</td>
    </tr>
    <tr>
      <th>38</th>
      <td>G</td>
      <td>31</td>
    </tr>
    <tr>
      <th>39</th>
      <td>G</td>
      <td>22</td>
    </tr>
  </tbody>
</table>
</div>

In our data, there is a number of DNA fragments that have identical length (just look at the dataframe above). We can condense these by grouping dataframe entries first by nucleotide (`['base']`) and then by length (`['length']`). For each group we will then compute `count` and put it into a new column named, ..., `count`:

```python
sequences_grouped_by_length = sequences.groupby(
    ['base','length']
).agg(
    count=pd.NamedAgg(
        column='length',
        aggfunc='count'
    )
).reset_index()
```

```python
sequences_grouped_by_length
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>base</th>
      <th>length</th>
      <th>count</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>A</td>
      <td>10</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>A</td>
      <td>15</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A</td>
      <td>16</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A</td>
      <td>19</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>A</td>
      <td>21</td>
      <td>1</td>
    </tr>
    <tr>
      <th>5</th>
      <td>A</td>
      <td>34</td>
      <td>1</td>
    </tr>
    <tr>
      <th>6</th>
      <td>A</td>
      <td>54</td>
      <td>1</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A</td>
      <td>81</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td>A</td>
      <td>87</td>
      <td>1</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A</td>
      <td>88</td>
      <td>1</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>11</th>
      <td>C</td>
      <td>8</td>
      <td>1</td>
    </tr>
    <tr>
      <th>12</th>
      <td>C</td>
      <td>24</td>
      <td>3</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C</td>
      <td>39</td>
      <td>2</td>
    </tr>
    <tr>
      <th>14</th>
      <td>C</td>
      <td>47</td>
      <td>1</td>
    </tr>
    <tr>
      <th>15</th>
      <td>C</td>
      <td>67</td>
      <td>2</td>
    </tr>
    <tr>
      <th>16</th>
      <td>G</td>
      <td>4</td>
      <td>1</td>
    </tr>
    <tr>
      <th>17</th>
      <td>G</td>
      <td>6</td>
      <td>1</td>
    </tr>
    <tr>
      <th>18</th>
      <td>G</td>
      <td>14</td>
      <td>1</td>
    </tr>
    <tr>
      <th>19</th>
      <td>G</td>
      <td>22</td>
      <td>1</td>
    </tr>
    <tr>
      <th>20</th>
      <td>G</td>
      <td>23</td>
      <td>1</td>
    </tr>
    <tr>
      <th>21</th>
      <td>G</td>
      <td>28</td>
      <td>1</td>
    </tr>
    <tr>
      <th>22</th>
      <td>G</td>
      <td>31</td>
      <td>1</td>
    </tr>
    <tr>
      <th>23</th>
      <td>G</td>
      <td>43</td>
      <td>1</td>
    </tr>
    <tr>
      <th>24</th>
      <td>G</td>
      <td>65</td>
      <td>1</td>
    </tr>
    <tr>
      <th>25</th>
      <td>G</td>
      <td>100</td>
      <td>1</td>
    </tr>
    <tr>
      <th>26</th>
      <td>T</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>27</th>
      <td>T</td>
      <td>11</td>
      <td>1</td>
    </tr>
    <tr>
      <th>28</th>
      <td>T</td>
      <td>27</td>
      <td>1</td>
    </tr>
    <tr>
      <th>29</th>
      <td>T</td>
      <td>51</td>
      <td>1</td>
    </tr>
    <tr>
      <th>30</th>
      <td>T</td>
      <td>57</td>
      <td>1</td>
    </tr>
    <tr>
      <th>31</th>
      <td>T</td>
      <td>60</td>
      <td>2</td>
    </tr>
    <tr>
      <th>32</th>
      <td>T</td>
      <td>84</td>
      <td>1</td>
    </tr>
    <tr>
      <th>33</th>
      <td>T</td>
      <td>100</td>
      <td>2</td>
    </tr>
  </tbody>
</table>
</div>

The following chart is created using the `alt.Chart()` function and passing the data as an argument. The `mark_tick()` function is used to create a tick chart with a thickness of 4 pixels.

The chart is encoded with two main axis:

- y-axis which represents the length of the data and is encoded by the `'length'` field of the data.
- x-axis which represents the base of the data and is encoded by the `'base'` field of the data.
The chart also encodes a color, it encodes the `'count'` field of the data, sets the legend to `None`, and uses the `'greys'` scale from the Altair library.

Finally, the chart properties are set to a width of 100 pixels and a height of 800 pixels.

```python
import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q'),
    x = alt.X('base'),
    color=alt.Color('count:Q',legend=None,
                    scale=alt.Scale(scheme="greys"))
).properties(
    width=100,
    height=800)
```

![Gel rendering 1](./images/gel1.svg "A simulated gel image")

And here is a color version of the same graph using just one line of the gel:


```python
import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q'),
    color=alt.Color('base:N',#legend=None,
                    scale=alt.Scale(scheme="set1"))
).properties(
    width=20,
    height=800)
```

![Gel rendering 2](./images/gel2.svg "A simulated gel image using \"colored dyes\"")

## Putting everything together

```python
# Generate random sequences

seq = ''
for _ in range(300):
    seq += random.choice('ATCG')
```

```python
seq
```

```python
# Function simulating a single run of a single polymerase molecule

def ddN(number_of_iterations, template, base, ddN_ratio):
    new_strands = []
    for _ in range(number_of_iterations):
        synthesized_strand = ''
        for nucleotide in template:
            if nucleotide == base and random.random() > ddN_ratio:
                synthesized_strand += nucleotide.lower()
                break
            synthesized_strand += nucleotide
        new_strands.append(synthesized_strand)
    return(new_strands)
```

```python
# Generating simulated sequencing run

seq_run = {}
for nt in 'ATCG':
    seq_run[nt] = ddN(100000,seq,nt,0.95)
```

```python
# Computing lengths

seq_lengths = {'base':[],'length':[]}
for key in seq_run.keys():
    for sequence in seq_run[key]:
        seq_lengths['base'].append(key)
        seq_lengths['length'].append(len(sequence))
```

```python
# Converting dictionary into Pandas dataframe

sequences = pd.DataFrame(seq_lengths)
```

```python
# Grouping by nucleotide and length

sequences_grouped_by_length = sequences.groupby(
    ['base','length']
).agg(
    count=pd.NamedAgg(
        column='length',
        aggfunc='count'
    )
).reset_index()
```

```python
# Plotting (note the quadratic scale for realism)

import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q',scale=alt.Scale(type='sqrt')),
    x = alt.X('base'),
    color=alt.Color('count:Q',legend=None,
                    scale=alt.Scale(type='log',scheme="greys")),
    tooltip='count:Q'
).properties(
    width=100,
    height=800)
```

![Gel rendering 3](./images/gel3.svg "A simulated gel image using quadratic scale")

```python
# Plotting using color

import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q',scale=alt.Scale(type="sqrt")),
    color=alt.Color('base:N',#legend=None,
                    scale=alt.Scale(scheme="set1")),
    opacity=alt.Opacity('count:N',legend=None),
    tooltip='count:Q'
).properties(
    width=20,
    height=800)
```

![Gel rendering 4](./images/gel4.svg "A simulated gel image using quadratic scale with colors")


topic: data-science
tutorial: gnmx-lecture5
title: Introduction to sequencing with Python (part four)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: e7fd37c2f11e59ba771fa561f37d41a6
last_modified: 2026-02-26T21:12:14.881661
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture5/tutorial.html

content:
# Reading and writing files in Python

First let's download a file we will be using to your notebooks:

```
!wget https://raw.githubusercontent.com/nekrut/BMMB554/master/2023/data/l9/mt_cds.fa
```

In Python, you can handle files using the built-in `open` function. The `open` function creates a file object, which you can use to read, write, or modify the file.

Here's an example of how to open a file for reading:


```python
f = open("mt_cds.fa", "r")
```

In this example, the open function takes two arguments: the name of the file, and the mode in which you want to open the file. The `r` mode indicates that you want to open the file for reading.

After you've opened the file, you can read its contents using the read method:

```python
contents = f.read()
print(contents)
```

You can also read the file line by line using the readline method:

```python
line = f.readline()
print(line)
```

When you're done reading the file, you should close it using the close method:

```python
f.close()
```

You can also use the `with` statement to automatically close the file when you're done:

```python
with open("mt_cds.fa", "r") as f:
    contents = f.read()
    print(contents)
```

You can also write to files using `write` method (note the `"w"` mode):

```python
f = open("sample.txt", "w")
f.write("This is a new line.")
f.close()
```

If you open an existing file in write mode, its contents will be overwritten. If you want to append to an existing file instead, you can use the `"a"` mode:

```python
f = open("sample.txt", "a")
f.write("This is another line.")
f.close()
```

When you're writing to a file, it's important to make sure you close the file when you're done. If you don't close the file, any changes you make may not be saved.

In addition to reading and writing text files, you can also use Python to handle binary files, such as images or audio files.

Let's download an image:

```
!wget https://imgs.xkcd.com/comics/file_extensions.png
```

Here's an example of how to read an image file:

```python
with open("file_extensions.png", "rb") as f:
    contents = f.read()
```

Note that when working with binary files, you must use the `"rb"` mode for reading and the `"wb"` mode for writing.

There are many more features and methods related to file handling in Python, but the basics covered here should be enough to get you started.

## [Fasta](https://en.wikipedia.org/wiki/FASTA_format)

Fasta is a file format that is commonly used to store biological sequences, such as DNA or protein sequences. In Python, you can read a Fasta file by opening the file, reading the lines one by one, and processing the data as needed.

Here's an example of how you might read a Fasta file in Python:

```python
sequences = {}
with open("mt_cds.fa", "r") as file:
  header = ""
  sequence = ""
  for line in file:
    line=line.rstrip()
    if line.startswith('>'):
      if header != "":
        sequences[header] = sequence
        sequence = ""
      header = line[1:]
    else:
      sequence += line
  if header != "":
    sequences[header] = sequence
```
The code above uses a `with` statement to open the file and read the lines one by one. If a line starts with a `">"`, it is assumed to be a header, and the current sequence is stored in the dictionary using the current header as the key. If the line does not start with a `">"`, it is assumed to be part of the current sequence.


## [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format)

Let's download a sample fastq file:

```python
!wget https://raw.githubusercontent.com/nekrut/BMMB554/master/2023/data/l9/reads.fq
```

Fastq is a file format that is commonly used to store high-throughput sequencing data. It consists of a series of records, each of which includes a header, a sequence, a quality score header, and a quality score string. In Python, you can read a Fastq file by opening the file, reading the lines four at a time, and processing the data as needed.

Here's an example of how you might read a Fastq file in Python:

```python
def read_fastq(file_path):
    records = []
    with open(file_path, "r") as f:
        while True:
            header = f.readline().strip()
            if header == "":
                break
            sequence = f.readline().strip()
            quality_header = f.readline().strip()
            quality = f.readline().strip()
            records.append((header, sequence, quality))
    return records
```
In this example, the `read_fastq` function takes a file path as an argument, and returns a list of records, where each record is a tuple of four strings: the header, the sequence, the quality score header, and the quality score string. The function uses a `while` loop to read the lines four at a time until the end of the file is reached.

You can use this function to read a Fastq file like this:

```python
records = read_fastq("reads.fq")
for header, sequence, quality_header, quality in records:
    print(header)
    print(sequence)
    print(quality_header)
    print(quality)
```
This will print the headers, sequences, quality score headers, and quality scores in the Fastq file. You can modify the read_fastq function to process the data in any way you need.


## [SAM](https://en.wikipedia.org/wiki/SAM_(file_format)) 

Let's download an example SAM file:

```
!wget https://raw.githubusercontent.com/nekrut/BMMB554/master/2023/data/l9/sam_example.sam
```

SAM (Sequence Alignment/Map) is a file format that is used to store the results of DNA sequencing alignments. In Python, you can read a SAM file by opening the file, reading the lines one by one, and processing the data as needed.

Here's an example of how you might read a SAM file in Python:

```python
def read_sam(file_path):
    records = []
    with open(file_path, "r") as f:
        for line in f:
            if line.startswith("@"):
                continue
            fields = line.strip().split("\t")
            records.append(fields)
    return records
```
In this example, the `read_sam` function takes a file path as an argument, and returns a list of records, where each record is a list of fields. The function uses a `with` statement to open the file and read the lines one by one. If a line starts with an `"@"`, it is assumed to be a header and is ignored. If the line does not start with an `"@"`, it is assumed to be a record, and the fields are extracted by splitting the line on tabs.

You can use this function to read a SAM file like this:

```python
records = read_sam("sam_example.sam")
for fields in records:
    print(fields)
```

This will print the fields in the SAM file. You can modify the read_sam function to process the data in any way you need. For example, you might want to extract specific fields, such as the reference name, the start position, and the cigar string.


topic: contributing
tutorial: design
title: Design and plan session, course, materials
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["learning-principles"]
tags_json: ["elixir", "train-the-trainers"]
content_hash: 6e3120afa30b6e26d9a29ef618f23b89
last_modified: 2026-02-26T21:12:14.707599
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/design/tutorial.html

content:
**Teaching and training**, core elements of academic life, can be enormously
rewarding but also quite challenging. Instructors are often required to perform
under various constraints, and frequently have to accommodate, engage and
motivate student cohorts with very different backgrounds and aptitudes in
limited time-frames. This can be daunting for experienced teachers, but
especially so for those who're relatively new to teaching and training.

> <comment-title>Definitions of key terms</comment-title>
>
> **Training**: instruction delivered via short courses designed to expand or
> build knowledge & practical skills in a given field, often conducted in the
> workplace or training centre to 'up-skill' members of a workforce
>
> **Teaching**: usually, instruction delivered over long time-scales via a
> series of courses (in schools, colleges, universities, etc.) designed to
> contribute to a formal programme that, if completed successfully, yields an
> accredited qualification in a given field (e.g., a degree)
>
{: .comment}

Formal education enterprises generally begin with **curriculum**
({% cite european2008terminology %}) design: this involves specifying

1. its purpose or **Teaching Goals (TGs)**;
2. its duration;
3. the **Knowledge, Skills and Abilities (KSAs)** intended to be achieved,
   expressed as a set of **Learning Outcomes (LOs)**;
4. how learners will demonstrate achievement of those LOs;
5. the materials, **Learning Experiences (LEs)** and assignments instructors
   will use to support learner progression towards the LOs;
6. the assessments for evaluating student learning and teaching effectiveness.

> <comment-title>Definitions of key terms</comment-title>
>
> **Curriculum**: the inventory of tasks involving the design, organisation &
> planning of an education or training enterprise, including specification of
> learning outcomes, content, materials & assessments, & arrangements for
> training teachers & trainers
>
> **Knowledge, Skills and Abilities (KSAs)**: list of special qualifications and
> personal attributes that learners should have after a training
> - **Knowledge**: the subjects, topics, and items of information that learners
>   should know
> - **Skills**: technical or manual proficiencies which are usually learned or
>   acquired through training.
> - **Abilities**: the present demonstrable capacity to apply several knowledge
>   and skills simultaneously in order to complete a task or perform an
>   observable behaviour.
>
> **Learning Outcomes (LOs)**: the KSAs that learners should be able to
> demonstrate after instruction, the tangible evidence that the teaching goals
> have been achieved; LOs are learner-centric
>
> **Learning Experience (LE)**: any setting or interaction in or via which
> learning takes place: e.g., a lecture, game, exercise, role-play, etc.
>
> **Teaching Goal (TG)**: the intentions of an instructor regarding the purpose
> of a curriculum/course/lesson/activity/set of materials; TGs are
> instructor-centric (also termed instructional objectives)
>
{: .comment}

Emphasis is placed on teaching and learning, and mechanisms for collecting
evidence that learners have changed over time
({% cite tractenberg2020guidelines %}). In this latter sense, the concept of
'curriculum' differentiates formal teaching from training, as formal programmes
usually afford time for learners both to be able to progress and to **demonstrate
their progression**.

By contrast, training courses are much shorter (measured in days or weeks,
rather than years); they hence necessarily focus on acquiring specific KSAs in
limited time-frames, generally without consideration of learner progression
beyond the course. However, the essential features of **effective curricula**
(i.e., those that achieve their stated LOs for the majority of learners) pertain
to instruction on any time-scale; they are thus also relevant for short courses,
and provide important considerations for those involved in, or embarking upon,
course design (whether face-to-face or online).

With this in mind, this tutorial outlines key steps of curriculum development –
and the role of Bloom's taxonomy ({% cite bloom1956taxonomy %}) – that can be
used to inform the design of effective courses.

> <comment-title>Sources</comment-title>
>
> This tutorial is significantly based on {% cite via2020course %}, one session
> of ELIXIR-GOBLET Train the Trainer curriculum and
> {% cite tractenberg2020guidelines %}
>
{: .comment}

> <hands-on-title>Select a topic for a 3 minute training - {% icon time %} 1 min - Silent reflection</hands-on-title>
>
> - Choose a topic for a 3 minutes training related to Galaxy
>
>   Examples:
>   - Introduction to Galaxy interface
>   - How to upload data to Galaxy
>   - Galaxy Training Network
>
> - Identify the target audience
> - Identify the prerequisites
{: .hands_on}

> <tip-title>Check the tutorial on principles of learning</tip-title>
>
> For more details about how learning works and values, check our
> [tutorial]({% link topics/contributing/tutorials/learning-principles/tutorial.md %})
>
{: .tip }

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Formal curriculum design

## Different types of curriculum

To set the scene for our considerations of course design, we examine some of the
foundations for effective curriculum development, drawing heavily on the
curriculum- and course- development guidelines developed by
{% cite tractenberg2020guidelines %}. Notable here is the fact that different
types of curriculum have been defined: *i.e.*, **intended**, **implemented**,
**attained** and **hidden curricula**. Recognising the existence of different
curriculum types (or, perhaps, different curricular outcomes) is important
because, while the intended curriculum is the starting point, it may not be the
curriculum actually attained: *i.e.*, what you aimed to teach and what students
actually learned may not be the same.

To improve outcomes, differences between the intended and attained curricula
need to be minimised. The only way to discover the attained curriculum is to
find out what learning actually occurred. This requires **actionable**
evaluation, to assess whether the teaching goals (TGs) and learning outcomes (LOs) were achieved, to identify
weaknesses in the implementation and to highlight improvements needed to
remediate them. These considerations are key to developing effective curricula
and courses.

> <comment-title>Definitions of key terms</comment-title>
>
> **Actionable**: supportive of a decision, or the taking of some action by a
> learner, instructor or institution
>
> **Attained curriculum**: what learners actually acquire & can demonstrate
> having followed the implemented curriculum ({% cite unesco-ibe-glossary %})
>
> **Hidden curriculum**: unintended curricular effects: unofficial norms,
> behaviours & values that are transferred (not necessarily consciously) by the
> school culture or ethos; this recognises that schooling happens in broad
> social & cultural environments that influence learning ({% cite glossary-education-reform %})
>
> **Implemented curriculum**: or taught curriculum, how the intended curriculum
> is delivered in practice: *i.e.*, the teaching & learning activities, & the
> interactions between learners & teachers, & among learners
>
> **Intended curriculum**: the formal specification of KSAs that students are
> expected to achieve & be able to demonstrate having followed the implemented
> curriculum
>
{: .comment}

> <hands-on-title>Define audience and teaching goal - {% icon time %} 10 min - Silent reflection</hands-on-title>
>
> 1. Take the topic you chose before
> 2. Define teaching objectives
>
>    Describe your goals and intentions as instructor:
>    - stakeholders
>    - potential professions
>    - desired performance and/or competences of course completers
>    - duration
>
{: .hands_on}

## Structuring curriculum design

Curriculum design benefits from being systematic: structured approaches help to
orchestrate and clarify what will be taught, **why it will be taught** and how;
they also afford opportunities to evaluate what does and doesn't work, and hence
what needs to change, ultimately leading to improvements in learning outcomes
({% cite fink2013creating %}, {% cite diamond2008designing %},
{% cite nilson2016teaching %}). Several different frameworks have been devised to
facilitate the design process, but each is motivated by the same underlying
philosophy: to help formulate programmes that promote **meaningful and enduring
learning**. If we're to understand whether we've really achieved this, we must

1. determine the purpose of the programme (what needs it addresses, why it's
   being developed, what learners will gain from it, why it's important),
2. define the intended LOs, and
3. develop **assessment** and evaluation mechanisms that will allow us to
   measure whether the programme successfully met its goals.

> <comment-title>Definitions of key terms</comment-title>
>
> **Assessment**: the evaluation or estimation of the nature, quality or ability
> of someone or something
>
{: .comment}

Not surprisingly, the same principles apply to course design. The process may
seem daunting, but for the sake of simplicity, we focus here on one model: *i.e.*,
that proposed by {% cite nicholls2002developing %}. Before discussing
this further, however, it's helpful to consider another very important tool used
in teaching and learning – Bloom's taxonomy ({% cite bloom1956taxonomy %}).

### Bloom's taxonomy of cognitive complexity

Learning taxonomies are useful tools that can help both to formulate and clarify
LOs, and to arrange them on a scale of increasing complexity. Bloom’s taxonomy
({% cite bloom1956taxonomy %}), probably the most easily understood and widely
used today, features a **six-level hierarchy of cognitive complexity**, ranging
from **Remember** (being able to recall facts and basic concepts) to **Evaluate**
being able to defend opinions or decisions), as illustrated in the following
figure:

![6 levels of Bloom's taxonomy (and their verbs) from bottom (lower-order thinking skills) to top (higher-order 'critical' thinking skills): Remember (recall or reiterate information) with memorize, recognise, identify, describe, reproduce, list, define, label, name, state, outline, order, arrange; Understand (demonstrate understanding of facts) with distinguish, classify, select, review, discuss, indicate, explain, estimate, locate, summarise; Apply (apply knowledge to real situations) with operate, manipulate, experiment, choose, modify, prepare, produce; Analyze (resolve ideas into simple parts, identify patterns) with calculate, examine, model, test, break down, infer, predict, solve; Synthesise (pull ideas into a coherent whole, create new ideas) with combine, formulate, illustrate, imagine, design, invent, compose; Evaluate (make & defend judgements, assess theories & outcomes) with critique, compare ideas, solve, recommend, rate](images/bloom_taxonomy.png "Bloom’s taxonomy of cognitive complexity, & a sample of associated verbs that can be used to create LOs. The verbs reflect a developmental trajectory, from lower- to higher-order thinking skills; they should be concretely observable & relate to how achievement of target LOs will be assessed. Source: {% cite via2020course %}")

As can be seen from the figure, each Bloom's level is accompanied by a set of
active verbs that express expected, measurable learner behaviours at that level:
*e.g.*, achieving the level **Understand** means to be able to classify, select or
explain a piece of information: here, _classify, select, explain_ are
observable, assessable behaviours that can be readily encapsulated in coherent
LOs.

Typical illustrations of the taxonomy, like that in the figure, depict
successive cognitive levels, suggesting that learners must achieve one level
before advancing to the next, implying a **developmental trajectory** from lower-
to higher-order cognitive skills. However, this structure should not be regarded
as completely rigid; indeed, {% cite anderson2001taxonomy %} published a revised
version in 2001 in which they placed **Synthesise** (the ability to create new or
original work) at the top of the hierarchy in place of **Evaluate**.
Notwithstanding the minutiae, it is perhaps more fruitful to regard the taxonomy
as a continuum or spectrum of cognitive levels, where each merges into the next,
providing a structured tool to help express measurable, assessable LOs, in which
the cognitive levels are made explicit.

Ultimately, we can use Bloom's taxonomy to **align instruction with learners' levels of complexity of thinking** (and experience). In practice, you can use the Bloom’s levels of cognitive complexity (and the [accompanied actionable verbs in the previous link](https://tips.uark.edu/blooms-taxonomy-verb-chart/)) to:
* write Learning Outcomes
* design instruction and learning experiences i.e. how you are to teach/train, which infrastructure and how to present the materials etc
* assess learning (taking place during curricula or course and after)

The cognitive aspects embodied in LOs are important. Teaching should promote
more complex behaviours than just recall or recognition (unless remembering is
the intended LO), and push learners to achieve greater cognitive complexity
({% cite nilson2016teaching %}, {% cite ambrose2010learning %},
{% cite weinstein2018understanding %}, {% cite roediger2006power %},
{% cite jensen2014teaching %}). This can be done by embedding development in
learning activities and materials, and ensuring that LOs reflect the lowest to
the highest levels of cognitive complexity realistically achievable on
completing those activities or having engaged with those materials (realistic
aims are key, especially for short courses: *e.g.*, expecting learners with no
prior subject knowledge to achieve the level **Evaluate**, say in a 1-day course,
will guarantee failure and frustration for learners and instructors alike).

# Nicholl's five phases of curriculum design

The backdrop for our considerations of course design is Nicholls' paradigm for
curriculum development, illustrated in:

![(in a clockwise direction) Phase 1: Learning Outcomes (Los) --> Simple, measurable, achievable, realistic, time-bound? --> Phase 2: Learning Experiences (LEs) --> LEs aligned with LOs? --> Phase 3: Content --> Support LEs? Promote LOs? --> Phase 4: Assessment --> Aligned with LEs? Support progress towards LOs? --> Phase 5: Evaluation --> LOs achievable by most learners? Content, LEs & assessments support LOs?](images/nicholl_paradigme.png "Nicholls’ phases of curriculum design & their dependencies. For each phase, key considerations are shown (diamonds). Where these are not satisfied, that or previous phases should be revisited (red arrows), otherwise it is safe to move to the next phase(s) (green arrows). When all considerations are satisfied, the curriculum or course can be characterised, with concrete evidence, as successful (star). By simplified we mean that here you only see the arrows going from one box to the following in line. However, usually we need to revise and go back one or several boxes if we have failed to specify or articulated Learning outcomes from the beginning. This is what we mean by Learning Outcomes drives the whole process. Source: {% cite via2020course %}")

Its five-phase structure has been briefly summarised by
{% cite tractenberg2020guidelines %}, as follows:

1.  Select LOs;
2.  Select or develop LEs that will help learners achieve the LOs;
3.  Select or develop content relevant to LOs;
4.  Develop assessments to ensure learners progress toward LOs;
5.  Evaluate the effectiveness of LEs for leading learners to LOs.

As can be seen from [Figure 2](#figure-2), the model's phases are interdependent; all are
ultimately dependent on the first – defining LOs. Moreover, the phases are
iterative: this means that LOs influence later decisions, but later decisions
may also reflect backwards, thereby providing opportunities to check for
alignment of each phase to the target LOs (in other words, to ensure that
successive phases are mutually consistent with, and supportive of, the LOs).
Thus, the role of LOs is pivotal: they must have specific characteristics to
function, and support each of the other phases as they do.

[Figure 2](#figure-2) illuminates an important feature of the model: that **LOs are the
starting point, and drive all decision-making**. This is just as true for courses
as it is for programmes ({% cite diamond2008designing %}). Missing from the model,
however, is the dependence of LOs on a hierarchy of cognitive complexity that
establishes a developmental trajectory, like that seen in Bloom's taxonomy. We
reflect on this crucial point, and its relevance for course design, in the
discussion of each of the five phases below.

## Define intended Learning Outcomes (LOs)

Just as for curricula, Phase 1 of course design begins with stating the LOs. To
help formulate LOs, it's important to take a step back and think about what you
aim to achieve (i.e., what are your TGs and the KSAs you intend to be achieved?),
how you propose to get there, and how you’ll know you succeeded.
{% cite messick1994interplay %} encapsulated this process in the form of three
succinct questions:

1. What KSAs are the targets of instruction (and assessment)?
2. What learner actions/behaviours will reveal these KSAs?
3. What tasks will elicit these specific actions or behaviours?

These questions were originally posed in 1994 in the context of assessment.
Their focus on KSAs – the LOs – thus guides not only the creation of relevant
tasks (to reveal the target KSAs) but also the rational development of
**appropriate** assessments: i.e., they provide a framework for, and clarify,
**what to assess**. The questions can thus support all phases of course
development, starting with the selection of intended KSAs stated in a set of
LOs.

The "Learning outcomes" (LOs) are essentially statements that tell what the learners will be to do at the end of the session. For this reason, every well-designed training should start with LOs.

The first step of good instructional design consists is identifying the set of Knowledge, Skills, Abilities (KSAs) that learners are expected to acquire by the end of instruction.

### Teaching objectives and Learning Outcomes

**Objectives** are statements of what you are setting out to teach, although expressed as if the students were going to learn it. Despite Aims are usually intended to be more general and encompassing than Objectives, they are essential the same.

**Outcomes** (more accurately "desired outcomes") are statements of what you might (in principle) assess. You may not end up assessing all of them, but they are statements of what a student will know or be able to do, if she or he has learned everything in the course or session.

### Learning objectives and learning outcomes

> <comment-title>Sources</comment-title>
>
> The following section is adapted from the webpage [http://provost.rpi.edu/learning-assessment/learning-outcomes/objectives-vs-outcomes](http://provost.rpi.edu/learning-assessment/learning-outcomes/objectives-vs-outcomes) and the [Operational definitions:](https://irds.stanford.edu/sites/g/files/sbiybj10071/f/clo.pdf) document from the [Rensselaer Polytechnic Institute Office of the Provost](http://provost.rpi.edu/institutional-research).
{: .comment}

The terms Learning Objectives and Learning Outcomes are often used interchangeably by the community. Even worse, they are often abbreviated by the acronym (LO), thus introducing even more ambiguity. In our opinion there is no need. Objectives (aims or goals) and Outcomes (results) may overlap, but are not genuinely the same.

Furthermore, American educational psychologists tend to use "Learning Objectives" to mean "Learning Outcomes" and "Teaching aims, goals or objectives" to mean "Learning Objectives".

In the attempt of removing ambiguities we will use the term **Teaching Objectives** to mean:

* Objectives describe the goals and intentions of the instructor.
* Objectives, often termed the input in the course, state the purpose and goals of the course.
* Objectives focus on content and skills important within the classroom or programme.
* Objectives may describe what the instructors will do.
* Objectives can often be numerous, specific, and detailed.

And the term **Learning Outcomes** to mean:

* Student Learning Outcomes catalog the overarching "products" of the course and are the evidence that the goals or objectives were achieved.
* Learning Outcomes are statements that describe or list measurable and essential mastered content-knowledge—reflecting skills, competencies, and knowledge that students have achieved and can demonstrate upon successfully completing a course.
* Outcomes express higher-level thinking skills that integrate course content and activities and can be observed as a behavior, skill, or discrete useable knowledge upon completing the
course.
* Outcomes are exactly what assessments are intended to show – specifically what the student will be able to do upon completing the course.
* An assessable outcome can be displayed or observed and evaluated against criteria.
* Outcomes are clear and measurable criteria for guiding the teaching, learning, and assessment process in the course.
* Statements expressing which KSAs learners will be able to demonstrate upon completion of a learning experience or a sequence of learning experiences
* What learners will be able to do for the end of instruction and the teacher / instructor can in principle evaluate

Going beyond the definition however, a learning outcome needs to be also practical, i.e. there is a need to assign observable actions in order to let teacher assess them.

#### How to write learning outcomes using OBSERVABLE verbs

In order to write learning outcomes, you can use the following scheme:

1. Think about what learners will be able to do by the end of instruction
2. Use the sentences:

    *By the end of the lesson (session/course/instruction) the learner will be able to.........*

    OR

    *The successful learner will be able to......*

3. Replace dots with a verb that you can assess (name, explain, solve, distinguish, etc.).
4. Avoid verbs that are open to many interpretations: appreciate, have faith in, know, learn, understand, believe
5. Keep in mind that learning outcomes can be as specific as you want. You may even think about a different learning outcome for each task you assign to learners.

Identify observable and non observable actions (verbs)

> <hands-on-title>Identify observable and non observable actions (verbs) - {% icon time %} 3 min</hands-on-title>
>
> Here is a list of verbs:
> - Compare
> - Classify
> - Distinguish
> - Explain
> - Believe
> - Give examples of
> - Define
> - Learn
> - Describe
> - List
> - Recognize
> - Know
> - Connect
> - Measure
> - Identify
> - Have faith in
>
> * Write in the shared notes which ones describe observable actions
> * Write in the shared notes which ones are instead open to multiple interpretation
>
{: .hands_on}

<!--{% include _includes/quiz.html id="verbs.yaml" %}-->

> <tip-title>Observable and non observable actions (verbs)</tip-title>
>
> In classifying verbs, it may help to focus on how you would assess the corresponding action / behaviour.
> Example: "identify"
> - Q: What would I do to assess whether a learner is able to identify something?
> - A: Ask them to do this challenge.
{: .tip}


<!--
> <comment-title>What are Learning Outcomes (LOs)?</comment-title>
>
> - Statements expressing which KSAs learners will be able to demonstrate upon
>   completion of a learning experience or a sequence of learning experiences
> - What learners will be able to do for the end of a lesson and the teacher can
>   in principle evaluate
>
{: .comment}
-->



> <question-title>Evaluate LOs</question-title>
>
> 1. Select a tutorial or a course
>
>    Several possibilities:
>    1. Take a course you currently run, plan to run or have run in the
>       past
>    2. Take your favorite tutorial from the [GTN Material]({{ site.baseurl }})
>    3. Take the [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %})
>       introducing to data import, that we created for this lesson
>
> 2. Check the content
> 3. Are its intended LOs stated?
>
>    > <solution-title>for any GTN tutorials, including the dummy one</solution-title>
>    >
>    > LOs are always stated on the top for the tutorial, in the Overview box
>    >
>    {: .solution}
>
{: .question}

Writing coherent LOs is challenging: they must contain appropriate (Bloom's)
verbs ([Figure 1](#figure-1)) that express measurable, observable and assessable actions,
accurately describing what successful learners will be able to do – **and at what
level of cognitive complexity** – after instruction.

Various characteristics of, and principles for articulating, LOs have been
published (e.g. {% cite stanford-institutional-research-decision-support %} or
{% cite national2016higher %}): some of these are listed briefly in the box
below.

> <comment-title>Learning outcomes should</comment-title>
>
> - **be specific & well defined**: LOs should concisely state the specific KSAs
>   that learners should develop as a result of instruction;
> - **be realistic**: LOs must be attainable given the context and resources
>   available for instruction, and consistent with learners’ abilities,
>   developmental levels, prerequisite KSAs, and the time needed vs. time
>   available to achieve them;
> - **rely on active verbs, phrased in the future tense**: LOs should be stated
>   in terms of what successful learners will be able to do as a result of
>   instruction;
> - **focus on learning products, not the learning process**: LOs should not
>   state what instructors will do during instruction, but what learners will be
>   able to do as result of instruction;
> - **be simple, not compound**: LOs shouldn’t include compound statements that
>   join two or more KSAs into one statement;
> - **be appropriate in number**: LOs should be deliverable and assessable
>   within the time available for instruction;
> - **support assessment that generates actionable evidence**: here, actionable
>   means supportive of a decision, or taking some action by a learner or
>   instructor.
>
{: .comment}

<!-- NILOA five principles for articulating LOs -->

> <hands-on-title>Learning outcomes - {% icon time %} 2 min - Silent working</hands-on-title>
>
> 1. Take the small lesson you selected at the beginning
> 2. Try to jot a few LOs down
>
{: .hands_on}

Given their detail and complexity, and the importance of aligning the
instructional inputs you devise with the outcomes for learners you intend, it
can be hard to know where to start. This possibly explains why it may feel
easier to begin developing a course by selecting its content rather than first
trying to understand its impact on student learning. Nevertheless, ensuring that
target LOs meet, or are consistent with, the characteristics outlined in the box
below helps to promote better alignment of instructional inputs and learner
outcomes.

<!--{% include _includes/quiz.html id="evaluate_learning_outcomes.yaml" %}-->

> <question-title>Evaluate LOs</question-title>
>
> "By the end of the course learners will know the Pythagora's theorem
> 
> What do you think about this LO?
>
> > <solution-title>Evaluate Learning outcomes - Solution</solution-title>
> >
> > "By the end of the course learners will know the Pythagora's theorem"
> >
> >  This LO is not good:
> >
> > - How to assess whether learners know the Pythagora's theorem?
> > - What do we mean by "knowing" the Pythagorean theorem?
> >   - They are able state it?
> >   - They are able explain it?
> >   - They are able apply it?
> >   - They are able demonstrate it?
> >   - They are able to use it in solving problem?
> >
> > It would make more sense to ask: "What will learners be able to do to show they know the Pythagorean theorem?"
> >
> {: .solution}
{: .question}

In short, when defining LOs, the key question to ask is, are they SMART?

- **S**pecific / **S**imple
- **M**easurable
- **A**chievable / **A**ttainable
- **R**ealistic / **R**elevant
- **T**ime-bound / **T**ransferable

If they do not satisfy this test, they should be revised; only when they meet
these criteria is it safe to progress to Phase 2, as shown in [Figure 2](#figure-2).
SMART LOs will be the roadmap as we plan out instructional strategies, write the
content and **create assessments**.

> <hands-on-title>Create SMART LOs - {% icon time %} 10 min - Silent working</hands-on-title>
>
> 1. Take the LOs you created
> 2. Consider if your LOs are SMART
> 3. Try to revise any of them that do not meet the SMART criteria as
>    follows:
>
>    1. Select an active verb that can (in principle) be observed & assessed
>    2. Complete the sentence
>
>       *"At the end of this course, learners will be able to..."*
>
>       If it helps, review the verbs of the Bloom's taxonomy listed in [Figure 1](#figure-1).
>
>       It's important to:
>
>       - Focus here on what learners will be able to do at the end of
>         instruction, *e.g.* *will they be able to...*
>         - *... describe its content?*,
>         - *... explain a concept?*,
>         - *... implement an algorithm?,
>         - *... solve a problem?*,
>         - *... evaluate results?*
>
>       - Avoid verbs open to multiple interpretation
>       - Use a verb that describes an observable action
>
> 3. Determine how well you have structured your LOs
>
>    1. Visit the [Intended Learning Outcome Advisor](https://web.cs.manchester.ac.uk/iloadvisor)
>    2. Paste each LOs into the input box
>    3. Press the **SUBMIT** button
>
>    How well did you do?
>
> 4. Consider revising your LOs if the Advisor identified any issues
>
{: .hands_on}

<!-- ELIXIR TtT

> <hands-on-title>Define audience, goal and outcomes (10 min)</hands-on-title>
>
> 5. Identify the learning experiences
> 6. Start thinking about the content - but don’t spend time in preparing this
>    yet
>
{: .hands_on}

-->

Ultimately, **LOs provide the necessary structure and context for decision-making
by instructors (and learners), hence their primary role in course design**.

## Select LEs that will lead to the LOs

Phase 2 involves identifying the most appropriate LEs to lead learners to the
intended LOs. It is important to appreciate that different LEs can lead learners
to demonstrate different Bloom's-level accomplishments: *e.g.*,
- lectures differ from problem-sets – solving problems helps learners to work
  with, and manipulate, information rather than passively listening to it;
- similarly, lab exercises differ from writing computer programs – writing
  original code affords learners the opportunity to create something new rather
  than simply following instructions.

Some example LEs are listed in Table 1, together with the Bloom's level and the
kinds of TG (Teaching Goals) and LO that each may support.

| Learning experience | Highest Bloom's levels supported | Example TG(s) _"This LE will allow me to..."_                                                                                                                       | Example LO(s) _"Learners will be able to..."_                                                                                 |
| ------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Lecture, webinar    | Remember, Comprehend             | Inspire learners, ignite learners' enthusiasm, clarify/explain a concept, provide an overview, give context, summarise content                                    | _list_ the key points of the lecture/webinar, _summarise_ take home message(s)                                              |
| Exercise, practical | Apply, Analyse                   | Help learners digest course materials, solve typical problems, apply knowledge, show how to do things with appropriate guidance, give an idea of how a tool works | _follow_ a set of instructions or protocol, _calculate_ a set of results or outcomes from a given protocol                  |
| Flipped class       | Apply, Analyse                   | Teach learners how to formulate questions, help learners to memorise new information & concepts, or analyse & understand course materials                         | _summarise_ the content material, \*ask\* appropriate questions                                                             |
| Peer instruction    | Synthesise, Evaluate             | Prepare learners to defend an argument, give learners opportunities to explain things, thereby helping to develop critical thinking & awareness                   | _explain_ how they solved an exercise, _evaluate_ others' choices/decisions, _diagnose_ errors in the exercise-solving task |
| Group discussion    | Synthesise, Evaluate             | Give learners opportunities to practice questioning, develop new ideas & critical thinking                                                                        | _communicate_ their own ideas, _defend_ their own opinions                                                                  |
| Group work          | Synthesise, Evaluate             | Promote collaborative work & peer instruction, provide opportunities for giving/receiving feedback, & digesting course materials                                  | _provide_ feedback on their peers' work, _share_ ideas, _explain_ the advantages of team-work                               |
| Problem-solving     | Synthesise, Evaluate             | Promote learner abilities to identify & evaluate solutions, develop new ideas, make decisions, evaluate decision effectiveness, troubleshoot                      | _diagnose_ faulty reasoning or an underperforming result, _correct_ errors                                                  |

> <comment-title>Definitions of key terms</comment-title>
>
> **Exercise**: an activity designed to help learners to mentally put into
> practice learned skills & knowledge
>
> **Flipped class**: a learner-centred approach in which students are introduced
> to new topics prior to class; class time is then used to explore those topics
> in greater depth via interactive activities
>
> **Group discussion**: an in-class, learner-centred approach in which students
> discuss ideas, solve problems &/or answer questions, guided by the instructor
>
> **Group work**: a learner-centred approach in which students are organised
> into groups (& perhaps assigned specific roles) & are given tasks to perform
> collaboratively
>
> **Lecture**: a didactic approach in which oral presentation is used to
> describe & explain concepts & to impart facts
>
> **Peer-instruction**: an interactive, in-class, learner-centred approach in
> which groups of two or more students briefly discuss a question or assignment
> given by the instructor
>
> **Practical**: an activity to put into practice learned skills & knowledge,
> generally in a lab setting
>
> **Problem-solving**: a learner-centred approach in which students are required
> to systematically investigate a problem by building or determining the best
> strategy to solve it (using what is known to discover what is not known)
>
> **Webinar**: a lecture delivered online
>
{: .comment}

Having defined SMART LOs in Phase 1, Phase 2 thus hinges on choosing the most
appropriate LEs to best lead learners towards them: if LOs include, for example,
being able to write a computer program, then the LEs must allow learners to
apply the knowledge they have acquired and to demonstrate that they’ve written a
piece of functional code: i.e., LEs and LOs must be aligned (if they are not,
this can lead directly to a gap between instructional inputs and intended
outcomes, which is one reason why course evaluation to detect such misalignments
is so crucial). If LEs don’t satisfy this criterion, alternative LEs should be
found, or the LOs should be revisited and revised before progressing to Phase 3
(as shown in Figure 2).

Sometimes, it may be necessary to use specific LEs: e.g., if a course is
lecture-based, it may not be possible to choose alternatives; or a particular
teaching scenario may not allow for an ideal LE. Even if you’re not in a
position to select the most appropriate LEs, the LOs must still be consistent
with the given LEs; and if those LEs won’t help learners to achieve the LOs,
then the LOs should be revised – i.e., LOs are still the most important feature
to consider.**The key is to determine exactly what specific LEs can contribute
to learning or how they will help move learners towards the LOs.**

> <question-title>Evaluate LEs in GTN tutorials</question-title>
>
> - Think of the course or tutorial you selected before (*e.g.*, the
>   [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %}))
>
> - Are any of the LEs listed in Table 1 used in the course or tutorial you
>   selected?
>
>   > <solution-title>for GTN tutorial, including the dummy one</solution-title>
>   >
>   > Most of the GTN tutorials are using Practical and Exercises as LEs.
>   >
>   {: .solution}
>
> - What kinds of achievement or learning are they most likely to promote?
>   Is this consistent with the stated LOs?
>
>   > <solution-title>for GTN tutorial, including the dummy one</solution-title>
>   >
>   > They are most likely to show how to do things and give an idea of how a tool
>   > work.
>   >
>   {: .solution}
>
> - Does the course or tutorial have any LO that can't be achieved through the
>   LEs that the course currently uses?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   > "Determine the best strategy to get data into Galaxy" may not be achieved
>   > with only practical
>   >
>   {: .solution}
>
> > <solution-title>LEs in GTN tutorials</solution-title>
> >
> > Most of the GTN tutorials are using Practical and Exercises as LEs.
> >
> > They are most likely to show how to do things and give an idea of how a tool works.
> >
> > In the [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %}),
> > the LO "Determine the best strategy to get data into Galaxy" may not be achieved
> > with only practical
> >
> {: .solution}
{: .question} 

<!--{% include _includes/quiz.html id="LEs_GTN_tutorials.yaml" %}-->

## Select and develop content relevant to the LOs

### Select content

With LOs and LEs aligned, Phase 3 involves finding the most appropriate
**content** to support learners to achieve the intended LOs. Regardless of where
content is drawn from, what matters is how it supports the LOs.

> <comment-title>Definitions of key terms</comment-title>
>
> **Content**: a specific subject or topic item (e.g., DNA, RNA, proteins, a
> biochemical pathway, R programming) that is the target of learning
>
{: .comment}

To this end, content selection should be judicious: it shouldn't try to be all-encompassing
({% cite nilson2016teaching %}), but should consider the target Bloom’s levels in
the LOs, and the preparation of learners. Once content that's considered to be
the core of a course has been identified, additional **auxiliary** materials can
also be selected, and offered, say, as 'further reading'. Using LOs to drive
content selection in this way thereby both provides focus (avoiding the
temptation to squeeze as much as possible into a course to ensure coverage
({% cite mckeachie2013mckeachie %}, {% cite lujan2006too %})), and increases the
likelihood of accomplishing those LOs.

Sometimes, specific content may be deemed essential ({% cite de2017curriculum %}).
Nevertheless, this must still support the LEs and promote achievement of the
LOs; if it doesn't, then additional content and/or LEs should be considered that
will, or the LOs themselves should be revised, to prevent misalignment of
instructional inputs and learning outcomes. Overall, the role of **LOs here is to
help focus on relevant content, and avoid material that's either
non-essential/too broad or too narrow**.

If content is, or seems, fixed, it can be difficult to make the shift from
content- to LO-driven decisions. In this case, Messick and Bloom's can be used to make
adjustments so that what it means to learn that content or what it looks like to have learned that
content can be made concrete and observable. If such fixed content does not lead to observable
behaviors that are, or support the achievement of, LOs, then additional content should be
considered that will; otherwise, it may be necessary to discuss the feasibility of modifying the
content or revising the LOs with those who are able to make changes, to avoid misalignment of
intended and actual curricula

<!-- >> <question-title>Evaluate content</question-title>
>
> - Think of the course or tutorial you selected before (*e.g.*, the
>   [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %}))
>
> - Review the LOs and identify which parts of the course content support the
>   LOs
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Map out the relationships between the LOs, the LEs via which they will be
>   delivered & those specific items of content (e.g., item A supports LO 1, &
>   will be delivered using a lecture)
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Is there any piece of content that doesn't support any LO(s)?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Is there at least one piece of content for each LO?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Is there at least one LE for each piece of content?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
{: .question}-->

Defining content can be complicated, specially to identify how it can cover the
different LOs. Trainers may need to tools to help them defining content,
*e.g.* concept maps.

### Concept maps

Concept maps are rough and visual scheme on how the different concepts are linked.
They begin with a main idea (or concept) and then branch out to show how
that main idea can be broken down into specific topics.

For example, the concepts around the nucleus could be represented using the
following concept map:

![Example of concept map to explain the nucleus. It start on the top with "Nucleus", two arrows leave from this. On the left "Nucleus -- has function -->" links to "has function --> performing chemical activities" and "has function --> cellular reproduction". On the right, "Nucleus -- consist of --> chromosomes", which splits into: on the left "chromosomes -- has function --> inheritance"; on the right "chromosomes -- composed of -->" with 2 arrows to the left "-- composed of --> DNA" which then split into "DNA -- composed of --> genes" and "DNA --> has function --> carrying of information", to the right "-- composed of --> proteins"](images/nucleus_concept_map.jpg "Example of concept map to explain the nucleus. 9 nodes and 6 edges, so 15 concepts in total")

***Why concept maps?*** Concept maps are graphical tools for organizing and
representing knowledge. They include concepts and the relationships between them.

***How to use concept maps?*** Concept maps can be used by trainers to help building
content, by breaking training in small pieces (7+/2 concepts) and by helping
trainers and helpers to connect things together. A good concept map usually
starts with a focused question, a context, and is built with iteration and
feedback.

For example, to answer the questions *"What are the FAIR principles?"*
1. we start by listing the keywords that should be discussed: Research data,
Research models, Findable, Accessible, Interoperable, Re-usable
2. we link them together in a concept map:

  ![It starts on the top with a box "What are FAIR principles?". An arrow goes from this box to a box below "Set of guiding principles". From the 2nd box, an arrow goes down with "to make" written and then splits toward 2 boxes: "Research data" on the left, "Research models" on the right. A line connects these 2 boxes with "and" written on it. An arrow starts from research data and splits into 4 arrows pointing to boxes (from left to right): "Findable", "Accessible", "Interoperable", "Re-usable"](images/fair_concept_map.png "Example of concept map to explain the FAIR principles")

Concept maps try to objective but they depend on their creators. Everyone has
their own concept map, like the way to go from point A to point B from 2 persons
living in the same city.

In curriculum/lesson/session planning, concept maps present key concepts in a
highly concise manner. They can take into account the existing knowledge of
learners and new concepts This helps in the teaching plan to measure how much
can be covered, and the hierarchical organization suggests a sequence to cover
material.

> <hands-on-title>Draw a concept map - {% icon time %} 15 min - Silent working</hands-on-title>
>
> - Draw a concept map of your topic of interest
> - Start with a question
> - Include 7 (+/- 2) concepts
> - Include relationships and cross-links between these concepts
> - Arrange it in a hierarchical structure with the key concepts on top
> - Count the items (concepts and links)
{: .hands_on}

> <hands-on-title>Give / receive feedback on concept maps - {% icon time %} 10 min - Group of 2 persons</hands-on-title>
>
> - Pair with someone
> - Check the concept name of your partner
> - Write to your partner
>   - One thing you are confuse/not sure about the map
>   - One thing you like/it is clear about the map
>
{: .hands_on}

> <tip-title>Further reading</tip-title>
>
> For more details about concept maps, we recommend you {% cite novak2008theory %}
> ([PDF](http://cmap.ihmc.us/publications/researchpapers/TheoryUnderlyingConceptMaps.pdf))
>
{: .tip }

### Content

- **Content collection**: collecting appropriate content to the needs and
  capabilities of your target audience
- **Content reduction**: one of the biggest challenges in designing training
  courses is the reduction of content to the training format. key points!

> <hands-on-title>Producing the content and the training material - {% icon time %} 15 min</hands-on-title>
>
> - Prepare the content for your 3 min training
>
>    The structure should be something like:
>
>    - 0:20 sec - Introduction
>    - 2:20 sec - Topic
>    - 0:20 sec - Conclusion
>
> - Use your concept map and adapt as needed
{: .hands_on}

> <details-title>From a 2-min presentation to a lesson/session: a plan</details-title>
>
> Template to help developing plan:
>
> > | Time | Activity | Description | Goal |
> > | ---- | -------- | ----------- | ---- |
> > |      |          |             |      |
> > |      |          |             |      |
> > |      |          |             |      |
> {: .matrix}
>
> For example: for a 1h15 session
>
> > Time | Activity | Description | Goal
> > --- | --- | --- | ---
> > 9:00 - 9:15 | Warm-up | Learners summaries the key points of each session from the previous day and answer questions from the audience. The instructor describes the plan of the day in detail  | Retrieval from memory, repetition, get prepared for new topics, expose learners
> > 9:15 - 9:25 | Lecture  | Python functions | Learning to write a function, about function input and output, and how to call a the function
> > 9:20 - 10:00 | Practical activity | Two exercises to be solved in pairs on a single computer. After solving the 1st exercise, the "driver" andr"navigator" will swap. 2 learners (1 / exercise) will display their solutions to the audience. Questions and discussion | Learners will be able to write and call a function calculating the distance between 2 points in the 3D  space and a function taking the base and height of a triangle as input and returning its area
> > 10:00 - 10:15 | Wrap-up | Group test on functions (match input and output with specific functions: fill gaps in pieces of code). Game: repetition using ball throwing | Assess learning. Do we need to work more on functions? Repeat meaning and usage of all Python objects introduced so far
> {: .matrix}
{: .details}

## Develop assessments to ensure progression

In any course, learners benefit from having opportunities to show that they're
progressing. Generally, this is done using various types of assessment,
including tests, feedback surveys, and so on. Nicholls' model includes two
different types of assessment, to evaluate:

1. **learning**, to detect changes in learner performance during instruction, to
   identify their strengths, and diagnose their weaknesses – i.e., **formative
   assessment** (or feedback). This can inform decisions about how to modify
   instruction to better promote learning; it can also inform learners about
   changes they may need to make to improve their learning;
2. **instructional outcomes**, to verify whether learners achieved the stated LOs
   after instruction – i.e., **summative assessment**. This can help to inform
   decisions both about learners (e.g., ranking their performances) and about
   the course (*i.e.*, whether any of its LOs, LEs, etc. need to be redesigned).

> <comment-title>Definitions of key terms</comment-title>
>
> **Formative assessment**: formal or informal assessments (or feedback) made
> during learning so that instruction or practice can be better targeted for
> lear


