# tutorials

topic: data-science
tutorial: r-dplyr
title: dplyr & tidyverse for data processing
difficulty: advanced
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["R"]
content_hash: 682245c2388e34995f7b98c649a7ddff
last_modified: 2026-05-15T22:28:11.115481
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-dplyr/tutorial.html

content:
dplyr ({% cite r-dplyr %}) is a powerful R-package to transform and summarize tabular data with rows and columns. It is part of a group of packages (including `ggplot2`) called the `tidyverse` ({% cite r-tidyverse %}), a collection of packages for data processing and visualisation. For further exploration please see the dplyr package vignette: [Introduction to dplyr](https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html)

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [GenomicsClass/labs](https://github.com/genomicsclass/labs/blob/master/intro/dplyr_tutorial.Rmd).
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Why Is It Useful?

The package contains a set of functions (or "verbs") that perform common data manipulation operations such as filtering for rows, selecting specific columns, re-ordering rows, adding new columns and summarizing data.

In addition, dplyr contains a useful function to perform another common task which is the "split-apply-combine" concept.  We will discuss that in a little bit.

## How Does It Compare To Using Base Functions R?

If you are familiar with R, you are probably familiar with base R functions such as split(), subset(), apply(), sapply(), lapply(), tapply() and aggregate(). Compared to base functions in R, the functions in dplyr are easier to work with, are more consistent in the syntax and are targeted for data analysis around tibbles, instead of just vectors.

## How Do I Get dplyr?

To load the required packages:

```r
library(tidyverse)
```

> <tip-title>Package not found?</tip-title>
> Remember that you can install new packages by running
> ```
> install.packages("tidyverse")
> ```
> Or by using the Install button on the RStudio Packages interface
{: .tip}

Here we've imported the entire suite of tidyverse packages. We'll specifically be using:

Package    | Use
---        | ---
`readr`    | This provides the `read_csv` function which is identical to `read.csv` except it returns a tibble
`dplyr`    | All of the useful functions we'll be covering are part of dplyr
`magrittr` | A dependency of `dplyr` that provides the `%>%` operator
`ggplot2`  | The famous plotting library which we'll use at the very end to plot our aggregated data.

# Data: Mammals Sleep

The msleep (mammals sleep) data set contains the sleep times and weights for a set of mammals. This data set contains 83 rows and 11 variables.

```r
url <- "https://raw.githubusercontent.com/genomicsclass/dagdata/master/inst/extdata/msleep_ggplot2.csv"
msleep <- read_csv(url)
head(msleep)
```

The columns (in order) correspond to the following:

column name  | Description
---          | ---
name         | common name
genus        | taxonomic rank
vore         | carnivore, omnivore or herbivore?
order        | taxonomic rank
conservation | the conservation status of the mammal
sleep\_total | total amount of sleep, in hours
sleep\_rem   | rem sleep, in hours
sleep\_cycle | length of sleep cycle, in hours
awake        | amount of time spent awake, in hours
brainwt      | brain weight in kilograms
bodywt       | body weight in kilograms

Compare the above output with the more traditional `read.csv` that is built into R

```r
dfmsleep <- read.csv(url)
head(dfmsleep)
```

This is a "data frame" and was the basis of data processing for years in R, and is still quite commonly used! But notice how `dplyr` has a much prettier and more consice output. This is what is called a `tibble` (like a table). We can immediately see metadata about the table, the separator that was guessed for us, what datatypes each column was (dbl or chr), how many rows and columns we have, etc. The `tibble` works basically exactly like a data frame except it has a lot of features to integrate nicely with the `dplyr` package.


*That said*, all of the functions below you will learn about work equally well with data frames and tibbles, but tibbles will save you from filling your screen with hundreds of rows by automatically truncating large tables unless you specifically request otherwise.

# Important dplyr Verbs To Remember

dplyr verbs | Description | SQL Equivalent Operation
--- | --- | ---
`select()` | select columns  | `SELECT`
`filter()` | filter rows | `WHERE`
`arrange()` | re-order or arrange rows | `ORDER BY`
`mutate()` | create new columns | `SELECT x, x*2 ...`
`summarise()` | summarise values | n/a
`group_by()` | allows for group operations in the "split-apply-combine" concept | `GROUP BY`


# dplyr Verbs In Action

The two most basic functions are `select()` and `filter()`, which selects columns and filters rows respectively.

## Pipe Operator: %>%

Before we go any further, let's introduce the pipe operator: %>%. dplyr imports this operator from another package (magrittr).This operator allows you to pipe the output from one function to the input of another function. Instead of nesting functions (reading from the inside to the outside), the idea of piping is to read the functions from left to right. This is a lot more like how you would write a `bash` data processing pipeline and can be a lot more readable and intuitive than the nested version.

Here's is the more old fashioned way of writing the equivalent code:

```r
head(select(msleep, name, sleep_total))
```

Now in this case, we will pipe the msleep tibble to the function that will select two columns (name and sleep\_total) and then pipe the new tibble to the function `head()`, which will return the head of the new tibble.

```r
msleep %>% select(name, sleep_total) %>% head(2)
```

> <question-title></question-title>
> How would you rewrite the following code to use the pipe operator?
> ```
> prcomp(tail(read.csv("file.csv"), 10))
> ```
> > <solution-title></solution-title>
> > Just read from inside to outside, starting with the innermost `()` and use `%>%` between each step.
> > ```
> > read.csv("file.csv") %>% tail(10) %>% prcomp()
> > ```
> {: .solution}
{: .question}

## Selecting Columns Using `select()`

Select a set of columns: the name and the sleep\_total columns.

```r
msleep %>% select(name, sleep_total)
```

To select all the columns *except* a specific column, use the "-" (subtraction) operator (also known as negative indexing):

```r
msleep %>% select(-name)
```

To select a range of columns by name, use the ":" (colon) operator:

```r
msleep %>% select(name:order)
```

To select all columns that start with the character string "sl", use the function `starts_with()`:

```r
msleep %>% select(starts_with("sl"))
```

Some additional options to select columns based on a specific criteria include:

Function      | Usage
--------      | -----
`ends_with()` | Select columns that end with a character string
`contains()`  | Select columns that contain a character string
`matches()`   | Select columns that match a regular expression
`one_of()`    | Select column names that are from a group of names


## Selecting Rows Using `filter()`

Filter the rows for mammals that sleep a total of more than 16 hours.

```r
msleep %>% filter(sleep_total >= 16)
```

Filter the rows for mammals that sleep a total of more than 16 hours *and* have a body weight of greater than 1 kilogram.

```r
msleep %>% filter(sleep_total >= 16, bodywt >= 1)
```

Filter the rows for mammals in the Perissodactyla and Primates taxonomic order

```r
msleep %>% filter(order %in% c("Perissodactyla", "Primates"))
```

You can use the boolean operators (e.g. >, <, >=, <=, !=, %in%) to create the logical tests.

## Arrange Or Re-order Rows Using `arrange()`

To arrange (or re-order) rows by a particular column, such as the taxonomic order, list the name of the column you want to arrange the rows by:

```r
msleep %>% arrange(order) %>% select(order, genus, name)
```

Now we will select three columns from msleep, arrange the rows by the taxonomic order and then arrange the rows by sleep\_total. Finally, show the final tibble:

```r
msleep %>%
    select(name, order, sleep_total) %>%
    arrange(order, sleep_total)
```

Same as above, except here we filter the rows for mammals that sleep for 16 or more hours, instead of showing the whole tibble:

```r
msleep %>%
    select(name, order, sleep_total) %>%
    arrange(order, sleep_total) %>%
    filter(sleep_total >= 16)
```

Something slightly more complicated: same as above, except arrange the rows in the sleep\_total column in a descending order. For this, use the function `desc()`

```r
msleep %>%
    select(name, order, sleep_total) %>%
    arrange(order, desc(sleep_total)) %>%
    filter(sleep_total >= 16)
```


## Create New Columns Using `mutate()`

The `mutate()` function will add new columns to the tibble. Create a new column called rem_proportion, which is the ratio of rem sleep to total amount of sleep.


```r
msleep %>%
  mutate(rem_proportion = sleep_rem / sleep_total) %>%
  select(starts_with("sl"), rem_proportion)
```

You can many new columns using mutate (separated by commas). Here we add a second column called bodywt_grams which is the bodywt column in grams.

```r
msleep %>%
    mutate(rem_proportion = sleep_rem / sleep_total,
           bodywt_grams = bodywt * 1000) %>%
    select(sleep_total, sleep_rem, rem_proportion, bodywt, bodywt_grams)
```

## Create summaries of the tibble using `summarise()`

The `summarise()` function will create summary statistics for a given column in the tibble such as finding the mean. For example, to compute the average number of hours of sleep, apply the `mean()` function to the column sleep\_total and call the summary value avg\_sleep.

```r
msleep %>%
    summarise(avg_sleep = mean(sleep_total))
```

There are many other summary statistics you could consider such `sd()`, `min()`, `max()`, `median()`, `sum()`, `n()` (returns the length of vector), `first()` (returns first value in vector), `last()` (returns last value in vector) and `n_distinct()` (number of distinct values in vector).

```r
msleep %>%
    summarise(avg_sleep = mean(sleep_total),
              min_sleep = min(sleep_total),
              max_sleep = max(sleep_total),
              total = n())
```


## Group operations using `group_by()`

The `group_by()` verb is an important function in dplyr. As we mentioned before it's related to concept of "split-apply-combine". We literally want to split the tibble by some variable (e.g. taxonomic order), apply a function to the individual tibbles and then combine the output.

Let's do that: split the msleep tibble by the taxonomic order, then ask for the same summary statistics as above. We expect a set of summary statistics for each taxonomic order.

```r
msleep %>%
    group_by(order) %>%
    summarise(avg_sleep = mean(sleep_total),
              min_sleep = min(sleep_total),
              max_sleep = max(sleep_total),
              total = n())
```

## ggplot2

Most people want to slice and dice their data before plotting, so let's demonstrate that quickly by plotting our last dataset.

```r
library(ggplot2)
msleep %>%
    group_by(order) %>%
    summarise(avg_sleep = mean(sleep_total),
              min_sleep = min(sleep_total),
              max_sleep = max(sleep_total),
              total = n()) %>%
    ggplot() + geom_point(aes(x=min_sleep, y=max_sleep, colour=order))
```

Notice how we can just keep piping our data together, this makes it incredibly easier to experiment and play around with our data and test out what filtering or summarisation we want and how that will plot in the end. If we wanted, or if the data processing is an especially computationally expensive step, we could save it to an intermediate variable before playing around with plotting options, but in the case of this small dataset that's probably not necessary.


topic: data-science
tutorial: python-argparse
title: Python - Argparse
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 6555e5fb9d113dceed297aa1d9ea4fd5
last_modified: 2026-05-15T22:28:11.100483
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-argparse/tutorial.html

content:
[`argparse`](https://docs.python.org/3/library/argparse.html) is an argument parsing library for Python that's part of the stdlib. It lets you make command line tools significantly nicer to work with.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Unlike previous modules, this lesson won't use a Jupyter/CoCalc notebook, and that's because we'll be parsing command lines! You'll need to open a code editor on your platform of choice (`nano`, `vim`, `emacs`, VSCode are all options) and use the following blocks of code to construct your command line tool.

## `sys.argv`

In the coding world, whenever you run a Python script on the command line, it has a special variable available to it named `argv`. This is a list of all of the arguments used when you run a command line program.

> <hands-on-title>Print out argv</hands-on-title>
>
> 1. Create / open the file `run.py` in your text editor of choice
> 2. There we'll create a simple Python script that:
>   1. imports `sys`, the system module needed to access argv.
>   2. Prints out `sys.argv`
>
> > <solution-title></solution-title>
> > ```python
> > import sys
> >
> > print(sys.argv)
> > ```
> {: .solution}
>
> 3. Run this with different command line arguments:
>
>    ```
>    python run.py
>    python run.py 1 2 3 4
>    python run.py --help
>    ```
>
{: .hands_on}

> <question-title></question-title>
> What did you notice about the output? There are two main points.
> > <solution-title></solution-title>
> > 1. The name of the script (`run.py`) is included as the first value every time.
> > 2. All of the arguments are passed as strings, no numbers.
> {: .solution}
{: .question}

## Simple tasks

Let's sum up all of the numbers passed on the command line. We'll do this by hand, and then we'll replace it with `argparse` to see how much effort that saves us.

> <hands-on-title></hands-on-title>
> Update your script to sum up every number passed to it on the command line.
>
> It should handle:
> - 1 or more numbers
> - nothing (and maybe print out a message?)
> - invalid values (print out an error message that the value couldn't be processed.)
>
> Hints:
> - Skip the program name
> - Use `try` and `except` to try converting the string to a number.
>
> > <question-title></question-title>
> >
> > How does your updated script look?
> >
> > > <solution-title></solution-title>
> > >
> > > ```python
> > > import sys
> > >
> > > result = 0
> > >
> > > if len(sys.argv) == 1:
> > >     print("no arguments were supplied")
> > > else:
> > >     for arg in sys.argv[1:]:
> > >         try:
> > >             result += float(arg)
> > >         except:
> > >             print(f"Could not parse {arg}")
> > >
> > >     print(result)
> > > ```
> > {: .solution}
> {: .question}
{: .hands_on}

## Argparse

Argparse saves us a lot of work, because it can handle a number of things for us!

- Ensures that the correct number of arguments are provided (and provide a nice error message otherwise)
- Ensure that the correct types of arguments are provided (no strings for a number field)
- Provide a help message describing your program

Argparse is used as follows. First we need to import it

```python
import argparse
```

And then we can define a 'parser' which will parse our command line. Additionally we can provide a description field which tells people what our tool does:

```python
parser = argparse.ArgumentParser(description='Process some integers.')
```

And finally we can define some arguments that are available. Just like we have arguments to functions, we have arguments to command lines. These come in two flavours:

- required (without a `--`)
- optional "flags" (prefixed with `--`)

Here we have an argument named 'integers', which validates that all input values are of the type `int`. `nargs` is the number of arguments, `+` means '1 or more'. And we have some help text as well:

```python
parser.add_argument('integer', type=int, help='an integer parameter')
parser.add_argument('many_integers', type=int, nargs='+', help='an integer parameter')
```

We can also define an optional flag, here it's called `--sum`. We use `store_true` which will set it as true if the flag is used , otherwise false.

```python
parser.add_argument('--sum', action='store_true', help='Should we sum up the integers?')
```

Finally we parse the arguments, which reads `sys.argv` and processes it according to the above rules. The output is stored in `args`.

```python
args = parser.parse_args()
```

We have two main variables we can use now:

```
args.integer # A single integer
args.many_integers # A list of ints
args.sum # A boolean, True or False.
```

## Using argparse

Let's go back to our script, and replace `sys` with argparse.

> <hands-on-title>Replacing argv.</hands-on-title>
>
> 1. Given the following script, replace the use of `argv` with argparse.
>
>    ```python
>    import sys
>
>    result = 0
>
>    if len(sys.argv) == 1:
>        print("no arguments were supplied")
>    else:
>        for arg in sys.argv[1:]:
>            try:
>                result += float(arg)
>            except:
>                print(f"Could not parse {arg}")
>
>        print(result)
>    ```
>    {: .hands_on}
>
>    You should have one argument: numbers (type=float)
>
>    And print out the sum of those numbers.
>
>    > <question-title></question-title>
>    > How does your final script look?
>    > > <solution-title></solution-title>
>    > > ```python
>    > > import argparse
>    > >
>    > > parser = argparse.ArgumentParser(description='Sum some numbers')
>    > > parser.add_argument('integers', type=float, nargs='+',
>    > >                     help='a number to sum up.')
>    > > args = parser.parse_args()
>    > >
>    > > print(sum(args.integers))
>    > > ```
>    > {: .solution}
>    {: .question}
>
> 2. Try running the script with various values
>
>    ```bash
>    python run.py
>    python run.py 1 3 5
>    python run.py 2 4 O
>    python run.py --help
>    ```
{: .hands_on}

Wow that's a lot simpler! We have to learn how `argparse` is invoked but it handles a lot of cases for us:

- No arguments provided
- Responding to `--help`
- Raising an error for invalid values

`--help` is even written for us, without us writing any special code to handle that case! This is why you need to use `argparse`:
- It handles a lot of cases and input validation for you
- It produces a nice `--help` text that can help you if you've forgotten what your tool does
- It's nice for users of your scripts! They don't have to read the code to know how it behaves if you document it well.

There is a lot of documentation in the [`argparse`](https://docs.python.org/3/library/argparse.html) module for all sorts of use cases!


## Why Argparse?

Using argparse can be a big change to your tool but there are some benefits to using it!

1. Standardised interface to your tool that's familiar to everyone who uses command line tools
2. Automatic Help page
3. Automatic Galaxy Tools?

### Generating Automatic Galaxy Tools (Optional)

With the `argparse2tool` project, and eventually `pyGalGen` which will be merged into `planemo`, you can generate Galaxy tools automatically from `argparse` based Python scripts.

> <hands-on-title>Generate a Galaxy tool wrapper from your script</hands-on-title>
> 1. Write out the python script to a file named `main.py`
>
>    ```python
>    import argparse
>
>    parser = argparse.ArgumentParser(description='Sum some numbers')
>    parser.add_argument('integers', type=float, nargs='+',
>                        help='a number to sum up.')
>    args = parser.parse_args()
>
>    print(sum(args.integers))
>    ```
>
> 2. Create a virtual environment, just in case: ``
>
>    ```console
>    python -m venv .venv
>    . .venv/bin/activate
>    ```
>
> 3. Install `argparse2tool` via pip:
>
>    ```console
>    pip install argparse2tool
>    ```
>
> 4. Generate the tool interface:
>
>    > <code-in-title>Command</code-in-title>
>    > ```
>    > PYTHONPATH=$(argparse2tool) python main.py --generate_galaxy_xml
>    > ```
>    {: .code-in }
>
>    > <code-out-title>Galaxy XML</code-out-title>
>    > ```xml
>    > <tool name="main.py" id="main.py" version="1.0">
>    >   <description>Sum some numbers</description>
>    >   <stdio>
>    >     <exit_code range="1:" level="fatal"/>
>    >   </stdio>
>    >   <version_command><![CDATA[python main.py --version]]></version_command>
>    >   <command><![CDATA[python main.py
>    > #set repeat_var_1 = '" "'.join([ str($var.integers) for $var in $repeat_1 ])
>    > "$repeat_var_1"
>    >
>    > > $default]]></command>
>    >   <inputs>
>    >     <repeat title="repeat_title" min="1" name="repeat_1">
>    >       <param label="a number to sum up." value="0" type="float" name="integers"/>
>    >     </repeat>
>    >   </inputs>
>    >   <outputs>
>    >     <data name="default" format="txt" hidden="false"/>
>    >   </outputs>
>    >   <help><![CDATA[TODO: Write help]]></help>
>    > </tool>
>    >
>    > ```
>    {: .code-out }
>
{: .hands_on}


topic: data-science
tutorial: python-linting
title: Python - Coding Style
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 6adc9c07919dd619c0cb343b78be14f7
last_modified: 2026-05-15T22:28:11.103728
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-linting/tutorial.html

content:
But before you dive into writing some more code and
sharing it with others, ask yourself what kind of code should you be writing and publishing? It may be
worth spending some time learning a bit about Python
coding style conventions to make sure that your code is consistently formatted and readable by yourself and others.

> *"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."* - [Martin Fowler](https://en.wikiquote.org/wiki/Martin_Fowler), British software engineer, author and international speaker on software development
{: .quote}

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Python Coding Style Guide

One of the most important things we can do to make sure our code is readable by others
(and ourselves a
few months down the line) is to make sure that it is descriptive, cleanly and consistently formatted and uses sensible,
descriptive names for variable, function and module names. In order to help us format our code, we generally follow
guidelines known as a **style guide**. A style guide is a set of conventions that we agree upon with our colleagues or
community, to ensure that everyone contributing to the same project is producing code which looks similar in style.
While a group of developers may choose to write and agree upon a new style guide unique to each project,
in practice many programming languages have a single style guide which is
adopted almost universally by the communities around the world. In Python, although we do have a choice of style guides
available, the [PEP8](https://www.python.org/dev/peps/pep-0008/) style guide is most commonly used.
A {PEP} is a design document for the Python community, typically
specifications or conventions for how to do something in Python, a description of a new feature in Python, etc.

> <tip-title>Style consistency</tip-title>
> One of the
> [key insights from Guido van Rossum](https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds),
> one of the PEP8 authors, is that code is read much more often than it is
> written. Style guidelines are intended to improve the readability of code and make it consistent across the
> wide spectrum of Python code. Consistency with the style guide is important. Consistency within a project is more
> important. Consistency within one module or function is the most important. However, know when to be inconsistent --
> sometimes style guide recommendations are just not applicable. When in doubt, use your best judgment.
> Look at other examples and decide what looks best. And don't hesitate to ask!
>
{: .tip}

A full list of style guidelines for this style 
is available from the [PEP8 website](https://www.python.org/dev/peps/pep-0008/); here we highlight a few.

### Indentation

Python is a kind of language that uses indentation as a way of grouping statements that belong to a particular
block of code. Spaces are the recommended indentation method in Python code. The guideline is to use 4 spaces per indentation level -
so 4 spaces on level one, 8 spaces on level two and so on.

Many people prefer the use of tabs to spaces to indent the code for many reasons (e.g. additional typing, easy to
introduce an error by missing a single space character, etc.) and do not follow this guideline. Whether you decide to
follow this guideline or not, be consistent and follow the style already used in the project.

> <tip-title>Accessibility</tip-title>
> For many users of Braille Refreshable Displays, encoding the indentation with a tab character is easier to read and more efficient. Rather than 8 space characters, they would only see 2 tab characters which take up less space and are easier to comprehend the indentation level.
{: .tip}

> <tip-title>Indentation in Python 2 vs Python 3</tip-title>
> Python 2 allowed code
> indented with a mixture of tabs and spaces. Python 3 disallows mixing the use of tabs and spaces for indentation.
> Whichever you choose, be consistent throughout the project.
{: .tip}

> <tip-title>Editor support</tip-title>
> Many IDEs and editors have built in support for automatically managing tabs and spaces and auto-converting them back and forth.
{: .tip}

There are more complex rules on indenting single units of code that continue over several lines, e.g. function,
list or dictionary definitions can all take more than one line. The preferred way of wrapping such long lines is by
using Python's implied line continuation inside delimiters such as parentheses (`()`), brackets (`[]`) and braces
(`{}`), or a hanging indent.

```python
# Add an extra level of indentation (extra 4 spaces) to distinguish arguments from the rest of the code that follows
def long_function_name(
        var_one, var_two, var_three,
        var_four):
    print(var_one)


# Aligned with opening delimiter
foo = long_function_name(var_one, var_two,
                         var_three, var_four)

# Use hanging indents to add an indentation level like paragraphs of text where all the lines in a paragraph are
# indented except the first one
foo = long_function_name(
    var_one, var_two,
    var_three, var_four)

# Using hanging indent again, but closing bracket aligned with the first non-blank character of the previous line
a_long_list = [
    [[1, 2, 3], [4, 5, 6], [7, 8, 9]], [[0.33, 0.66, 1], [0.66, 0.83, 1], [0.77, 0.88, 1]]
    ]

# Using hanging indent again, but closing bracket aligned with the start of the multiline contruct
a_long_list2 = [
    1,
    2,
    3,
    # ...
    79
]
```

More details on good and bad practices for continuation lines can be found in
[PEP8 guideline on indentation](https://www.python.org/dev/peps/pep-0008/#indentation).

### Maximum Line Length

All lines should be up to 80 characters long; for lines containing comments or docstrings (to be covered later) the
line length limit should be 73 - see [this discussion](https://www.google.com/url?q=https://stackoverflow.com/questions/15438326/python-pep-8-docstring-line-length&sa=D&source=editors&ust=1619088968027000&usg=AOvVaw3jn26Qt-kwog_tJnaMR48x) for reasoning behind these numbers. Some teams strongly prefer a longer line length, and seemed to have settled on the
length of 100. Long lines of code can be broken over multiple lines by wrapping expressions in delimiters, as
mentioned above (preferred method), or using a backslash (`\`) at the end of the line to indicate
line continuation (slightly less preferred method).

```python
# Using delimiters ( ) to wrap a multi-line expression
if (a == True and
    b == False):

# Using a backslash (\) for line continuation
if a == True and \
    b == False:
```

### Should a Line Break Before or After a Binary Operator?

Lines should break before binary operators so that the operators do not get scattered across different columns
on the screen. In the example below, the eye does not have to do the extra work to tell which items are added
and which are subtracted:

```python
# PEP 8 compliant - easy to match operators with operands
income = (gross_wages
          + taxable_interest
          + (dividends - qualified_dividends)
          - ira_deduction
          - student_loan_interest)
```

### Blank Lines

Top-level function and class definitions should be surrounded with two blank lines. Method definitions inside a class
should be surrounded by a single blank line. You can use blank lines in functions, sparingly, to indicate logical sections.

### Whitespace in Expressions and Statements

Avoid extraneous whitespace in the following situations:

- Immediately inside parentheses, brackets or braces

  ```python
  # PEP 8 compliant:
  my_function(colour[1], {id: 2})

  # Not PEP 8 compliant:
  my_function( colour[ 1 ], { id: 2 } )
  ```

- Immediately before a comma, semicolon, or colon (unless doing slicing where the colon acts like a binary operator
in which case it should should have equal amounts of whitespace on either side)

  ```python
  # PEP 8 compliant:
  if x == 4: print(x, y); x, y = y, x

  # Not PEP 8 compliant:
  if x == 4 : print(x , y); x , y = y, x
  ```

- Immediately before the open parenthesis that starts the argument list of a function call

  ```python
  # PEP 8 compliant:
  my_function(1)

  # Not PEP 8 compliant:
  my_function (1)
  ```

- Immediately before the open parenthesis that starts an indexing or slicing

  ```python
  # PEP 8 compliant:
  my_dct['key'] = my_lst[id]
  first_char = my_str[:, 1]

  # Not PEP 8 compliant:
  my_dct ['key'] = my_lst [id]
  first_char = my_str [:, 1]
  ```

- More than one space around an assignment (or other) operator to align it with another

  ```python
  # PEP 8 compliant:
  x = 1
  y = 2
  student_loan_interest = 3

  # Not PEP 8 compliant:
  x                     = 1
  y                     = 2
  student_loan_interest = 3
  ```

- Avoid trailing whitespace anywhere - it is not necessary and can cause errors. For example, if you use
backslash (`\`) for continuation lines and have a space after it, the continuation line will not be
interpreted correctly.
- Surround these binary operators with a single space on either side: assignment (=),
augmented assignment (+=, -= etc.), comparisons (==, <, >, !=, <>, <=, >=, in, not in, is, is not),
booleans (and, or, not).
- Don't use spaces around the = sign when used to indicate a keyword argument assignment or to indicate a
default value for an unannotated function parameter

  ```python
  # PEP 8 compliant use of spaces around = for variable assignment
  axis = 'x'
  angle = 90
  size = 450
  name = 'my_graph'

  # PEP 8 compliant use of no spaces around = for keyword argument assignment in a function call
  my_function(
      1,
      2,
      axis=axis,
      angle=angle,
      size=size,
      name=name)
  ```

### String Quotes

In Python, single-quoted strings and double-quoted strings are the same. PEP8 does not make a recommendation for this
apart from picking one rule and consistently sticking to it. When a string contains single or double quote characters,
use the other one to avoid backslashes in the string as it improves readability.

### Naming Conventions

There are a lot of different naming styles in use, including:
- lowercase
- lower_case_with_underscores
- UPPERCASE
- UPPER_CASE_WITH_UNDERSCORES
- CapitalisedWords (or PascalCase) (note: when using acronyms in CapitalisedWords, capitalise all the letters of the acronym,
e.g HTTPServerError)
- camelCase (differs from CapitalisedWords/PascalCase by the initial lowercase character)
- Capitalised_Words_With_Underscores

As with other style guide recommendations - consistency is key. Pick one and stick to it, or follow the one already
established if joining a project mid-way. Some things to be wary of when naming things in the code:

- Avoid using the characters 'l' (lowercase letter L), 'O' (uppercase letter o), or 'I' (uppercase letter i)
  as single character variable names. In some fonts, these characters are indistinguishable from the numerals
  one and zero. When tempted to use 'l', use 'L' instead.
- If your audience is international and English is the common language, try to use English words for identifiers and 
  comments whenever possible but try to avoid abbreviations/local slang as they may not be understood by everyone. Also consider
  sticking with either ‘American’ or 'British' English spellings and try not to mix the two.

Stronlgy consider **not** using single character identifiers wherever possible. Use descriptive variable names when possible, it is no slower to execute, and a lot faster to read and comprehend for a reader.

> <tip-title>Function, Variable, Class, Module, Package Naming</tip-title>
>
> - Function and variable names should be lowercase, with words separated by underscores as necessary to improve readability.
> - Class names should normally use the CapitalisedWords convention.
> - Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability.
> - Packages should also have short, all-lowercase names, although the use of underscores is discouraged.
>
> A more detailed guide on
> [naming functions, modules, classes and variables](https://www.python.org/dev/peps/pep-0008/#package-and-module-names)
> is available from PEP8.
>
{: .tip}

### Comments

Comments allow us to provide the reader with additional information on what the code does - reading and understanding
source code is slow, laborious and can lead to misinterpretation, plus it is always a good idea to keep others in mind
when writing code. A good rule of thumb is to assume that someone will *always* read your code at a later date,
and this includes a future version of yourself. It can be easy to forget why you did something a particular way in six
months' time. Write comments as complete sentences and in English unless you are 100% sure the code will never be read
by people who don't speak your language.

> <tip-title>The Good, the Bad, and the Ugly Comments</tip-title>
> As a side reading, check out the ['Putting comments in code: the good, the bad, and the ugly' blogpost](https://www.freecodecamp.org/news/code-comments-the-good-the-bad-and-the-ugly-be9cc65fbf83/).
> Remember - a comment should answer the 'why' question. Occasionally the "what" question.
> The "how" question should be answered by the code itself.
{: .tip}

Block comments generally apply to some (or all) code that follows them, and are indented to the same level as that
code. Each line of a block comment starts with a `#` and a single space (unless it is indented text inside the comment).

```python
def fahr_to_cels(fahr):
    # Block comment example: convert temperature in Fahrenheit to Celsius
    cels = (fahr + 32) * (5 / 9)
    return cels
```

An inline comment is a comment on the same line as a statement. Inline comments should be separated by at least two
spaces from the statement. They should start with a `#` and a single space and should be used sparingly.

```python
def fahr_to_cels(fahr):
    cels = (fahr + 32) * (5 / 9)  # Inline comment example: convert temperature in Fahrenheit to Celsius
    return cels
```

Python doesn't have any multi-line comments, like you may have seen in other languages like C++ or Java. However, there
 are ways to do it using *docstrings* as we'll see in a moment.

The reader should be able to understand a single function or method from its code and its comments, and should not have to look elsewhere in the code for clarification. The kind of things that need to be commented are:

- Why certain design or implementation decisions were adopted, especially in cases where the decision may seem counter-intuitive
- The names of any algorithms or design patterns that have been implemented
- The expected format of input files or database schemas

However, there are some restrictions. Comments that simply restate what the code does are redundant, and comments must be
 accurate and updated with the code, because an incorrect comment causes more confusion than no comment at all.


## Exercise Improving Indentation

Now given all of the above rules we'll go ahread and re-format some code to match the style guide. Please review them carefully and prepare yourself for an hour of editing text… no, of course not. 

There is no reason to do this manually, we're learning about computers and reformatting text is something computers are great at! There is a project called [Black](https://pypi.org/project/black/) which automates this process, and removes the potential arguing over coding style, which is truly it's greatest success. 

> <hands-on-title>Reformatting Code with Black</hands-on-title>
> 1. Install black
>    
>    ```bash
>    pip3 install black
>    ```
>    
> 2. Run Black
>  
>    ```bash
>    black .
>    ```
>  
> 3. Done. That's it.
{: .hands_on}

> <hands-on-title>Optional Exercise: Improve Code Style of Your Other Python Projects</hands-on-title>
> If you have another Python project, check to which extent it conforms to PEP8 coding style.
{: .hands_on}

## Documentation Strings aka Docstrings

If the first thing in a function is a string that is not assigned to a variable, that string is attached to the
function as its documentation. Consider the following code implementing function for calculating the nth
Fibonacci number:

```python
def fibonacci(n):
    """Calculate the nth Fibonacci number.

    A recursive implementation of Fibonacci array elements.

    :param n: integer
    :raises ValueError: raised if n is less than zero
    :returns: Fibonacci number
    """
    if n < 0:
        raise ValueError('Fibonacci is not defined for N < 0')
    if n == 0:
        return 0
    if n == 1:
        return 1

    return fibonacci(n - 1) + fibonacci(n - 2)
```

Note here we are explicitly documenting our input variables, what is returned by the function, and also when the
`ValueError` exception is raised. Along with a helpful description of what the function does, this information can
act as a *contract* for readers to understand what to expect in terms of behaviour when using the function,
as well as how to use it.

A special comment string like this is called a **docstring**. We do not need to use triple quotes when writing one, but
if we do, we can break the text across multiple lines. Docstrings can also be used at the start of a Python module (a file
containing a number of Python functions) or at the start of a Python class (containing a number of methods) to list
their contents as a reference. You should not confuse docstrings with comments though - docstrings are context-dependent and should only
be used in specific locations (e.g. at the top of a module and immediately after `class` and `def` keywords as mentioned).
Using triple quoted strings in locations where they will not be interpreted as docstrings or
using triple quotes as a way to 'quickly' comment out an entire block of code is considered bad practice.

In our example case, we used
the [Sphynx/ReadTheDocs docstring style](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html) formatting
for the `param`, `raises` and `returns` - other docstring formats exist as well.

> <tip-title>Python PEP 257 - Recommendations for Docstrings</tip-title>
> PEP 257 is another one of Python Enhancement Proposals and this one deals with docstring conventions to
>standardise how they are used. For example, on the subject of module-level docstrings, PEP 257 says:
>
> > The docstring for a module should generally list the classes, exceptions and functions (and any other objects) that
> > are exported by the module, with a one-line summary of each. (These summaries generally give less detail than the
> > summary line in the object's docstring.) The docstring for a package
> > (i.e., the docstring of the package's `__init__.py` module) should also list the modules and subpackages exported by
> > the package.
> {: .quote}
>
> Note that `__init__.py` file used to be a required part of a package (pre Python 3.3) where a package was typically
>implemented as a directory containing an `__init__.py` file which got implicitly executed when a package was imported.
{: .tip}

So, at the beginning of a module file we can just add a docstring explaining the nature of a module. For example, if
`fibonacci()` was included in a module with other functions, our module could have at the start of it:

```python
"""A module for generating numerical sequences of numbers that occur in nature.

Functions:
  fibonacci - returns the Fibonacci number for a given integer
  golden_ratio - returns the golden ratio number to a given Fibonacci iteration
  ...
"""
...
```

The docstring for a function or a module is returned when
calling the `help` function and passing its name - for example from the interactive Python console/terminal available
from the command line or when rendering code documentation online 
(e.g. see [Python documentation](https://docs.python.org/3.8/library/index.html)).
PyCharm also displays the docstring for a function/module in a little help popup window when using tab-completion.

```python
help(fibonacci)
```


topic: data-science
tutorial: python-flow
title: Python - Flow Control
difficulty: introductory
time_estimation: 40M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 5827d030ca4903c65dc2861dd1c00f23
last_modified: 2026-05-15T22:28:11.102512
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-flow/tutorial.html

content:
"Flow Control" is how we describe when we change the flow of code's execution, based on some conditions. Here we'll learn how to take different actions depending on what data out program sees, or how to run code only if some condition is true.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which are licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Comparators

In Python we have the following comparators to do compare two values

- `>`: greater than
- `<`: less than
- `==`: equal to
- `!=`: does not equal
- `>=`: greater than or equal to
- `<=`: less than or equal to

They're all "binary" comparators, we can only compare two values at a time.

```python
print(37 < 38)
print(38 < 38)
print(39 < 38)
```

These print out `True` or `False`, these are the two possible values of the [*boolean*](https://en.wikipedia.org/wiki/Boolean_algebra) datatype in Python.

We can use `<=` to check if it's less than or equal to:

```python
print(19 <= 20)
print(20 <= 20)
print(21 <= 20)
```

And we can use `==` for comparing numbers in Python

```
print(11 == 11)
print(11 != 11)
print(22 != 33)
```

And now that we can compare numbers, we can start doing useful things with them!

## Conditionals

We can ask Python to take different actions, depending on a condition, with an `if` statement:

```python
num = 37
if num > 100:
    print('greater')
else:
    print('not greater')
print('done')
```

The second line of this code uses the keyword `if` to tell Python that we want to make a choice.
If the test that follows the `if` statement is true,
the body of the `if`
(i.e., the set of lines indented underneath it) is executed, and "greater" is printed.
If the test is false,
the body of the `else` is executed instead, and "not greater" is printed.
Only one or the other is ever executed before continuing on with program execution to print "done":

![A flowchart diagram of the if-else construct that tests if variable num is greater than 100](../../images/python-flowchart-conditional.png)

Conditional statements don't have to include an `else`. If there isn't one,
Python simply does nothing if the test is false:

```python
num = 53
print('before conditional...')
if num > 100:
    print(f'{num} is greater than 100')
print('...after conditional')
```

> <question-title>If behaviour</question-title>
>
> Try changing the `num` value and see what happens for different values.
>
> What happens if `num` is a:
> 1. 202
> 2. 3.145
> 3. "test"
> 4. 100.000001
>
> > <solution-title></solution-title>
> > 1. Condition is activated!
> > 2. Nothing, but not because it is a float! Because it's less than 100
> > 3. Traceback, a `TypeError`, you cannot compare strings with integers
> > 4. Condition is activated!
> {: .solution}
{: .question}

## Multiple Branches

But what if you want more branches? What if you need to handle more cases? `elif` to the rescue!

We can chain several tests together using `elif`, which is short for "else if".

```
if todays_temperature > 30:
    print("Wear shorts! Remember your sunscreen")
elif todays_temperature > 20:
    print("It's nice weather finally! Gasp!")
elif todays_temperature < 10:
    print("Time to bundle up!")
else:
    print("Dress normally")
```

> <tip-title>If/Elif/Elif/Elif/Else:</tip-title>
> if/elif/else cases follow these rules:
>
> - must start with an `if`
> - can have 0 or more `elif` conditions
> - can have 0 or 1 `else` condition (if no else condition is supplied, it's equivalent to `else: <nothing>`)
{: .tip}

Each of these three sections is a **branch**, the code pauses, and chooses to go down one of the branches based on the conditions.

The following Python code uses `elif` to print the sign of a number.

```python
num = -3

if num > 0:
    print(f'{num} is positive')
elif num == 0:
    print(f'{num} is zero')
else:
    print(f'{num} is negative')
```

**NB**: To test for equality we use a double equals sign `==`
rather than a single equals sign `=` which is used to assign values.

## Combining Tests

We can also combine tests using `and` and `or`.
`and` is only true if both parts are true:

```python
a = 1
b = -1
if (a > 0) and (b <= 0):
    print('both parts are true')
else:
    print('at least one part is false')
```

> <question-title>Predict what happens</question-title>
> Predict the outcomes of the following values of `a` and `b` above. Predicting what you think the code will do is a useful skill to practice
> 1. a = 0; b = -1
> 2. a = 0; b = 10
> 3. a = 4; b = -22
> 4. a = 99; b = 99
>
> > <solution-title></solution-title>
> > 1. at least one part is false
> > 2. at least one part is false
> > 3. both parts are true
> > 4. at least one part is false
> {: .solution}
{: .question}


while `or` is true if at least one part is true:

```python
a = 1
b = -1
if (a < 0) or (b > 0):
    print('at least one test is true')
```


> <tip-title>`True` and `False`</tip-title>
> `True` and `False` are special words in Python called `booleans`,
> which represent truth values. A statement such as `1 < 0` returns
> the value `False`, while `-1 < 0` returns the value `True`.
{: .tip}

`True` and `False` booleans are not the only values in Python that are true and false.
In fact, *any* value can be used in an `if` or `elif`. This is commonly used to
check, for instance, if a string is empty or if some data is provided:

```python
if '':
    print('empty string is true')
if 'word':
    print('word is true')
```

You can also use it to check if a list is empty or full:

```python
if []:
    print('empty list is true')
if [1, 2, 3]:
    print('non-empty list is true')
# The last statement is equivalent to:
if len([1, 2, 3]) > 0:
    print('non-empty list is true')
```

Or you can check if a number is zero, or non-zero:

```python
if 0:
    print('zero is true')
if 1:
    print('one is true')
```

## Inverting Conditions

Sometimes it is useful to check whether some condition is not true.
The Boolean operator `not` can do this explicitly.
After reading and running the code below,
write some `if` statements that use `not` to test the rule
that you formulated in the previous question.
`not` is a `unary` (not `binary`) operator: it only takes a single value

```python
if not '':
    print('empty string is not true')
if not 'word':
    print('word is not true')
if not not True:
    print('not not True is true')
```

## Ranges

Python makes it super easy to check if a number is within a range.

```python
quality_score = 32 # Try out different values!

if quality_score > 40:
    print("Your data is a bit sus")
elif 20 < quality_score <= 40:
    print("Hey that looks ok")
elif 4 < quality_score <= 20:
    print("Oh you did nanopore sequencing")
else:
    print("It shouldn't be *that* bad. Try again.")
```

There are two important points here:

- `20 < x < 40` is equivalent to `20 < x and x < 40`, checking both sides of the condition, to make sure it's greater than one value and smaller than another
- Note that we checked in the second case `20 < x` and then in the third we had to check `x <= 20`. If we had not had a `<=` on one side, what would have happened to 20? It would have gone straight to else!


## Exercises

> <question-title></question-title>
> `if`s, `elif`s and `else`s get evaluated in blocks. Look at the following code and list the lines that are part of a single block.
>
> ```
> 1.  if x:
> 2.      # ..
> 3.  if y:
> 4.      # ..
> 5.  elif z:
> 6.      # ..
> 7.  if q:
> 8.      # ..
> 9.  else:
> 10.     # ..
> 11. elif t:
> 12.     # ..
> 13. else e:
> 14.     # ..
> ```
>
> > <solution-title></solution-title>
> > "Blocks" of if/elif/elses
> > - must start with an `if`
> > - can have 0 or more `elif` conditions
> > - can have 0 or 1 `else` condition (if no else condition is supplied, it's equivalent to `else: <nothing>`)
> >
> > The above blocks are parsed together, you could not insert a `print` anywhere within the blocks, but between the blocks it would work.
> >
> > - 1-2, Just an `if` by itself. There's no elif, or else, so that's the end of that block
> > - 3-6, `if` and `elif` get evaluated, there is no `else`, so that's the end of that block
> > - 7-10, `if` and `else` is fine
> > - 11-14, error! This is missing an `if` case, it will fail with a syntaxerror.
> {: .solution}
{: .question}

```python
# Test code here.
```

> <question-title>How Many Paths?</question-title>
>
> Consider this code:
>
> ```
> if 4 > 5:
>     print('A')
> elif 4 == 5:
>     print('B')
> elif 4 < 5:
>     print('C')
> ```
>
> Which of the following would be printed if you were to run this code?
> Why did you pick this answer?
>
> 1.  A
> 2.  B
> 3.  C
> 4.  B and C
>
> > <solution-title></solution-title>
> > C gets printed because the first two conditions, `4 > 5` and `4 == 5`, are not true,
> > but `4 < 5` is true.
> {: .solution}
{: .question}

```python
# Test code here.
```

> <question-title>Close Enough</question-title>
>
> Write some conditions that print `True` if the variable `a` is within `10` of the variable `b`
> and `False` otherwise.
> Compare your implementation with your partner's:
> do you get the same answer for all possible pairs of numbers?
>
> > <tip-title>abs</tip-title>
> > There is a [built-in function `abs`][abs-function] that returns the absolute value of
> > a number:
> > ```python
> > print(abs(-12))
> > ```
> >
> > ```
> > 12
> > ```
> {: .tip}
>
> > <solution-title>1</solution-title>
> > ```
> > a = 5
> > b = 5.1
> >
> > if abs(a - b) <= 10:
> >     print('True')
> > else:
> >     print('False')
> > ```
> {: .solution}
>
> > <solution-title>2</solution-title>
> > ```
> > print(abs(a - b) <= 10)
> > ```
> >
> > This works because the Booleans `True` and `False`
> > have string representations which can be printed.
> {: .solution}
{: .question}

```python
# Test code here.
```

> <question-title>Pitfalls</question-title>
>
> A *integer* number between 0 and 100 will be provided to this function. Answer these two questions:
>
> - Will it always print something? If not, which value(s) fail?
> - Can you find any numbers the programmer explicitly wanted to handle, that aren't handled as expected?
>
> ```
> num = 42 # Randomly chosen so the code will execute, try changing it around.
> if num > 90:
>     print("great score")
> elif num < 32:
>     print("Very cold")
> elif num >= 86:
>     print("Almost")
> elif num == 86:
>     print("It's exactly this value!")
> elif 32 < num < 58:
>     print("Getting warmer")
> elif 57 < num <= 86:
>     print("Everything else goes here")
> ```
>
> > <solution-title></solution-title>
> > 1. No, it won't. 32 is the only value there that doesn't print anything. You can either do `x < 57` and later `57 <= x` to test the bigger and smaller values, or you can make use `x < 57` and `56 < x`, which have the same results, but **only with integers**. If your code accepted a float, e.g. `56.5`, both of those tests would be true. So `x < 57` and later `57 <= x` is the preferred way to write that.
> > 2. `86` is the most obvious solution to this, the programmer added a check specifically to see if the value was 86, but instead it's caught by the previous case.
> {: .solution}
{: .question}

```python
num = 42 # Randomly chosen so the code will execute, try changing it around.
if num > 90:
    print("great score")
elif num < 32:
    print("Very cold")
elif num >= 86:
    print("Almost")
elif num == 86:
    print("It's exactly this value!")
elif 32 < num < 58:
    print("Getting warmer")
elif 57 < num <= 86:
    print("Everything else goes here")
```

> <tip-title>Why a synthetic example like this?</tip-title>
> Complicated if/elif/else cases are common in code, you need to be able to spot these sort of issues. For example there are large if/else cases in the [Galaxy codebase](https://github.com/galaxyproject/galaxy/blob/9143dd7ca46d150ebfb26febbe187979f682da51/tools/stats/grouping.py#L153-L176), sometimes nested even, and being ale to predict their behaviour is really important to being able to work with the code. [Missing else cases](https://github.com/galaxyproject/galaxy/blob/9143dd7ca46d150ebfb26febbe187979f682da51/tools/stats/grouping.py#L185-L195) are sometimes important, sometimes a bug, sometimes just the code hasn't been implemented yet, which is why we always write good code comments!
{: .tip}


topic: data-science
tutorial: python-exceptions
title: Python - Try & Except
difficulty: introductory
time_estimation: 20M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: e51cafc688c55f15b7be0f146885487d
last_modified: 2026-05-15T22:28:11.101957
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-exceptions/tutorial.html

content:
Try/except are a construct in Python used to catch a potential exception. Sometimes things go wrong in your code! Or in someone else's code in a module. Sometimes some errors might be expected like when you try and read a user supplied file, maybe it isn't available because they've specified the wrong path.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Raise

When you're writing code, sometimes there are errors you might need to handle:

- If you're calculating the mean of a set of numbers, and the list is empty. You don't want to divide by `len(numbers)` and trigger a Zero Division Error.
- If you're opening a file, there maybe a chance that the file is not found.

Using try/excepts allow you to:

- provide friendlier and more useful error messages
- handle expected error cases

For instance, returning to the `mean()` function example, what should `mean([])` return for a hypothetical mean function that calculates the average of a list of numbers. Should it return 0? It probably should return an error. Let's look at some example code:


```python
def mean(numbers):
    return sum(numbers) / len(numbers)

mean([1, 2, 3])
mean([])
```

This raises a `ZeroDivisionError`  but we can make this a more friendly error message by raising our own exception.

```python
def mean(numbers):
    if not numbers:
        raise ValueError("You are calculating the mean of an empty list, which is not possible.")
    return sum(numbers) / len(numbers)

mean([1, 2, 3])
mean([])
```

> <tip-title>Where do ValueError, ZeroDivisionError come from?</tip-title>
> There are loads of different types of exception codes! [The python documentation has a large list](https://docs.python.org/3/library/exceptions.html) of exceptions and some descriptions for when or why those exceptions might be raised.
{: .tip}

Now we get a much more useful error message from the function! Using `raise` is especially important for *library authors*, people writing python modules that we all use. If they provide useful error messages, it helps you as an end user understand what's happening.

## Try & Except

Let's look at how you would handle one of these exceptions, we'll continue with the `mean()` example above.

```python
numbers = []
try:
    result = mean(numbers)
    print(f"Result is {result}")
except ValueError:
    print("We cannot calculate the mean")
```

Here we use `try:` to setup a new block, and this code is **tried**, Python attempts to execute it. Below are one or more `except:` blocks which catch specific errors. Here we have specifically said we know a ValueError can happen, and decided to handle it.

Or for another example, accessing a user supplied file. Oftentimes users will call your program and supply a non-existent, or inacessible file. Here you can use multiple `except` blocks to catch all of those potential errors.

```python
user_supplied_file = 'does-not-exist.txt'
try:
    open(user_supplied_file, 'r')
except FileNotFoundError:
    print(f"The path you supplied ({user_supplied_file}) doesn't exist, please double check it!")
except PermissionError:
    print(f"You supplied a valid file, but it is unreadable. Try changing it's permissions with `chmod +r {user_supplied_file}`")
```

Failing to open a file raises a `FileNotFoundError` which indicates the file isn't available, and `PermissionError` indicates that a file is unreadable. However in practice, sometimes you'll see something like this:

```
# Bad!
try:
    doSomething()
except:
    print("error")
```

This is called a **bare exception**, and will catch any exception, compared with `except ValueError` which only catches value errors. People consider this generally a bad idea, termed *code smell*. (Because it smells (appears) bad!)

## Finally

The last portion of the `try:`/`except:` block is `finally:`, a third block which lets us do cleanup. It's often very nice to your users that if your program fails halfway through, that you cleanup after yourself.

```python
import os

try:
    with open('gene_query.fa', 'w') as handle:
        handle.write(">some fasta sequence we want to search against a database")

    # runQuery('gene_query.fa')

    # But here we have an error! Something goes wrong!
    1 / 0
except:
    # And now our results are invalid.
    print("Something went wrong! Oh no! Check your inputs.")
finally:
    # So we should cleanup this temporary file we created, so it doesn't cause
    # problems or distract the user from the results file.
    # This function will delete a file from your computer:
    os.remove('gene_query.fa')
```

## Fallback

Sometimes we can use `try`/`except` to have a fallback option. Consider the pseudocode below:

```
try:
    runBlast()
except BlastNotAvailable:
    try:
        runBLAT()
    except BLATnotAvailable:
        print("Neither Blast nor BLAT were available.")
```

Sometimes you have a fallback option, some other tool you can use in its place. When that's possible, you can use `try` and `except` to handle those cases and work around potential issues. But this isn't always the case, sometimes you just need to print your error message and stop executing code.


topic: data-science
tutorial: python-functions
title: Python - Functions
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: e7ec47f79b2c90c3322986ccd95c01e4
last_modified: 2026-05-15T22:28:11.102822
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-functions/tutorial.html

content:
Functions are the basic unit of all work in Python! Absolutely everything you do uses functions. Conceptually, functions are super simple. Just like in maths, they take an input, do some transformation, and return an output. As an example, `f(x) = x + 2` is a function that calculates whatever value you request, plus two. But functions are foundational, so, you should understand them well before moving on.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## What is a Function

Functions are a way to re-use some computation you want to do, multiple times. If you don't have a function, you need to re-write the calculation every time, so we use functions to collect those statements and make them easy to re-run. Additionally they let us "parameterise" some computation. Instead of computing the same value every time, we can template it out, we can re-run the computation with new inputs, and see new results.

> > <code-in-title>Math</code-in-title>
> > ```
> > # Define our function
> > f(x) = 3 * x
> > # Compute some value
> > f(3) # is 9
> > ```
> {: .code-in}
>
> > <code-out-title>Python</code-out-title>
> > ```
> > # Define our function
> > def f(x):
> >    return 3 * x
> > # Compute some value
> > f(3)
> > ```
> {: .code-out}
>
{: .code-2col}

We've talked about mathematical functions before, but now we'll talk about more programing-related functions


## Create Functions

Human beings can only keep a few items in working memory at a time. Breaking down larger/more complicated pieces of code in functions helps in understanding and using it. A function can be re-used. Write one time, use many times. (Known as staying {DRY} in the industry.)

Knowing what Americans mean when they talk about temperatures and weather can be difficult, but we can wrap the temperature conversion calculation ($$^{\circ}\text{C} = (^{\circ}\text{F} - 32) * \dfrac{5}{9}$$) up as a function that we can easily re-use.

```python
def fahr_to_celsius(temp):
    return ((temp - 32) * (5/9))
```

![The above function fahr to celsius is shown except annotated. def is labelled "def statement", fahr_to_celsius is noted as the function name. Inside parentheses is temp and an arrow shows it is called parameter names. The next line which is indented is annotated as the function body which has a return statement and the calculation from above.](../../images/python-basics/Figure7_Functions.png)

The function definition opens with the keyword `def` followed by the name of the function `fahr_to_celsius` and a parenthesized list of parameter names `temp`. The body of the function — the statements that are executed when it runs — is indented below the definition line. The body concludes with a `return` keyword followed by the return value.

When we call the function, the values we pass to it are assigned to those variables so that we can use them inside the function. Inside the function, we use a return statement to send a result back to whoever asked for it.

Let’s try running our function.

```python
fahr_to_celsius(32)
print(f"freezing point of water: {fahr_to_celsius(32)}C")
print(f"boiling point of water: {fahr_to_celsius(212)}C")
```

> <tip-title>Formatting Strings</tip-title>
> There are several ways to print out a few values in python. We'd recommend you to use `f-strings` as it's the cleanest and most modern way to do it.
>
> `f`-strings start with an `f` (very descriptive name eh?). Within the text between the single or double quotes (`'`/`"`) you can use curly braces to refer to variables or python code which will be placed there in the string.
>
> ```
> a = 10
> b = f"Here is the value of a: {a}"
> print(b)
> print(f"Here is the value of a: {5 + 5}")
> print(f"Here is the value of a: {function_that_returns_10()}")
> ```
>
> All of those would print out `Here is the value of a: 10`.
>
> f-strings can be a lot fancier for formatting decimal places, but we don't need that for now. Just know:
>
> 1. Start with an `f`
> 2. Use braces to use the value of a variable, a function, or some python expression.
{: .tip}

We’ve successfully called the function that we defined, and we have access to the value that we returned.

Now that we’ve seen how to turn Fahrenheit into Celsius, we can also write the function to turn Celsius into Kelvin:

```python
def celsius_to_kelvin(temp_c):
    return temp_c + 273.15

print(f'freezing point of water in Kelvin: {celsius_to_kelvin(0.)}')
```

> <tip-title>What is `0.`</tip-title>
> That's a float! A `.` in a number makes it a float, rather than an integer.
{: .tip}

What about converting Fahrenheit to Kelvin? We could write out both formulae, but we don’t need to. Instead, we can *compose* the two functions we have already created:

```python
def fahr_to_kelvin(temp_f):
    temp_c = fahr_to_celsius(temp_f)
    temp_k = celsius_to_kelvin(temp_c)
    return temp_k

print(f'boiling point of water in Kelvin: {fahr_to_kelvin(212.0)}')
```

This is our first taste of how larger programs are built: we define basic operations, then combine them in ever-larger chunks to get the effect we want. Real-life functions will usually be larger than the ones shown here — typically half a dozen to a few dozen lines — but they shouldn’t ever be much longer than that, or the next person who reads it won’t be able to understand what’s going on.

### Documentation

Documenting your code is extremely, *extremely*, **extremely** important to do. We all forget what we're doing, it's only normal, so documenting what you're doing is key to being able to restart work later.

```python
def fahr_to_kelvin(temp_f):
    """
    Converts a temperature from Fahrenheit to Kelvin

    temp_f: the temperature in Fahrenheit

    returns the temperature in Celsius
    """
    temp_c = fahr_to_celsius(temp_f)
    temp_k = celsius_to_kelvin(temp_c)
    return temp_k

print(f'boiling point of water in Kelvin: {fahr_to_kelvin(212.0)}')
```

For a function this small, with such a descriptive name (`fahr_to_kelvin`) it feels quite obvious what the function should do, what inputs it takes, what outputs it produces. However, you will thank yourself in the future if you do this now. You may think you will remember what the code does, but, be kind to your future self who is busy and stressed and may not want to spend time reading the code over again to figure out what every single function does.

> <question-title>Converting statements to functions</question-title>
> A lot of what you'll do in programing is to turn a procedure that you want to do, into statements and a function.
>
> Fill in the missing portions of this function, two average numbers. Then use it to find the average of 32326 and 631
>
> > <solution-title></solution-title>
> > ```
> > def average2(a, b):
> >     c = (a + b) / 2
> >     return c
> > ```
> > We call it "average2" here because it will only average two numbers. It will not work for three numbers or a list of them.
> {: .solution}
{: .question}

```python
# Test out solutions here!
def average2(a, b):
    c =
    return c

print(average2(32326, 631))
```

> <question-title>A more complicated example</question-title>
>
> The formula for a 90° triangle can be expressed as: $$c = \sqrt{a^2 + b^2}$$
>
> 1. Write a function which takes `a` and `b`, and calculates `c`
> 2. Name this function "pythagorus"
> 3. Remember to import math, if you haven't already
>
> > <solution-title></solution-title>
> > ```
> > def pythagorus(a, b):
> >     c = math.sqrt(math.pow(a, 2) + math.pow(b, 2))
> >     return c
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!


print(pythagorus(1234, 4321)) # Should return 4493.750883170984
```


### Variable Scope

In composing our temperature conversion functions, we created variables inside of those functions, `temp`, `temp_c`, `temp_f`, and `temp_k`. We refer to these variables as local variables because they no longer exist once the function is done executing. If we try to access their values outside of the function, we will encounter an error:

```python
print(f'Again, temperature in Kelvin was: {temp_k}')
```

If you want to reuse the temperature in Kelvin after you have calculated it with fahr_to_kelvin, you can store the result of the function call in a variable:

```python
temp_kelvin = fahr_to_kelvin(212.0)
print(f'temperature in Kelvin was: {temp_kelvin}')
```

Watch out for scope issues:

- Variables created inside a function will stay inside the function
- Variables created outside of the function can be accessible inside the function, but you should not do this!
- Ensure that variables are properly scoped will prevent later errors when working with modules or testing big projects.


```python
a = 1
b = 2.0

# Location 1

def some_generic_computation(x, y):
    c = x + y
    d = x * y
    e = c / d
    # Location 2
    return e

# Location 3
```

> <question-title>Scope</question-title>
> Given the above code, which variables are accessible at Locations 1, 2, and 3?
>
> > <solution-title></solution-title>
> > 1. a, b
> > 2. a and b are there but you shouldn't use these. x, y, c, d, e are also accessible.
> > 3. a, b.
> {: .solution}
{: .question}

### Defining Default parameters

If we usually want a function to work one way, but occasionally need it to do something else, we can allow people to pass a parameter when they need to but provide a default to make the normal case easier. The example below shows how Python matches values to parameters:

```python
def display(a=1, b=2, c=3):
    print(f'a: {a}, b: {b}, c: {c}')

# no parameters:
display()
# one parameter:
display(55)
# two parameters:
display(55, 66)
```

As this example shows, parameters are matched up from left to right, and any that haven’t been given a value explicitly get their default value. We can override this behavior by naming the value as we pass it in:

```python
# only setting the value of c
display(c=77)
```


> <question-title>Exercise: Signing a message</question-title>
> Let's test out a default argument. Imagine you are printing out a message, and at the bottom it should have a signature.
>
> Inputs:
> - `message`: a variable that is always provided to the function, it has no default.
> - `signature`: a variable that can be *optionally* provided, it should have a default like your name.
>
> You can accomplish this with *three print statements*:
> 1. Print the message
> 2. Print nothing (i.e. `print()`)
> 3. Print a signature variable.
>
> > <solution-title></solution-title>
> > ```
> > def myFunction(message, signature="Your name"):
> >     print(message)
> >     print()
> >     print(signature)
> > ```
> {: .solution}
{: .question}

```python
# Test things out here!
def myFunction # Fix this function!


# Here are some test cases, for you to check if your function works!
myFunction('This is a message')
myFunction('This is a message', signature='Jane Doe')
```


topic: data-science
tutorial: sql-basic
title: Introduction to SQL
difficulty: introductory
time_estimation: 3H
tools_json: []
requirements_json: ["[]"]
tags_json: ["SQL"]
content_hash: 5f8644b505f509e1a5bd63f555b6f442
last_modified: 2026-05-15T22:28:11.120461
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-basic/tutorial.html

content:
This tutorial will introduce you to {SQL} which can be used to query databases!

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


```sql
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c http://swcarpentry.github.io/sql-novice-survey/files/survey.db
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///survey.db")
%load_ext sql
%sql sqlite:///survey.db
%config SqlMagic.displaycon=False
```

# Selecting Data

A relational database
is a way to store and manipulate information.
Databases are arranged as table.
Each table has columns (also known as fields) that describe the data,
and rows (also known as records) which contain the data.

When we are using a spreadsheet,
we put formulas into cells to calculate new values based on old ones.
When we are using a database,
we send commands
(usually called queries)
to a database manager:
a program that manipulates the database for us.
The database manager does whatever lookups and calculations the query specifies,
returning the results in a tabular form
that we can then use as a starting point for further queries.

Queries are written in a language called {SQL},
SQL provides hundreds of different ways to analyze and recombine data.
We will only look at a handful of queries,
but that handful accounts for most of what scientists do.

> <tip-title>Changing Database Managers</tip-title>
>
> Many database managers --- Oracle,
> IBM DB2, PostgreSQL, MySQL, Microsoft Access, and SQLite ---  understand
> SQL but each stores data in a different way,
> so a database created with one cannot be used directly by another.
> However, every database manager
> can import and export data in a variety of formats like .csv, SQL,
> so it *is* possible to move information from one to another.
{: .tip}

Before we get into using {SQL} to select the data, let's take a look at the tables of the database we will use in our examples:

**Person**: people who took readings.

|id      |personal |family
|--------|---------|----------
|dyer    |William  |Dyer
|pb      |Frank    |Pabodie
|lake    |Anderson |Lake
|roe     |Valentina|Roerich
|danforth|Frank    |Danforth

**Site**: locations where readings were taken.

|name |lat   |long   |
|-----|------|-------|
|DR-1 |-49.85|-128.57|
|DR-3 |-47.15|-126.72|
|MSK-4|-48.87|-123.4 |

**Visited**: when readings were taken at specific sites.

|id   |site |dated     |
|-----|-----|----------|
|619  |DR-1 |1927-02-08|
|622  |DR-1 |1927-02-10|
|734  |DR-3 |1930-01-07|
|735  |DR-3 |1930-01-12|
|751  |DR-3 |1930-02-26|
|752  |DR-3 |None      |
|837  |MSK-4|1932-01-14|
|844  |DR-1 |1932-03-22|

**Survey**: the actual readings.  The field `quant` is short for quantitative and indicates what is being measured.  Values are `rad`, `sal`, and `temp` referring to 'radiation', 'salinity' and 'temperature', respectively.

|taken|person|quant|reading|
|-----|------|-----|-------|
|619  |dyer  |rad  |9.82   |
|619  |dyer  |sal  |0.13   |
|622  |dyer  |rad  |7.8    |
|622  |dyer  |sal  |0.09   |
|734  |pb    |rad  |8.41   |
|734  |lake  |sal  |0.05   |
|734  |pb    |temp |-21.5  |
|735  |pb    |rad  |7.22   |
|735  |None  |sal  |0.06   |
|735  |None  |temp |-26.0  |
|751  |pb    |rad  |4.35   |
|751  |pb    |temp |-18.5  |
|751  |lake  |sal  |0.1    |
|752  |lake  |rad  |2.19   |
|752  |lake  |sal  |0.09   |
|752  |lake  |temp |-16.0  |
|752  |roe   |sal  |41.6   |
|837  |lake  |rad  |1.46   |
|837  |lake  |sal  |0.21   |
|837  |roe   |sal  |22.5   |
|844  |roe   |rad  |11.25  |

Notice that three entries --- one in the `Visited` table,
and two in the `Survey` table --- don't contain any actual
data, but instead have a special `None` entry:
we'll return to these missing values.

For now,
let's write an SQL query that displays scientists' names.
We do this using the SQL command `SELECT`,
giving it the names of the columns we want and the table we want them from.
Our query and its output look like this:

```sql
SELECT family, personal FROM Person;
```

The semicolon at the end of the query
tells the database manager that the query is complete and ready to run.
We have written our commands in upper case and the names for the table and columns
in lower case,
but we don't have to:
as the example below shows,
SQL is case insensitive.

```sql
SeLeCt FaMiLy, PeRsOnAl FrOm PeRsOn;
```

You can use SQL's case insensitivity to your advantage. For instance,
some people choose to write SQL keywords (such as `SELECT` and `FROM`)
in capital letters and **field** and **table** names in lower
case. This can make it easier to locate parts of an SQL statement. For
instance, you can scan the statement, quickly locate the prominent
`FROM` keyword and know the table name follows.  Whatever casing
convention you choose, please be consistent: complex queries are hard
enough to read without the extra cognitive load of random
capitalization.  One convention is to use UPPER CASE for SQL
statements, to distinguish them from tables and column names. This is
the convention that we will use for this lesson.

> <question-title>Is a personal and family name column a good design?</question-title>
> If you were tasked with designing a database to store this same data, is storing the name data in
> this way the best way to do it? Why or why not?
>
> Can you think of any names that would be difficult to enter in such a schema?
>
> > <solution-title></solution-title>
> > No, it is generally not. There are a lot of [falsehoods that programmers believe about names](https://shinesolutions.com/2018/01/08/falsehoods-programmers-believe-about-names-with-examples/).
> > The situation is much more complex as you can read in that article, but names vary wildly and
> > generally placing constraints on how names are entered is only likely to frustrate you or your
> > users later on when they need to enter data into that database.
> >
> > In general you should consider using a single text field for the name and allowing users to
> > specify them as whatever they like (if it is a system with registration), or asking what they
> > wish to be recorded (if you are doing this sort of data collection).
> >
> > If you are doing scientific research, you might know that names are generally very poor
> > identifiers of a single human, and in that case consider recording their
> > [ORCiD](https://orcid.org/) which will help you reference that individual when you are
> > publishing later.
> >
> > This is also a good time to consider what data you really *need* to collect. If you are working
> > in the EU under GDPR, do you really need their full legal name? Is that necessary? Do you have a
> > plan for ensuring that data is correct when publishing, if any part of their name has changed
> > since?
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

While we are on the topic of SQL's syntax, one aspect of SQL's syntax
that can frustrate novices and experts alike is forgetting to finish a
command with `;` (semicolon).  When you press enter for a command
without adding the `;` to the end, it can look something like this:

~~~
SELECT id FROM Person
...>
...>
~~~

This is SQL's prompt, where it is waiting for additional commands or
for a `;` to let SQL know to finish.  This is easy to fix!  Just type
`;` and press enter!

Now, going back to our query,
it's important to understand that
the rows and columns in a database table aren't actually stored in any particular order.
They will always be *displayed* in some order,
but we can control that in various ways.
For example,
we could swap the columns in the output by writing our query as:

```sql
SELECT personal, family FROM Person;
```

or even repeat columns:

```sql
SELECT id, id, id FROM Person;
```

As a shortcut,
we can select all of the columns in a table using `*`:

```sql
SELECT * FROM Person;
```

> <question-title>Selecting Site Names</question-title>
>
> Write a query that selects only the `name` column from the `Site` table.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT name FROM Site;
> > ~~~
> >
> > |name      |
> > |----------|
> > |DR-1      |
> > |DR-3      |
> > |MSK-4     |
> {: .solution}
{: .question}


```sql
-- Try solutions here!
```

> <question-title>Query Style</question-title>
>
> Many people format queries as:
>
> ~~~
> SELECT personal, family FROM person;
> ~~~
>
> or as:
>
> ~~~
> select Personal, Family from PERSON;
> ~~~
>
> What style do you find easiest to read, and why?
{: .question}

```sql
-- Try solutions here!
```


# Sorting and Removing Duplicates

In beginning our examination of the Antarctic data, we want to know:

* what kind of quantity measurements were taken at each site;
* which scientists took measurements on the expedition;

To determine which measurements were taken at each site,
we can examine the `Survey` table.
Data is often redundant,
so queries often return redundant information.
For example,
if we select the quantities that have been measured
from the `Survey` table,
we get this:

```sql
SELECT quant FROM Survey;
```


This result makes it difficult to see all of the different types of
`quant` in the Survey table.  We can eliminate the redundant output to
make the result more readable by adding the `DISTINCT` keyword to our
query:

```sql
SELECT DISTINCT quant FROM Survey;
```

If we want to determine which visit (stored in the `taken` column)
have which `quant` measurement,
we can use the `DISTINCT` keyword on multiple columns.
If we select more than one column,
distinct *sets* of values are returned
(in this case *pairs*, because we are selecting two columns):

```sql
SELECT DISTINCT taken, quant FROM Survey;
```

Notice in both cases that duplicates are removed
even if the rows they come from didn't appear to be adjacent in the database table.


Our next task is to identify the scientists on the expedition by looking at the `Person` table.
As we mentioned earlier,
database records are not stored in any particular order.
This means that query results aren't necessarily sorted,
and even if they are,
we often want to sort them in a different way,
e.g., by their identifier instead of by their personal name.
We can do this in SQL by adding an `ORDER BY` clause to our query:

```sql
SELECT * FROM Person ORDER BY id;
```

|id     |personal |family  |
|-------|---------|--------|
|danfort|Frank    |Danforth|
|dyer   |William  |Dyer    |
|lake   |Anderson |Lake    |
|pb     |Frank    |Pabodie |
|roe    |Valentina|Roerich |

By default, when we use `ORDER BY`,
results are sorted in ascending order of the column we specify
(i.e.,
from least to greatest).

We can sort in the opposite order using `DESC` (for "descending"):

> <tip-title>A note on ordering</tip-title>
>
> While it may look that the records are consistent every time we ask for them in this lesson, that is because no one has changed or modified any of the data so far. Remember to use `ORDER BY` if you want the rows returned to have any sort of consistent or predictable order.
{: .tip}
```sql
SELECT * FROM person ORDER BY id DESC;
```

(And if we want to make it clear that we're sorting in ascending order,
we can use `ASC` instead of `DESC`.)


In order to look at which scientist measured quantities during each visit,
we can look again at the `Survey` table.
We can also sort on several fields at once.
For example,
this query sorts results first in ascending order by `taken`,
and then in descending order by `person`
within each group of equal `taken` values:

```sql
SELECT taken, person, quant FROM Survey ORDER BY taken ASC, person DESC;
```

This query gives us a good idea of which scientist was involved in which visit,
and what measurements they performed during the visit.

Looking at the table, it seems like some scientists specialized in
certain kinds of measurements.  We can examine which scientists
performed which measurements by selecting the appropriate columns and
removing duplicates.

```sql
SELECT DISTINCT quant, person FROM Survey ORDER BY quant ASC;
```

> <question-title>Finding Distinct Dates</question-title>
>
> Write a query that selects distinct dates from the `Visited` table.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT DISTINCT dated FROM Visited;
> > ~~~
> >
> > |dated     |
> > |----------|
> > |1927-02-08|
> > |1927-02-10|
> > |1930-01-07|
> > |1930-01-12|
> > |1930-02-26|
> > |&nbsp;    |
> > |1932-01-14|
> > |1932-03-22|
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Displaying Full Names</question-title>
>
> Write a query that displays the full names of the scientists in the `Person` table,
> ordered by family name.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT personal, family FROM Person ORDER BY family ASC;
> > ~~~
> >
> > |personal  |family    |
> > |----------|----------|
> > |Frank     |Danforth  |
> > |William   |Dyer      |
> > |Anderson  |Lake      |
> > |Frank     |Pabodie   |
> > |Valentina |Roerich   |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <tip-title>Is sorting names useful?</tip-title>
> If you are someone with a name which falls at the end of the alphabet, you've likely been
> penalised for this your entire life. Alphabetically sorting names should always be looked at
> critically and through a lens to whether you are fairly reflecting everyone's contributions,
> rather than just the default sort order.
>
> There are many options, either by some metric of contribution that everyone could agree on, or
> better, consider random sorting, like the GTN uses with our [Hall of Fame]({% link hall-of-fame.md %})
> page where we intentionally order randomly to tell contributors that no one persons
> contributions matter more than anothers.
>
> > The evidence provided in a variety of studies leaves no doubt that an
> > alphabetical author ordering norm disadvantages researchers with
> > last names toward the end of the alphabet. There is furthermore con-
> > vincing evidence that researchers are aware of this and that they
> > react strategically to such alphabetical discrimination, for example
> > with their choices of who to collaborate with. See {% cite Weber_2018 %} for more.
> {: .quote}
{: .tip}

> <tip-title>Name collation</tip-title>
> When you are sorting things in SQL, you need to be aware of something called collation which can
> affect your results if you have values that are not the letters A-Z. Collating is the process of
> sorting values, and this affects many human languages when storing data in a database.
>
> Here is a Dutch example. In the old days their alphabet contained a `ÿ` which was later replaced
> with `ĳ`, a digraph of two characters squished together. This is commonly rendered as `ij`
> however, two separate characters, due to the internet and widespread use of keyboards featuring
> mainly ascii characters. However, it is still the 25th letter of their alphabet.
>
> ```
> sqlite> create table nl(value text);
> sqlite> insert into nl values ('appel'), ('beer'), ('index'), ('ijs'), ('jammer'), ('winkel'), ('zon');
> sqlite> select * from nl order by value;
> appel
> beer
> index
> ijs
> jammer
> winkel
> zon
> ```
>
> Find a dutch friend and ask them if this is the correct order for this list. Unfortunately it
> isn't. Even though it is `ij` as two separate characters, it should be sorted as if it was `ĳ` or
> `ÿ`, before `z`. Like so: appel, beer, index, jammer, winkel, ijs, zon
>
> While there is not much you can do about it now (you're just beginning!) it is something you
> should be aware of. When you later need to know about this, you will find the term 'collation'
> useful, and you'll find the procedure is different for every database engine.
{: .tip}

# Filtering

One of the most powerful features of a database is
the ability to filter data,
i.e.,
to select only those records that match certain criteria.
For example,
suppose we want to see when a particular site was visited.
We can select these records from the `Visited` table
by using a `WHERE` clause in our query:

```sql
SELECT * FROM Visited WHERE site = 'DR-1';
```

The database manager executes this query in two stages.
First,
it checks at each row in the `Visited` table
to see which ones satisfy the `WHERE`.
It then uses the column names following the `SELECT` keyword
to determine which columns to display.

This processing order means that
we can filter records using `WHERE`
based on values in columns that aren't then displayed:

```sql
SELECT id FROM Visited WHERE site = 'DR-1';
```

![SQL Filtering in Action]`(../../images/carpentries-sql/sql-filter.svg)

We can use many other Boolean operators to filter our data.
For example,
we can ask for all information from the DR-1 site collected before 1930:

```sql
SELECT * FROM Visited WHERE site = 'DR-1' AND dated < '1930-01-01';
```

> <tip-title>Date Types</tip-title>
>
> Most database managers have a special data type for dates.
> In fact, many have two:
> one for dates,
> such as "May 31, 1971",
> and one for durations,
> such as "31 days".
> SQLite doesn't:
> instead,
> it stores dates as either text
> (in the ISO-8601 standard format "YYYY-MM-DD HH:MM:SS.SSSS"),
> real numbers
> ([Julian days](https://en.wikipedia.org/wiki/Julian_day), the number of days since November 24, 4714 BCE),
> or integers
> ([Unix time](https://en.wikipedia.org/wiki/Unix_time), the number of seconds since midnight, January 1, 1970).
> If this sounds complicated,
> it is,
> but not nearly as complicated as figuring out
> [historical dates in Sweden](https://en.wikipedia.org/wiki/Swedish_calendar).
{: .tip}

> <tip-title>'30: 1930 or 2030?</tip-title>
> Storing the year as the last two digits causes problems in databases, and is part of what caused
> [Y2K](https://en.wikipedia.org/wiki/Year_2000_problem). Be sure to use the databases' built in
> format for storing dates, if it is available as that will generally avoid any major issues.
>
> Similarly there is a ["Year 2038 problem"](https://en.wikipedia.org/wiki/Year_2000_problem#Year_2038_problem),
> as the timestamps mentioned above that count seconds since Jan 1, 1970 were running out of space
> on 32-bit machines. Many systems have since migrated to work around this with 64-bit timestamps.
{: .tip}

If we want to find out what measurements were taken by either Lake or Roerich,
we can combine the tests on their names using `OR`:

```sql
SELECT * FROM Survey WHERE person = 'lake' OR person = 'roe';
```

Alternatively,
we can use `IN` to see if a value is in a specific set:

```sql
SELECT * FROM Survey WHERE person IN ('lake', 'roe');
```

We can combine `AND` with `OR`,
but we need to be careful about which operator is executed first.
If we *don't* use parentheses,
we get this:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND person = 'lake' OR person = 'roe';
```

which is salinity measurements by Lake,
and *any* measurement by Roerich.
We probably want this instead:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND (person = 'lake' OR person = 'roe');
```

We can also filter by partial matches.  For example, if we want to
know something just about the site names beginning with "DR" we can
use the `LIKE` keyword.  The percent symbol acts as a
wildcard, matching any characters in that
place.  It can be used at the beginning, middle, or end of the string
[See this page on wildcards](https://www.w3schools.com/sql/sql_wildcards.asp) for more information:

```sql
SELECT * FROM Visited WHERE site LIKE 'DR%';
```


Finally,
we can use `DISTINCT` with `WHERE`
to give a second level of filtering:

```sql
SELECT DISTINCT person, quant FROM Survey WHERE person = 'lake' OR person = 'roe';
```

But remember:
`DISTINCT` is applied to the values displayed in the chosen columns,
not to the entire rows as they are being processed.

> <tip-title>Growing Queries</tip-title>
>
> What we have just done is how most people "grow" their {SQL} queries.
> We started with something simple that did part of what we wanted,
> then added more clauses one by one,
> testing their effects as we went.
> This is a good strategy --- in fact,
> for complex queries it's often the *only* strategy --- but
> it depends on quick turnaround,
> and on us recognizing the right answer when we get it.
>
> The best way to achieve quick turnaround is often
> to put a subset of data in a temporary database
> and run our queries against that,
> or to fill a small database with synthesized records.
> For example,
> instead of trying our queries against an actual database of 20 million Australians,
> we could run it against a sample of ten thousand,
> or write a small program to generate ten thousand random (but plausible) records
> and use that.
{: .tip}

> <question-title>Fix This Query</question-title>
>
> Suppose we want to select all sites that lie within 48 degrees of the equator.
> Our first query is:
>
> ~~~
> SELECT * FROM Site WHERE (lat > -48) OR (lat < 48);
> ~~~
>
> Explain why this is wrong,
> and rewrite the query so that it is correct.
>
> > <solution-title></solution-title>
> >
> > Because we used `OR`, a site on the South Pole for example will still meet
> > the second criteria and thus be included. Instead, we want to restrict this
> > to sites that meet _both_ criteria:
> >
> > ~~~
> > SELECT * FROM Site WHERE (lat > -48) AND (lat < 48);
> > ~~~
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Finding Outliers</question-title>
>
> Normalized salinity readings are supposed to be between 0.0 and 1.0.
> Write a query that selects all records from `Survey`
> with salinity values outside this range.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT * FROM Survey WHERE quant = 'sal' AND ((reading > 1.0) OR (reading < 0.0));
> > ~~~
> >
> > |taken     |person    |quant     |reading   |
> > |----------|----------|----------|----------|
> > |752       |roe       |sal       |41.6      |
> > |837       |roe       |sal       |22.5      |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title> Matching Patterns</question-title>
>
> Which of these expressions are true?
>
> 1. `'a' LIKE 'a'`
> 2. `'a' LIKE '%a'`
> 3. `'beta' LIKE '%a'`
> 4. `'alpha' LIKE 'a%%'`
> 5. `'alpha' LIKE 'a%p%'`
>
> > <solution-title></solution-title>
> >
> > 1. True because these are the same character.
> > 2. True because the wildcard can match _zero_ or more characters.
> > 3. True because the `%` matches `bet` and the `a` matches the `a`.
> > 4. True because the first wildcard matches `lpha` and the second wildcard matches zero characters (or vice versa).
> > 5. True because the first wildcard matches `l` and the second wildcard matches `ha`.
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <tip-title>Case-insensitive matching</tip-title>
> But what about if you don't care about if it's `ALPHA` or `alpha` in the database, and you are
> using a language that has a notion of case (unlike e.g. Chinese, Japenese)?
>
> Then you can use the `ILIKE` operator for 'case Insensitive LIKE'.
> for example the following are true:
>
> - `'a' ILIKE 'A'`
> - `'AlPhA' ILIKE '%lpha'`
{: .tip}


# Calculating New Values

After carefully re-reading the expedition logs,
we realize that the radiation measurements they report
may need to be corrected upward by 5%.
Rather than modifying the stored data,
we can do this calculation on the fly
as part of our query:

```sql
SELECT 1.05 * reading FROM Survey WHERE quant = 'rad';
```

When we run the query,
the expression `1.05 * reading` is evaluated for each row.
Expressions can use any of the fields,
all of usual arithmetic operators,
and a variety of common functions.
(Exactly which ones depends on which database manager is being used.)
For example,
we can convert temperature readings from Fahrenheit to Celsius
and round to two decimal places:

```sql
SELECT taken, round(5 * (reading - 32) / 9, 2) FROM Survey WHERE quant = 'temp';
```


As you can see from this example, though, the string describing our
new field (generated from the equation) can become quite unwieldy. {SQL}
allows us to rename our fields, any field for that matter, whether it
was calculated or one of the existing fields in our database, for
succinctness and clarity. For example, we could write the previous
query as:

```sql
SELECT taken, round(5 * (reading - 32) / 9, 2) as Celsius FROM Survey WHERE quant = 'temp';
```

We can also combine values from different fields,
for example by using the string concatenation operator `||`:

```sql
SELECT personal || ' ' || family FROM Person;
```

But of course that can also be solved by simply having a single name field which avoids other
issues.

> <question-title> Fixing Salinity Readings</question-title>
>
> After further reading,
> we realize that Valentina Roerich
> was reporting salinity as percentages.
> Write a query that returns all of her salinity measurements
> from the `Survey` table
> with the values divided by 100.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT taken, reading / 100 FROM Survey WHERE person = 'roe' AND quant = 'sal';
> > ~~~
> >
> > |taken     |reading / 100|
> > |----------|-------------|
> > |752       |0.416        |
> > |837       |0.225        |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Unions</question-title>
>
> The `UNION` operator combines the results of two queries:
>
> ~~~
> SELECT * FROM Person WHERE id = 'dyer' UNION SELECT * FROM Person WHERE id = 'roe';
> ~~~
>
> |id  |personal |family |
> |----|-------- |-------|
> |dyer|William  |Dyer   |
> |roe |Valentina|Roerich|
>
> The `UNION ALL` command is equivalent to the `UNION` operator,
> except that `UNION ALL` will select all values.
> The difference is that `UNION ALL` will not eliminate duplicate rows.
> Instead, `UNION ALL` pulls all rows from the query
> specifics and combines them into a table.
> The `UNION` command does a `SELECT DISTINCT` on the results set.
> If all the records to be returned are unique from your union,
> use `UNION ALL` instead, it gives faster results since it skips the `DISTINCT` step.
> For this section, we shall use UNION.
>
> Use `UNION` to create a consolidated list of salinity measurements
> in which Valentina Roerich's, and only Valentina's,
> have been corrected as described in the previous challenge.
> The output should be something like:
>
> |taken|reading|
> |-----|-------|
> |619  |0.13   |
> |622  |0.09   |
> |734  |0.05   |
> |751  |0.1    |
> |752  |0.09   |
> |752  |0.416  |
> |837  |0.21   |
> |837  |0.225  |
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT taken, reading FROM Survey WHERE person != 'roe' AND quant = 'sal' UNION SELECT taken, reading / 100 FROM Survey WHERE person = 'roe' AND quant = 'sal' ORDER BY taken ASC;
> > ~~~
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Selecting Major Site Identifiers</question-title>
>
> The site identifiers in the `Visited` table have two parts
> separated by a '-':
>
> ~~~
> SELECT DISTINCT site FROM Visited;
> ~~~
>
> |site |
> |-----|
> |DR-1 |
> |DR-3 |
> |MSK-4|
>
> Some major site identifiers (i.e. the letter codes) are two letters long and some are three.
> The "in string" function `instr(X, Y)`
> returns the 1-based index of the first occurrence of string Y in string X,
> or 0 if Y does not exist in X.
> The substring function `substr(X, I, [L])`
> returns the substring of X starting at index I, with an optional length L.
> Use these two functions to produce a list of unique major site identifiers.
> (For this data,
> the list should contain only "DR" and "MSK").
>
> > <solution-title></solution-title>
> > ```
> > SELECT DISTINCT substr(site, 1, instr(site, '-') - 1) AS MajorSite FROM Visited;
> > ```
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

# Missing Data

Real-world data is never complete --- there are always holes.
Databases represent these holes using a special value called `null`.
`null` is not zero, `False`, or the empty string;
it is a one-of-a-kind value that means "nothing here".
Dealing with `null` requires a few special tricks
and some careful thinking.

By default, the Python SQL interface does not display NULL values in its output, instead it shows `None`.

To start,
let's have a look at the `Visited` table.
There are eight records,
but #752 doesn't have a date --- or rather,
its date is null:

```sql
SELECT * FROM Visited;
```

Null doesn't behave like other values.
If we select the records that come before 1930:

```sql
SELECT * FROM Visited WHERE dated < '1930-01-01';
```

we get two results,
and if we select the ones that come during or after 1930:

```sql
SELECT * FROM Visited WHERE dated >= '1930-01-01';
```

we get five,
but record #752 isn't in either set of results.
The reason is that
`null<'1930-01-01'`
is neither true nor false:
null means, "We don't know,"
and if we don't know the value on the left side of a comparison,
we don't know whether the comparison is true or false.
Since databases represent "don't know" as null,
the value of `null<'1930-01-01'`
is actually `null`.
`null>='1930-01-01'` is also null
because we can't answer to that question either.
And since the only records kept by a `WHERE`
are those for which the test is true,
record #752 isn't included in either set of results.

Comparisons aren't the only operations that behave this way with nulls.
`1+null` is `null`,
`5*null` is `null`,
`log(null)` is `null`,
and so on.
In particular,
comparing things to null with = and != produces null:

```sql
SELECT * FROM Visited WHERE dated = NULL;
```

produces no output, and neither does:

```sql
SELECT * FROM Visited WHERE dated != NULL;
```

To check whether a value is `null` or not,
we must use a special test `IS NULL`:

```sql
SELECT * FROM Visited WHERE dated IS NULL;
```

or its inverse `IS NOT NULL`:

```sql
SELECT * FROM Visited WHERE dated IS NOT NULL;
```

Null values can cause headaches wherever they appear.
For example,
suppose we want to find all the salinity measurements
that weren't taken by Lake.
It's natural to write the query like this:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND person != 'lake';
```

but this query filters omits the records
where we don't know who took the measurement.
Once again,
the reason is that when `person` is `null`,
the `!=` comparison produces `null`,
so the record isn't kept in our results.
If we want to keep these records
we need to add an explicit check:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND (person != 'lake' OR person IS NULL);
```


We still have to decide whether this is the right thing to do or not.
If we want to be absolutely sure that
we aren't including any measurements by Lake in our results,
we need to exclude all the records for which we don't know who did the work.

In contrast to arithmetic or Boolean operators, aggregation functions
that combine multiple values, such as `min`, `max` or `avg`, *ignore*
`null` values. In the majority of cases, this is a desirable output:
for example, unknown values are thus not affecting our data when we
are averaging it. Aggregation functions will be addressed in more
detail in [the next section](#).

> <question-title>Sorting by Known Date</question-title>
>
> Write a query that sorts the records in `Visited` by date,
> omitting entries for which the date is not known
> (i.e., is null).
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT * FROM Visited WHERE dated IS NOT NULL ORDER BY dated ASC;
> > ~~~
> >
> > |id        |site      |dated     |
> > |----------|----------|----------|
> > |619       |DR-1      |1927-02-08|
> > |622       |DR-1      |1927-02-10|
> > |734       |DR-3      |1930-01-07|
> > |735       |DR-3      |1930-01-12|
> > |751       |DR-3      |1930-02-26|
> > |837       |MSK-4     |1932-01-14|
> > |844       |DR-1      |1932-03-22|
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>NULL in a Set</question-title>
>
> What do you expect the following query to produce?
>
> ~~~
> SELECT * FROM Visited WHERE dated IN ('1927-02-08', NULL);
> ~~~
>
> What does it actually produce?
> > <solution-title></solution-title>
> >
> > You might expect the above query to return rows where dated is either '1927-02-08' or NULL.
> > Instead it only returns rows where dated is '1927-02-08', the same as you would get from this
> > simpler query:
> >
> > ~~~
> > SELECT * FROM Visited WHERE dated IN ('1927-02-08');
> > ~~~
> >
> > The reason is that the `IN` operator works with a set of *values*, but NULL is by definition
> > not a value and is therefore simply ignored.
> >
> > If we wanted to actually include NULL, we would have to rewrite the query to use the IS NULL condition:
> >
> > ~~~
> > SELECT * FROM Visited WHERE dated = '1927-02-08' OR dated IS NULL;
> > ~~~
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Pros and Cons of Sentinels</question-title>
>
> Some database designers prefer to use
> a sentinel value
> to mark missing data rather than `null`.
> For example,
> they will use the date "0000-00-00" to mark a missing date,
> or -1.0 to mark a missing salinity or radiation reading
> (since actual readings cannot be negative).
> What does this simplify?
> What burdens or risks does it introduce?
{: .question}

```sql
-- Try solutions here!
```

{% include _includes/quiz.html id="recap.yml" %}


topic: data-science
tutorial: python-venv
title: Virtual Environments For Software Development
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: b479120bca8c0065bc6a7789e54141d7
last_modified: 2026-05-15T22:28:11.109471
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-venv/tutorial.html

content:
"Virtual Environments" allow you to easily manage your installed Python packages and prevent conflicts between different project's dependencies. In general most modern projects should use `conda` for dependency management, but `venv` can be convenient for Python-only projects.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}


If you have a python project you are using, you will often see something like
following two lines somewhere at the top.

```python
from matplotlib import pyplot as plt
import numpy as np
```

This means that our code requires two *external libraries* (also called third-party packages or dependencies) -
`numpy` and `matplotlib`.
Python applications often use external libraries that don’t come as part of the standard Python distribution. This means
that you will have to use a *package manager* tool to install them on your system.
Applications will also sometimes need a
specific version of an external library (e.g. because they require that a particular
bug has been fixed in a newer version of the library), or a specific version of Python interpreter.
This means that each Python application you work with may require a different setup and a set of dependencies so it
is important to be able to keep these configurations separate to avoid confusion between projects.
The solution for this problem is to create a self-contained *virtual
environment* per project, which contains a particular version of Python installation plus a number of
additional external libraries.

Virtual environments are not just a feature of Python - all modern programming languages use them to isolate code
of a specific project and make it easier to develop, run, test and share code with others. In this tutorial, we learn how
to set up a virtual environment to develop our code and manage our external dependencies.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Virtual Environments

So what exactly are virtual environments, and why use them?

A Python virtual environment is an **isolated working copy** of a specific version of
Python interpreter together with specific versions of a number of external libraries installed into that
virtual environment. A virtual environment is simply a *directory with a particular
structure* which includes links to and enables multiple side-by-side installations of
different Python interpreters or different versions of the same external library to coexist on your machine and only one to be selected for each of our projects. This allows you to work on a particular
project without worrying about affecting other projects on your machine.

As more external libraries are added to your Python project over time, you can add them to
its specific virtual environment and avoid a great deal of confusion by having separate (smaller) virtual environments
for each project rather than one huge global environment with potential package version clashes. Another big motivator
for using virtual environments is that they make sharing your code with others much easier (as we will see shortly).
Here are some typical scenarios where the usage of virtual environments is highly recommended (almost unavoidable):

- You have an older project that only works under Python 2. You do not have the time to migrate the project to Python 3
  or it may not even be possible as some of the third party dependencies are not available under Python 3. You have to
  start another project under Python 3. The best way to do this on a single machine is to set up two separate Python virtual
  environments.
- One of your Python 3 projects is locked to use a particular older version of a third party dependency. You cannot use the
  latest version of the
  dependency as it breaks things in your project. In a separate branch of your project, you want to try and fix problems
  introduced by the new version of the dependency without affecting the working version of your project. You need to set up
  a separate virtual environment for your branch to 'isolate' your code while testing the new feature.

You do not have to worry too much about specific versions of external libraries that your project depends on most of the time.
Virtual environments enable you to always use the latest available version without specifying it explicitly.
They also enable you to use a specific older version of a package for your project, should you need to.

> <tip-title>A Specific Python or Package Version is Only Ever Installed Once</tip-title>
> Note that you will not have a separate Python or package installations for each of your projects - they will only
> ever be installed once on your system but will be referenced from different virtual environments.
{: .tip}

### Managing Python Virtual Environments

There are several commonly used command line tools for managing Python virtual environments:
- `venv`, available by default from the standard `Python` distribution from `Python 3.3+`
- `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions
- `pipenv`, created to fix certain shortcomings of `virtualenv`
- `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community)
- `poetry`, a modern Python packaging tool which handles virtual environments automatically

While there are pros and cons for using each of the above, all will do the job of managing Python
virtual environments for you and it may be a matter of personal preference which one you go for.
In this course, we will use `venv` to create and manage our
virtual environment (which is the preferred way for Python 3.3+).

Until you encounter the needs of a project which goes beyond what is available
in the Python ecosystem, e.g. when you depend on external packages like htslib
or bioinformatics tools that are simply not distributed as part of PyPI, then
`venv` is a good choice to get started with.


### Managing Python Packages

Part of managing your (virtual) working environment involves installing, updating and removing external packages
on your system. The Python package manager tool `pip` is most commonly used for this - it interacts
 and obtains the packages from the central repository called [Python Package Index (PyPI)](https://pypi.org/).
`pip` can now be used with all Python distributions (including Anaconda).

> <tip-title>A Note on Anaconda and `conda`</tip-title>
> Anaconda is an open source Python
> distribution commonly used for scientific programming - it conveniently installs Python, package and environment management `conda`, and a 
> number of commonly used scientific computing packages so you do not have to obtain them separately. 
> `conda` is an independent command line tool (available separately from the Anaconda distribution too) with dual functionality: (1) it is a package manager that helps you find Python packages from
> remote package repositories and install them on your system, and (2) it is also a virtual environment manager. So, you can use `conda` for both tasks instead of using `venv` and `pip`.
{: .tip}

### Many Tools for the Job

Installing and managing Python distributions, external libraries and virtual environments is, well,
complex. There is an abundance of tools for each task, each with its advantages and disadvantages, and there are different
ways to achieve the same effect (and even different ways to install the same tool!).
Note that each Python distribution comes with its own version of
`pip` - and if you have several Python versions installed you have to be extra careful to use the correct `pip` to
manage external packages for that Python version.

`venv` and `pip` are considered the *de facto* standards for virtual environment and package management for Python 3.
However, the advantages of using Anaconda and `conda` are that you get (most of the) packages needed for
scientific code development included with the distribution. If you are only collaborating with others who are also using
Anaconda, you may find that `conda` satisfies all your needs. It is good, however, to be aware of all these tools,
and use them accordingly. As you become more familiar with them you will realise that equivalent tools work in a similar
way even though the command syntax may be different (and that there are equivalent tools for other programming languages
too to which your knowledge can be ported).

![Python environment hell XKCD comic  showing boxes like pip, easy_install, homebrew 2.7, anaconda, homebrew 3.6, /usr/local/Cellar, ~/python/, and a chaotic mess of arrows moving between them all. At the bottom is the text: My python environment has become so degraded that my laptop has been declared a superfund site. (A superfund site is generally an environmental disaster area.)](../../images/xkcd/python_environment.png "Python Environment Hell from XKCD 1987 (CC-BY-NC 2.5)")

Let us have a look at how we can create and manage virtual environments from the command line using `venv` and manage packages using `pip`.

### Creating a `venv` Environment
Creating a virtual environment with `venv` is done by executing the following command:

```bash
python3 -m venv /path/to/new/virtual/environment
```

where `/path/to/new/virtual/environment` is a path to a directory where you want to place it - conventionally within
your software project so they are co-located.
This will create the target directory for the virtual environment (and any parent directories that don’t exist already).

For our project, let's create a virtual environment called `venv` off the project root:

```bash
python3 -m venv venv
```

If you list the contents of the newly created `venv` directory, on a Mac or Linux system 
(slightly different on Windows as explained below) you should see something like:

```bash
ls -l venv
```

So, running the `python3 -m venv venv` command created the target directory called `venv`
containing:

- `pyvenv.cfg` configuration file with a home key pointing to the Python installation from which the command was run,
- `bin` subdirectory (called `Scripts` on Windows) containing a symlink of the Python interpreter binary used to create the
environment and the standard Python library,
- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) to contain its own independent set of installed Python packages isolated from other projects,
- various other configuration and supporting files and subdirectories.

> <tip-title>Naming Virtual Environments</tip-title>
> What is a good name to use for a virtual environment? Using "venv" or ".venv" as the
> name for an environment and storing it within the project's directory seems to be the recommended way -
> this way when you come across such a subdirectory within a software project,
> by convention you know it contains its virtual environment details.
> A slight downside is that all different virtual environments
> on your machine then use the same name and the current one is determined by the context of the path
> you are currently located in. A (non-conventional) alternative is to
> use your project name for the name of the virtual environment, with the downside that there is nothing to indicate
> that such a directory contains a virtual environment. In our case, we have settled to use the name "venv" since it is 
> not a hidden directory and we want it to be displayed by the command line when listing directory contents (hence, 
> no need for the "." in its name that would, by convention, make it hidden). In the future,
> you will decide what naming convention works best for you. Here are some references for each of the naming conventions:
> - [The Hitchhiker's Guide to Python](https://docs.python-guide.org/dev/virtualenvs/) notes that "venv" is the general convention used globally
> - [The Python Documentation](https://docs.python.org/3/library/venv.html) indicates that ".venv" is common
> - ["venv" vs ".venv" discussion](https://discuss.python.org/t/trying-to-come-up-with-a-default-directory-name-for-virtual-environments/3750)
{: .tip}

Once you’ve created a virtual environment, you will need to activate it:

```bash
source venv/bin/activate
```

Activating the virtual environment will change your command line’s prompt to show what virtual environment
you are currently using (indicated by its name in round brackets at the start of the prompt),
and modify the environment so that running Python will get you the particular
version of Python configured in your virtual environment.

You can verify you are using your virtual environment's version of Python by checking the path using `which`:
```bash
which python3
```

When you’re done working on your project, you can exit the environment with:

```bash
deactivate
```

If you've just done the `deactivate`, ensure you reactivate the environment ready for the next part:

```bash
source venv/bin/activate
```

> <tip-title>Python Within A Virtual Environment</tip-title>
> 
> Within a virtual environment, commands `python` and `pip` will refer to the version of Python you created the environment with. If you create a virtual environment with `python3 -m venv venv`, `python` will refer to `python3` and `pip` will refer to `pip3`. 
>
> On some machines with Python 2 installed, `python` command may refer to the copy of Python 2 installed outside of the virtual environment instead, which can cause confusion. You can always check which version of Python you are using in your virtual environment with the command `which python` to be absolutely sure. We continue using `python3` and `pip3` in this material to avoid confusion for those users, but commands `python` and `pip` may work for you as expected.
{: .tip}

### Installing External Libraries in an Environment with `pip`

We noticed earlier that our code depends on two *external libraries* - `numpy` and `matplotlib`. In order
for the code to run on your machine, you need to
install these two dependencies into your virtual environment.

To install the latest version of a package with `pip` you use pip's `install` command and specify the package’s name, e.g.:

```bash
pip3 install numpy
pip3 install matplotlib
```

or like this to install multiple packages at once for short:

```bash
pip3 install numpy matplotlib
```

> <tip-title>How About `python3 -m pip install`?</tip-title>
> Why are we not using `pip` as an argument to `python3` command, in the same way we did with `venv` 
> (i.e. `python3 -m venv`)? `python3 -m pip install` should be used according to the 
> [official Pip documentation](https://pip.pypa.io/en/stable/user_guide/#running-pip); other official documentation
> still seems to have a mixture of usages. Core Python developer Brett Cannon offers a 
> [more detailed explanation](https://snarky.ca/why-you-should-use-python-m-pip/) of edge cases when the two options may produce 
> different results and recommends `python3 -m pip install`. We kept the old-style command (`pip3 install`) as it seems more 
> prevalent among developers at the moment - but it may be a convention that will soon change and certainly something you should consider. 
{: .tip}

If you run the `pip3 install` command on a package that is already installed, `pip` will notice this and do nothing.

To install a specific version of a Python package give the package name followed by `==` and the version number, e.g.
`pip3 install numpy==1.21.1`.

To specify a minimum version of a Python package, you can
do `pip3 install numpy>=1.20`.

To upgrade a package to the latest version, e.g. `pip3 install --upgrade numpy`.

To display information about a particular installed package do:

```bash
pip3 show numpy
```

To list all packages installed with `pip` (in your current virtual environment):

```bash
pip3 list
```

To uninstall a package installed in the virtual environment do: `pip3 uninstall package-name`.
You can also supply a list of packages to uninstall at the same time.

### Exporting/Importing an Environment with `pip`

You are collaborating on a project with a team so, naturally, you will want to share your environment with your
collaborators so they can easily 'clone' your software project with all of its dependencies and everyone
can replicate equivalent virtual environments on their machines. `pip` has a handy way of exporting,
saving and sharing virtual environments.

To export your active environment - use `pip freeze` command to
produce a list of packages installed in the virtual environment.
A common convention is to put this list in a `requirements.txt` file:

```bash
pip3 freeze > requirements.txt
cat requirements.txt
```

The first of the above commands will create a `requirements.txt` file in your current directory.
The `requirements.txt` file can then be committed to a version control system and
get shipped as part of your software and shared with collaborators and/or users. They can then replicate your environment and
install all the necessary packages from the project root as follows:

```bash
pip3 install -r requirements.txt
```

As your project grows - you may need to update your environment for a variety of reasons. For example, one of your project's dependencies has
just released a new version (dependency version number update), you need an additional package for data analysis
(adding a new dependency) or you have found a better package and no longer need the older package (adding a new and
removing an old dependency). What you need to do in this case (apart from installing the new and removing the
packages that are no longer needed from your virtual environment) is update the contents of the `requirements.txt` file
accordingly by re-issuing `pip freeze` command and propagate the updated `requirements.txt` file to your collaborators
via your code sharing platform (e.g. GitHub).

> <tip-title>Official Documentation</tip-title>
> For a full list of options and commands, consult the [official `venv` documentation](https://docs.python.org/3/library/venv.html)
> and the [Installing Python Modules with `pip` guide](https://docs.python.org/3/installing/index.html#installing-index). Also check out the guide ["Installing packages using `pip` and virtual environments"](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-packages-using-pip-and-virtual-environments).
{: .tip}

## Running Python Scripts From Command Line
Congratulations! Your environment is now activated and set up to run your script
from the command line.


topic: data-science
tutorial: python-glob
title: Python - Globbing
difficulty: intermediate
time_estimation: 15M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: b92d9b7742f1b580a74dc6d4092f134d
last_modified: 2026-05-15T22:28:11.102992
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-glob/tutorial.html

content:
Globbing is the term used in computer science when we have a bunch of files and we want to list all of them matching some pattern.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Setup

We'll start by creating some files for use in the rest of this tutorial

```python
import os
import subprocess

dirs = ['a', 'a/b', 'c', 'c/e', 'd', '.']
files = ['a.txt', 'a.csv', 'b.csv', 'b.txt', 'e.glm']

for d in dirs:
    # Create some directories
    os.makedirs(d, exist_ok=True)
    # Create some files
    for f in files:
        subprocess.check_output(['touch', os.path.join(d, f)])
```

Now we should have a pretty full folder!

# Finding Files

We can use the glob module to find files:

```python
import glob
print(glob.glob('*.csv'))
print(glob.glob('*.txt'))
```

Here we use an asterisk (`*`) as a wildcard, it matches any bit of text (but not into folders!) to all matching files. Here we list all matching `csv` or `txt` files. This is great to find files matching a pattern.

We can also use asterisks anywhere in the glob, it doesn't just have to be the filename portion:

```python
print(glob.glob('a*'))
```

Here we even see a third entry: the directory.

# Finding files in directories

Until now we've found only files in a single top level directory, but what if we wanted to find files in subdirectories?

Only need a single directory? Just include that!

```python
print(glob.glob('a/*.csv'))
```

But if you need more levels, or want to look in *all* folders, then you need the double wildcard! With two asterisks `**` we can search recursively through directories for files:

```python
print(glob.glob('**/a.csv'))
```

# Exercise

> <question-title>Where in the world is the CSV?</question-title>
>
> 1. How would you find all `.csv` files?
> 2. How would you find all `.txt` files?
> 3. How would you find all files starting with the letter 'e'?
>
> > <solution-title></solution-title>
> >
> > 1. `glob.glob('**/*.csv')`
> > 2. `glob.glob('**/*.txt')`
> > 3. `glob.glob('**/e*')`
> {: .solution}
{: .question}

```python
# Try things out here!
```

# Pitfalls

Some analyses (especially simultaions) can be dependent on data input order or data sorting. This was recently seen in {% cite Bhandari_Neupane_2019 %} where the data files used were sorted one way on Windows, and another on Linux, resulting in different results for the same code and the same datasets! Yikes!

If you know your analyses are dependent on file ordering, then you can use `sorted()` to make sure the data is provided in a uniform way every time.

```python
print(sorted(glob.glob('**/a.csv')))
```

If you're not sure if your results will be dependent, you can try sorting anyway. Or better yet, randomising the list of inputs to make sure your code behaves properly in any scenario.


topic: data-science
tutorial: python-basics
title: Introduction to Python
difficulty: introductory
time_estimation: 5H
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 38ba164a33052724b5e4f6b854b231b2
last_modified: 2026-05-15T22:28:11.101465
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-basics/tutorial.html

content:
In this lesson, we will be using Python 3 with some of its most popular scientific libraries. We will be using JupyterNotebook, a Python interpreter that comes with everything we need for the lesson.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which is licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Overview

Python was developed by Guido van Rossum. Guido van Rossum started implementing Python in 1989. Python is a very simple programming language so even if you are new to programming, you can learn python without facing any issues.

Features of Python language:
1. **Readable**: Python is a very readable language.

2. **Easy to Learn**: Learning python is easy as this is a expressive and high level programming language, which means it is easy to understand the language and thus easy to learn.

3. **Cross platform**: Python is available and can run on various operating systems such as Mac, Windows, Linux, Unix etc. This makes it a cross platform and portable language.

4. **Open Source**: Python is a open source programming language.

5. **Large standard library**: Python comes with a large standard library that has some handy codes and functions which we can use while writing code in Python.

6. **Free**: Python is free to download and use. This means you can download it for free and use it in your application.

7. **Supports exception handling**: Python supports exception handling which means we can write less error prone code and can test various scenarios that can cause an exception later on.

8. **Automatic memory management**: Python supports automatic memory management which means the memory is cleared and freed automatically. You do not have to bother clearing the memory.


> <tip-title>Why Jupyter?</tip-title>
>
> JupyterLab is a User Interface including notebooks. A user can open several notebooks or files as tabs in the same window, like an IDE. JupyterNotebook is a web-based interactive computational environment for creating Jupyter notebook documents. It supports several languages like Python (IPython), Julia, R etc. and is largely used for data analysis, data visualization and further interactive, exploratory computing.
>
> JupyterNotebook has several advantages:
>
> - You can easily type, edit, and copy and paste blocks of code.
> - Tab complete allows you to easily access the names of things you are using and learn more about them.
> - It allows you to annotate your code with links, different sized text, bullets, etc. to make it more accessible to you and your collaborators.
> - It allows you to display figures next to the code that produces them to tell a complete story of the analysis.
>
> Each notebook contains one or more cells that contain code, text, or images. Each notebook can be exported (File, Export as, Executable script) as Python script that can be run from the command line.
>
> We will be using JupyterNotebook in Galaxy and as a result you need to frequently save the notebook in the workspace. This is both for good practice and to protect you in case you accidentally close the browser. Your environment will still run, so it will contain the last saved notebook you have.  Furthermore, you need to download a notebook, before you delete or close a notebook in your history or you will lose it.
{: .tip}


# Python Fundamentals

## Variables

Any Python interpreter can be used as a calculator:

```python
3 + 5 * 4
```

This is great but not very interesting. To do anything useful with data, we need to assign its value to a variable. In Python, we can assign a value to a variable, using the equals sign ```=```. For example, we can track the weight of a patient who weighs 60 kilograms by assigning the value 60 to a variable ```weight_kg```:

```python
weight_kg = 60
```

From now on, whenever we use `weight_kg`, Python will substitute the value we assigned to it.

In Python, variable names:

- can include letters, digits, and underscores
- cannot start with a digit
- are case sensitive

This means that, for example:

- `weight0` is a valid variable name, whereas `0weight` is not
- `weight` and `Weight` are different variables



### Types of data

Python knows various types of data. Three common ones are:

- integer numbers
- floating point numbers, and
- strings.

In the example above, variable `weight_kg` has an integer value of `60`. If we want to more precisely track the weight of our patient, we can use a floating point value by executing:

```python
weight_kg = 60.3
```

To create a string, we add single or double quotes around some text. To identify and track a patient throughout our study, we can assign each person a unique identifier by storing it in a string:

```python
patient_id = '001'
```

### Using Variables in Python

Once we have data stored with variable names, we can make use of it in calculations. We may want to store our patient’s weight in pounds as well as kilograms:

```python
weight_lb = 2.2 * weight_kg
```

We might decide to add a prefix to our patient identifier:

```python
patient_id = 'inflam_' + patient_id
```

### Variables Persist Between Cells
Be aware that it is the order of execution of cells that is important in a Jupyter notebook, not the order in which they appear. Python will remember all the code that was run previously, including any variables you have defined, irrespective of the order in the notebook. Therefore if you define variables lower down the notebook and then (re)run cells further up, those defined further down will still be present. As an example, create two cells with the following content, in this order:

```python
print(myval)
myval = 1
```
If you execute this in order, the first cell will give an error. However, if you run the first cell after the second cell, it will print out 1. To prevent confusion, it can be helpful to use the Kernel -> Restart & Run All option which clears the interpreter and runs everything from a clean slate going top to bottom.


> <question-title>Variables and values</question-title>
>
> What values do the variables `mass` and `age` have after each of the following statements? Test your answer by executing the lines.
> ```
> mass = 47.5
> age = 122
> mass = mass * 2.0
> age = age - 20
> ```
>
> > <solution-title></solution-title>
> >
> > `mass` holds a value of 47.5, `age` does not exist
> > `mass` still holds a value of 47.5, `age` holds a value of 122
> > `mass` now has a value of 95.0, `age`'s value is still 122
> > `mass` still has a value of 95.0, `age` now holds 102
> >
> {: .solution}
{: .question}


> <question-title>Variables</question-title>
>
> Python allows you to assign multiple values to multiple variables in one line by separating the variables and values with commas. What does the following program print out?
> ```
> first, second = 'Grace', 'Hopper'
> third, fourth = second, first
> print(third, fourth)
> ```
>
> > <solution-title></solution-title>
> >
> > Hopper Grace
> >
> {: .solution}
{: .question}


> <question-title>Variables and data types</question-title>
>
> What are the data types of the following variables?
>
> ```
> planet = 'Earth'
> apples = 5
> distance = 10.5
> ```
>
> > <solution-title></solution-title>
> >
> > ```
> > type(planet)
> > type(apples)
> > type(distance)
> > ```
> >
> {: .solution}
{: .question}



## Lists

Lists are built into the language so we do not have to load a library to use them. We create a list by putting values inside square brackets and separating the values with commas:

```python
odds = [1, 3, 5, 7]
print('odds are:', odds)
```


We can access elements of a list using indices – numbered positions of elements in the list. These positions are numbered starting at 0, so the first element has an index of 0.

```python
print('first element:', odds[0])
print('last element:', odds[3])
print('"-1" element:', odds[-1])
```

Yes, we can use negative numbers as indices in Python. When we do so, the index `-1` gives us the last element in the list, `-2` the second to last, and so on. Because of this, `odds[3]` and `odds[-1]` point to the same element here.

There is one important difference between lists and strings: we can change the values in a list, but we cannot change individual characters in a string. For example:

```python
names = ['Curie', 'Darwing', 'Turing']  # typo in Darwin's name
print('names is originally:', names)
names[1] = 'Darwin'  # correct the name
print('final value of names:', names)
```


works, but the following does not:

```python
name = 'Darwin'
name[0] = 'd'
```

Lists in Python can contain elements of different types. Example:

```python
sample_ages = [10, 12.5, 'Unknown']
```

There are many ways to change the contents of lists besides assigning new values to individual elements:

We can append new elements to a list

```python
odds.append(11)
print('odds after adding a value:', odds)
```

We can pop an element off the end of a list

```python
removed_element = odds.pop(0)
print('odds after removing the first element:', odds)
print('removed_element:', removed_element)
```

Or we can reverse the list

```python
odds.reverse()
print('odds after reversing:', odds)
```


### Mutable vs Immutable data
Data which can be modified in place is called mutable, while data which cannot be modified is called immutable. Strings and numbers are immutable. This does not mean that variables with string or number values are constants, but when we want to change the value of a string or number variable, we can only replace the old value with a completely new value.

Lists and arrays, on the other hand, are mutable: we can modify them after they have been created. We can change individual elements, append new elements, or reorder the whole list. For some operations, like sorting, we can choose whether to use a function that modifies the data in-place or a function that returns a modified copy and leaves the original unchanged.

Be careful when modifying data in-place. If two variables refer to the same list, and you modify the list value, it will change for both variables!

```python
salsa = ['peppers', 'onions', 'cilantro', 'tomatoes']
my_salsa = salsa        # <-- my_salsa and salsa point to the *same* list data in memory
salsa[0] = 'hot peppers'
print('Ingredients in my salsa:', my_salsa)
```

If you want variables with mutable values to be independent, you must make a copy of the value when you assign it.

```python
salsa = ['peppers', 'onions', 'cilantro', 'tomatoes']
my_salsa = salsa.copy()        # <-- makes a *copy* of the list
salsa[0] = 'hot peppers'
print('Ingredients in my salsa:', my_salsa)
```


Because of pitfalls like this, code which modifies data in place can be more difficult to understand. However, it is often far more efficient to modify a large data structure in place than to create a modified copy for every small change. You should consider both of these aspects when writing your code.

### Nested lists
Since a list can contain any Python variables, it can even contain other lists.

For example, we could represent the products in the shelves of a small grocery shop:

```python
x = [['pepper', 'zucchini', 'onion'],
   ['cabbage', 'lettuce', 'garlic'],
   ['apple', 'pear', 'banana']]
```

Here is an example of how indexing a list of lists `x` works:

```python
print([x[0]])
print(x[0])
print(x[0][0])
```


### Slicing
Subsets of lists can be accessed by specifying ranges of values in brackets. This is commonly referred to as “slicing” the list.

```python
chromosomes = ['X', 'Y', '2', '3', '4']
autosomes = chromosomes[2:5]
print('autosomes:', autosomes)
last = chromosomes[-1]
print('last:', last)
```


> <question-title>Negative indices</question-title>
>
> Use slicing to access only the last four characters of a string or entries of a list.
>
> ```
> string_for_slicing = 'Observation date: 02-Feb-2013'
> list_for_slicing = [['fluorine', 'F'], ['chlorine', 'Cl'], ['bromine', 'Br'], ['iodine', 'I'], ['astatine', 'At']]
> ```
> Your output should be:
> '2013'
> [['chlorine', 'Cl'], ['bromine', 'Br'], ['iodine', 'I'], ['astatine', 'At']]
>
> Would your solution work regardless of whether you knew beforehand the length of the string or list (e.g. if you wanted to apply the solution to a set of lists of different lengths)? If not, try to change your approach to make it more robust.
> Hint: Remember that indices can be negative as well as positive.
>
> > <solution-title></solution-title>
> >
> > Use negative indices to count elements from the end of a container (such as list or string):
> > ```
> > string_for_slicing[-4:]
> > list_for_slicing[-4:]
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Slicing</question-title>
>
> So far we’ve seen how to use slicing to take single blocks of successive entries from a sequence. But what if we want to take a subset of entries that aren’t next to each other in the sequence?
>
> You can achieve this by providing a third argument to the range within the brackets, called the step size. The example below shows how you can take every third entry in a list:
>
> ```
> primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37]
> subset = primes[0:12:3]
> print('subset', subset)
> ```
>
> Notice that the slice taken begins with the first entry in the range, followed by entries taken at equally-spaced intervals (the steps) thereafter. What if you wanted to begin the subset with the third entry? Use the previous example to write your solution that gives the following output.
>
> > <solution-title></solution-title>
> >
> > You would need to specify that as the starting point of the sliced range:
> >
> > ```
> > subset = primes[2:12:3]
> > print('subset', subset)
> > ```
> >
> {: .solution}
{: .question}



## Strings

### Use an index to get a single character from a string.

The characters (individual letters, numbers,and so on) in a string are ordered. For example, the string 'AB' is not the same as 'BA'. Because of this ordering, we can treat the string as a list of characters. Each position in the string (first, second, etc.) is given a number. This number is called an index or sometimes a subscript. Indices are numbered from 0.You can use the position’s index in square brackets to get the character at that position.

```python
atom_name = 'helium'
print(atom_name[0])
```

### Use a slice to get a substring.
A part of a string is called a substring. A substring can be as short as a single character. An item in a list is called an element. Whenever we treat a string as if it were a list, the string’s elements are its individual characters. A slice is a part of a string (or, more generally, any list-like thing). We take a slice by using [start:stop], where start is replaced with the index of the first element we want and stop is replaced with the index of the element just after the last element we want. Mathematically, you might say that a slice selects [start:stop). The difference between stop and start is the slice’s length. Taking a slice does not change the contents of the original string. Instead, the slice is a copy of part of the original string.

```python
atom_name = 'sodium'
print(atom_name[0:3])
```

You can use the built-in function `len` to find the length of a string.

```python
print(len('helium'))
```

Nested functions are evaluated from the inside out, like in mathematics.

> <question-title>Slicing</question-title>
>
> Use what you have learnt in this tutorial to answer the following questions:
> a. What does the following program print?
> ```
> atom_name = 'carbon'
> print('atom_name[1:3] is:', atom_name[1:3])
> ```
> b. What does `thing[low:high]` do?
> c. What does `thing[low:]` (without a value after the colon) do?
> d. What does `thing[:high]` (without a value before the colon) do?
> e. What does `thing[:]` (just a colon) do?
> f. What does `thing[number:some-negative-number]` do?
> g. What happens when you choose a high value which is out of range? (i.e., try `atom_name[0:15]`)
>
> > <solution-title></solution-title>
> >
> > a. `atom_name[1:3] is: ar`
> > b. `thing[low:high]` returns a slice from low to the value before high
> > c. `thing[low:]` returns a slice from low all the way to the end of thing
> > d. `thing[:high]` returns a slice from the beginning of thing to the value before high
> > e. `thing[:]` returns all of thing
> > f. `thing[number:some-negative-number]` returns a slice from number to some-negative-number values from the end of thing
> > g. If a part of the slice is out of range, the operation does not fail. `atom_name[0:15]` gives the same result as `atom_name[0:]`.
> >
> {: .solution}
{: .question}



### You can use the “+” and “*” operators on strings/lists.

“Adding” character strings concatenates them.

```python
full_name = 'Ahmed' + ' ' + 'Walsh'
print(full_name)
```


Multiplying a character string by an integer "N" creates a new string that consists of that character string repeated N times.

Since multiplication is repeated addition.

```python
separator = '=' * 10
print(separator)
```


The same rules apply for lists. Consider the following example:

```python
counts = [2, 4, 6, 8, 10]
repeats = counts * 2
print(repeats)
```


It’s equivalent to:

```python
counts + counts
```

Strings have a length (but numbers don’t).The built-in function `len` counts the number of characters in a string.

```python
print(len(full_name))
```


But numbers don’t have a length (not even zero). For example, the following command returns an error message.

```python
print(len(52))
```


## Type conversion

Python converts automatically integers to floats, when needed, but you must convert numbers to strings or vice versa when operating on them.
It is not allowed to add numbers and strings. For example `print(1 + '2')` is ambiguous: should `1 + '2'` be `3` or `'12'`?

Some types can be converted to other types by using the type name as a function.

```python
print(1 + int('2'))
print(str(1) + '2')
```

> <question-title>Data types</question-title>
>
> What type of value is 3.25 + 4?
>
> > <solution-title></solution-title>
> >
> > It is a float: integers are automatically converted to floats as necessary.
> {: .solution}
{: .question}

> <question-title>Division</question-title>
>
> In Python 3, the `//` operator performs integer (whole-number) floor division, the `/` operator performs floating-point division, and the `%` (or *modulo*) operator calculates and returns the remainder from integer division:
>
> > <code-in-title>Python</code-in-title>
> > ```
> > print('5 // 3:', 5 // 3)
> > print('5 / 3:', 5 / 3)
> > print('5 % 3:', 5 % 3)
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > 5 // 3: 1
> > 5 / 3: 1.6666666666666667
> > 5 % 3: 2
> > ```
> {: .code-out}
>
> If `num_subjects` is the number of subjects taking part in a study, and `num_per_survey` is the number that can take part in a single survey, write an expression that calculates the number of surveys needed to reach everyone once.
>
> > <solution-title></solution-title>
> >
> > We want the minimum number of surveys that reaches everyone once, which is the rounded up value of `num_subjects/ num_per_survey`. This is equivalent to performing a floor division with `//` and adding 1. Before the division we need to subtract 1 from the number of subjects to deal with the case where `num_subjects` is evenly divisible by `num_per_survey`.
> >
> > ```
> > num_subjects = 600
> > num_per_survey = 42
> > num_surveys = (num_subjects - 1) // num_per_survey + 1
> >
> > print(num_subjects, 'subjects,', num_per_survey, 'per survey:', num_surveys)
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Typecasting</question-title>
>
> Where reasonable, `float()` will convert a string to a floating point number, and `int()` will convert a floating point number to an integer:
>
> > <code-in-title>Python</code-in-title>
> > ```
> > print("string to float:", float("3.4"))
> > print("float to int:", int(3.4))
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > string to float: 3.4
> > float to int: 3
> > ```
> {: .code-out}
>
> If the conversion doesn’t make sense, however, an error message will occur.
> Given this information, what do you expect the following program to do? What does it actually do? Why do you think it does that?
> ```
> print("fractional string to int:", int("3.4"))
> ```
>
> > <solution-title></solution-title>
> >
> > Python 3 throws an error. If you ask Python to perform two consecutive typecasts, you must convert it explicitly in code.
> >
> > ```
> > int(float("3.4"))
> > ```
> >
> {: .solution}
{: .question}

> <question-title>Typecasting</question-title>
>
> Which of the following will return the floating point number `2.0`? Note: there may be more than one right answer.
> ```
> first = 1.0
> second = "1"
> third = "1.1"
> ```
>
> 1. `first + float(second)`
> 2. `float(second) + float(third)`
> 3. `first + int(third)`
> 4. `first + int(float(third))`
> 5. `int(first) + int(float(third))`
> 6. `2.0 * second`
>
> > <solution-title></solution-title>
> >
> > Answer: 1 and 4
> {: .solution}
{: .question}


> <question-title>Imaginary numbers</question-title>
>
> Python provides complex numbers, which are written as `1.0+2.0j`. If `val` is a complex number, its real and imaginary parts can be accessed using dot notation as `val.real` and `val.imag`.
>
> ```
> complex = 6 + 2j
> print(complex.real)
> print(complex.imag)
> ```
>
> Output:
> 6.0
> 2.0
>
> 1. Why do you think Python uses `j` instead of i for the imaginary part?
> 2. What do you expect `1+2j + 3` to produce?
> 3. What do you expect `4j` to be? What about `4 j` or `4 + j`?
>
> > <solution-title></solution-title>
> >
> >1. Standard mathematics treatments typically use i to denote an imaginary number. However, from media reports it was an early convention established from electrical engineering that now presents a technically expensive area to change.
> >2. `(4+2j)`
> >3. `4j` and Syntax Error: invalid syntax. In the latter cases, `j` is considered a variable and the statement depends on if `j` is defined and if so, its assigned value.
> >
> {: .solution}
{: .question}


## Built-in Python functions

To carry out common tasks with data and variables in Python, the language provides us with several built-in functions. To display information to the screen, we use the `print` function:

```python
print(weight_lb)
print(patient_id)
```


When we want to make use of a function, referred to as calling the function, we follow its name by parentheses. The parentheses are important: if you leave them off, the function doesn’t actually run! Sometimes you will include values or variables inside the parentheses for the function to use. In the case of `print`, we use the parentheses to tell the function what value we want to display. `print` automatically puts a single space between outputs to separate them and wraps around to a new line at the end.

We can display multiple things at once using only one `print` call:

```python
print(patient_id, 'weight in kilograms:', weight_kg)
```


We can also call a function inside of another function call. For example, Python has a built-in function called `type` that tells you a value’s data type:

```python
print(type(60.3))
print(type(patient_id))
```


Moreover, we can do arithmetic with variables right inside the `print` function:

```python
print('weight in pounds:', 2.2 * weight_kg)
```

The above command, however, did not change the value of `weight_kg`:

```python
print(weight_kg)
```

To change the value of the `weight_kg` variable, we have to assign `weight_kg` a new value using the equals `=` sign:

```python
weight_kg = 65.0
print('weight in kilograms is now:', weight_kg)
```

A function may take zero or more arguments.  An *argument* is a value passed into a function. You must always use parentheses, even if they’re empty, so that Python knows a function is being called.

Every function call produces some result. If the function doesn’t have a useful result to return, it usually returns the special value `None`. `None` is a Python object that stands in anytime there is no value.

```python
result = print('example')
print('result of print is', result)
```

Commonly-used built-in functions include `max`, `min`, and `round`. `max` and `min` work on character strings as well as numbers. From “larger” and “smaller”, they use the order: (0-9, A-Z, a-z) to compare letters.
```python
print(max(1, 2, 3))
print(min('a', 'A', '0'))
```

Functions may have default values for some arguments. `round` will round off a floating-point number. By default, it rounds to zero decimal places.
```python
round(3.712)
```

We can specify the number of decimal places we want.
```python
round(3.712, 1)
```


### Functions attached to objects are called methods

Methods have parentheses like functions, but come after the variable. Some methods are used for internal Python operations, and are marked with double underlines.

```python
my_string = 'Hello world!'  # creation of a string object

print(len(my_string))       # the len function takes a string as an argument and returns the length of the string

print(my_string.swapcase()) # calling the swapcase method on the my_string object

print(my_string.__len__())  # calling the internal __len__ method on the my_string object, used by len(my_string)
```


You might even see them chained together. They operate left to right.

```python
print(my_string.isupper())          # Not all the letters are uppercase
print(my_string.upper())            # This capitalizes all the letters

print(my_string.upper().isupper())  # Now all the letters are uppercase
```

### Use the built-in function `help` to get help for a function.

Every built-in function has online documentation.

```python
help(round)
```


## Conditionals

An if statement (more properly called a conditional statement) controls whether some block of code is executed or not.
The first line opens with if and ends with a colon and the block of code to be executed is indented. An example is showed below.

```python
num = 37
if num > 100:
    print('greater')
else:
    print('not greater')
print('done')
```

If the expression that follows the if statement is true, the body of the `if` (i.e., the set of lines indented underneath it) is executed, and `“greater”` is printed. If it is false, the body of the `else` is executed instead, and `“not greater”` is printed. Only one or the other is ever executed before continuing on with program execution to print `“done”`:

![A flow chart diagram is shown with a decision box of number > 100. For true it branches to print 'greater'. For false it branches to print 'not greater'. Both branches re-connect to print done.](../../images/python-basics/Figure5_Conditionals.png)

Conditional statements don’t have to include an `else`.  If there isn’t one, Python simply does nothing if the expression is false:

```python
num = 53
print('before conditional...')
if num > 100:
    print(num, 'is greater than 100')
print('...after conditional')
```

We can also chain several expressions together using `elif`, which is short for “else if”. The following Python code uses `elif` to print the sign of a number.

```python
num = -3

if num > 0:
    print(num, 'is positive')
elif num == 0:
    print(num, 'is zero')
else:
    print(num, 'is negative')
```


### Comparison operators in Python
The operators used for comparing values in conditionals are the following:

- `>` : greater than
- `<` : less than
- `==` : equal to
- `!=` : does not equal
- `>=` : greater than or equal to
- `<=` : less than or equal to


### Logical operators in Python
We can also combine expressions using `and` and `or`. `and` is only true if both parts are true:

```python
if (1 > 0) and (-1 >= 0):
    print('both parts are true')
else:
    print('at least one part is false')
```


while `or` is true if at least one part is true:

```python
if (1 < 0) or (1 >= 0):
    print('at least one test is true')
```


### True and False
`True` and `False` are special words in Python called booleans, which represent truth values. A statement such as `1 < 0` returns the value `False`, while `-1 < 0` returns the value `True`.


> <question-title>Conditionals</question-title>
>
> What does this program print?
>
> ```
> pressure = 71.9
> if pressure > 50.0:
>     pressure = 25.0
> elif pressure <= 50.0:
>     pressure = 0.0
> print(pressure)
> ```
>
> > <solution-title></solution-title>
> >
> > 25
> {: .solution}
{: .question}


> <question-title>Conditionals</question-title>
>
> Write some conditions that print `True` if the variable `a` is within 10% of the variable `b` and `False` otherwise. Compare your implementation with your partner’s: do you get the same answer for all possible pairs of numbers?
> Hint: There is a built-in function `abs()` that returns the absolute value of a number.
>
> > <solution-title></solution-title>
> >
> > ```
> > a = 5
> > b = 5.1
> > if abs(a - b) <= 0.1 * abs(b):
> >    print('True')
> > else:
> >    print('False')
> > ```
> >
> {: .solution}
{: .question}


## For loops

Doing calculations on the values in a list one by one is very time consuming.

```python
odds = [1, 3, 5, 7, 9, 11]
print(odds[0])
print(odds[1])
print(odds[2])
print(odds[3])
print(odds[4])
print(odds[5])

```


A for loop tells Python to execute some statements once for each value in a list, a character string, or some other collection.
“for each thing in this group, do these operations”. The for loop equivalent to the previous code is:

```python
for num in odds:
    print(num)
```
The improved version uses a for loop to repeat an operation — in this case, printing — once for each thing in a sequence. The general form of a loop is:

```python
for variable in collection:
    # do things using variable, such as print
```

Using the odds example above, the loop might look like this:
![A cartoon schematic shows a  small array named odds with the odd numbers from 1 to 11. A for loop is written "for num in odds" and for each of these items in the array print(num) happens, and the output is shown below, the odd numbers printed out.](../../images/python-basics/Figure6_For_loop.png)

where each number (`num`) in the variable odds is looped through and printed one number after another. The other numbers in the diagram denote which loop cycle the number was printed in (1 being the first loop cycle, and 6 being the final loop cycle).

We can call the loop variable anything we like, but there must be a colon at the end of the line starting the loop, and we must indent anything we want to run inside the loop. Unlike many other languages, there is no command to signify the end of the loop body (e.g. end for); what is indented after the for statement belongs to the loop. Python uses indentation to show nesting. Any consistent indentation is legal, but almost everyone uses four spaces.

When looping through a list, the position index and corresponding value can be retrieved at the same time using the `enumerate()` function.

```python
for i, v in enumerate(['tic', 'tac', 'toe']):
... print(i, v)
```


To loop over two or more lists at the same time, the entries can be paired with the `zip()` function.

```python
questions = ['name', 'quest', 'favorite color']
answers = ['lancelot', 'the holy grail', 'blue']
for q, a in zip(questions, answers):
   print('What is your {0}?  It is {1}.'.format(q, a))
```


We can choose any name we want for variables. It is a good idea to choose variable names that are meaningful, otherwise it would be more difficult to understand what the loop is doing.

Here’s another loop that repeatedly updates a variable:

```python
length = 0
names = ['Curie', 'Darwin', 'Turing']
for value in names:
    length = length + 1
print('There are', length, 'names in the list.')
print('After the loop, name is', name)
```

It is worth tracing the execution of this little program step by step. Since there are three names in `names`, the statement on line 4 will be executed three times. The first time around, `length` is `0` (the value assigned to it on line 1) and `value` is `Curie`. The statement adds 1 to the old value of `length`, producing 1, and updates `length` to refer to that new value. The next time around, `value` is `Darwin` and `length` is `1`, so `length` is updated to be `2`. After one more update, `length` is `3`; since there is nothing left in `names` for Python to process, the loop finishes and the print function on line 5 tells us our final answer.

Note that a loop variable is a variable that is being used to record progress in a loop. It still exists after the loop is over, stores the value assigned to it last, and we can re-use variables previously defined as loop variables as well.


> <question-title>range of numbers</question-title>
>
> Python has a built-in function called `range()` that generates a sequence of numbers. `range` can accept 1, 2, or 3 parameters.
> If one parameter is given, `range` generates a sequence of that length, starting at zero and incrementing by 1. For example, `range(3)` produces the numbers `0, 1, 2`.
> If two parameters are given, `range` starts at the first and ends just before the second, incrementing by one. For example, `range(2, 5)` produces `2, 3, 4`.
> If `range` is given 3 parameters, it starts at the first one, ends just before the second one, and increments by the third one. For example, `range(3, 10, 2)` produces `3, 5, 7, 9`.
> Using `range`, write a loop that uses `range` to print the first 3 natural numbers:
> ```
> 1
> 2
> 3
> ```
>
> > <solution-title></solution-title>
> >
> >  ```
> >  for i in range(1, 4):
> >     print(i)
> >  ```
> {: .solution}
{: .question}


> <question-title>Number of iterations</question-title>
>
> Given the following loop:
> ```
> word = 'oxygen'
> for char in word:
>    print(char)
> ```
> How many times is the for loop executed?
>
> > <solution-title></solution-title>
> >
> > The body of the loop is executed 6 times.
> {: .solution}
{: .question}


> <question-title>Exponentiation</question-title>
>
> Exponentiation is built into Python:
>
> ```
> print(5 ** 3)
> ```
>
> Output:
> 125
>
> Write a loop that calculates the same result as `5 ** 3` using multiplication (and without exponentiation).
>
> > <solution-title></solution-title>
> >
> > ```
> > result = 1
> > for number in range(0, 3):
> >     result = result * 5
> > print(result)
> > ```
> {: .solution}
{: .question}


> <question-title>Iterations over a list</question-title>
>
> Write a loop that calculates the sum of elements in a list by adding each element and printing the final value, so `[124, 402, 36]` prints `562`
>
> > <solution-title></solution-title>
> >
> > ```
> > numbers = [124, 402, 36]
> > summed = 0
> > for num in numbers:
> >     summed = summed + num
> > print(summed)
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Polynomial</question-title>
>
> Suppose you have encoded a polynomial as a list of coefficients in the following way: the first element is the constant term, the second element is the coefficient of the linear term, the third is the coefficient of the quadratic term, etc.
>
> ```
> x = 5
> coefs = [2, 4, 3]
> y = coefs[0] * x**0 + coefs[1] * x**1 + coefs[2] * x**2
> print(y)
> ```
>
> Output:
> 97
>
> Write a loop using `enumerate(coefs)` which computes the value `y` of any polynomial, given `x` and `coefs`.
>
> > <solution-title></solution-title>
> >
> > ```
> > y = 0
> > for idx, coef in enumerate(coefs):
> >     y = y + coef * x**idx
> > ```
> >
> {: .solution}
{: .question}


> <question-title>For loops and conditionals</question-title>
>
> Fill in the blanks so that this program creates a new list containing zeroes where the original list’s values were negative and ones where the original list’s values were positive.
> ```
> original = [-1.5, 0.2, 0.4, 0.0, -1.3, 0.4]
> result = ____
> for value in original:
>     if ____:
>         result.append(0)
>     else:
>         ____
>     print(result)
> ```
>
> Output:
> [0, 1, 1, 1, 0, 1]
>
> > <solution-title></solution-title>
> >
> > ```
> > original = [-1.5, 0.2, 0.4, 0.0, -1.3, 0.4]
> > result = []
> > for value in original:
> >     if value < 0:
> >         result.append(0)
> >     else:
> >         result.append(1)
> >     print(result)
> > ```
> >
> {: .solution}
{: .question}



## While loops
With the while loop we can execute a set of statements as long as an expression is true. The following example prints `i` as long as `i` is less than 6:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  print(odds[i])
  i = i + 1
```


Remember to increment `i`, or else the loop will continue forever. The while loop requires relevant variables to be ready, in this example we need to define an indexing variable, `i`, which we set to 0.

With the `break` statement we can stop the loop even if the while condition is true:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  print(odds[i])
  if odds[i] == 3:
    break
  i = i + 1
```


With the `continue` statement we can stop the current iteration, and continue with the next:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  i = i + 1
  if odds[i] == 3:
    continue
  print(odds[i])

```


With the `else` statement we can run a block of code once when the condition no longer is true:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  print(odds[i])
  i = i + 1
else:
  print("condition is no longer True")
```



## Create Functions
Human beings can only keep a few items in working memory at a time. Breaking down larger/more complicated pieces of code in functions helps in understanding and using it. A function can be re-used. Write one time, use many times.

```python
def fahr_to_celsius(temp):
    return ((temp - 32) * (5/9))
```

![The above function fahr to celsius is shown except annotated. def is labelled "def statement", fahr_to_celsius is noted as the function name. Inside parentheses is temp and an arrow shows it is called parameter names. The next line which is indented is annotated as the function body which has a return statement and the calculation from above.](../../images/python-basics/Figure7_Functions.png)

The function definition opens with the keyword `def` followed by the name of the function `fahr_to_celsius` and a parenthesized list of parameter names `temp`. The body of the function — the statements that are executed when it runs — is indented below the definition line. The body concludes with a `return` keyword followed by the return value.

When we call the function, the values we pass to it are assigned to those variables so that we can use them inside the function. Inside the function, we use a return statement to send a result back to whoever asked for it.

Let’s try running our function.

```python
fahr_to_celsius(32)
print('freezing point of water:', fahr_to_celsius(32), 'C')
print('boiling point of water:', fahr_to_celsius(212), 'C')
```


We’ve successfully called the function that we defined, and we have access to the value that we returned.

Now that we’ve seen how to turn Fahrenheit into Celsius, we can also write the function to turn Celsius into Kelvin:

```python
def celsius_to_kelvin(temp_c):
    return temp_c + 273.15

print('freezing point of water in Kelvin:', celsius_to_kelvin(0.))
```


What about converting Fahrenheit to Kelvin? We could write out the formula, but we don’t need to. Instead, we can compose the two functions we have already created:

```python
def fahr_to_kelvin(temp_f):
    temp_c = fahr_to_celsius(temp_f)
    temp_k = celsius_to_kelvin(temp_c)
    return temp_k

print('boiling point of water in Kelvin:', fahr_to_kelvin(212.0))
```


This is our first taste of how larger programs are built: we define basic operations, then combine them in ever-larger chunks to get the effect we want. Real-life functions will usually be larger than the ones shown here — typically half a dozen to a few dozen lines — but they shouldn’t ever be much longer than that, or the next person who reads it won’t be able to understand what’s going on.

### Variable Scope
In composing our temperature conversion functions, we created variables inside of those functions, `temp`, `temp_c`, `temp_f`, and `temp_k`. We refer to these variables as local variables because they no longer exist once the function is done executing. If we try to access their values outside of the function, we will encounter an error:

```python
print('Again, temperature in Kelvin was:', temp_k)
```

If you want to reuse the temperature in Kelvin after you have calculated it with fahr_to_kelvin, you can store the result of the function call in a variable:

```python
temp_kelvin = fahr_to_kelvin(212.0)
print('temperature in Kelvin was:', temp_kelvin)
```


### Defining Default parameters
If we usually want a function to work one way, but occasionally need it to do something else, we can allow people to pass a parameter when they need to but provide a default to make the normal case easier. The example below shows how Python matches values to parameters:

```python
def display(a=1, b=2, c=3):
    print('a:', a, 'b:', b, 'c:', c)

print('no parameters:')
display()
print('one parameter:')
display(55)
print('two parameters:')
display(55, 66)
```


As this example shows, parameters are matched up from left to right, and any that haven’t been given a value explicitly get their default value. We can override this behavior by naming the value as we pass it in:

```python
print('only setting the value of c')
display(c=77)
```

> <question-title>Variable scope</question-title>
>
> What does the following piece of code display when run — and why?
>
> ```
> f = 0
> k = 0
>
> def f2k(f):
>     k = ((f - 32) * (5.0 / 9.0)) + 273.15
>     return k
>
> print(f2k(8))
> print(f2k(41))
> print(f2k(32))
> print(k)
> ```
>
> > <solution-title></solution-title>
> >
> > Output:
> > 259.81666666666666
> > 278.15
> > 273.15
> > 0
> >
> > `k` is 0 because the `k` inside the function `f2k` doesn’t know about the `k` defined outside the function. When the `f2k` function is called, it creates a local variable `k`. The function does not return any values and does not alter `k` outside of its local copy. Therefore the original value of `k` remains unchanged.
> >
> {: .solution}
{: .question}



## Libraries

A library is a collection of files (called modules) that contains functions for use by other programs. It may also contain data values (e.g., numerical constants) and other things. A library’s contents are supposed to be related, but there’s no way to enforce that. The Python standard library is an extensive suite of modules that comes with Python itself. Many additional libraries are available from PyPI (the Python Package Index).

### Libraries and modules
A library is a collection of modules, but the terms are often used interchangeably, especially since many libraries only consist of a single module, so don’t worry if you mix them.

### A program must import a library module before using it.
You can use `import` to load a library module into a program’s memory, then refer to things from the `module as module_name.thing_name`. Python uses `.` to mean “part of”. For example, using `math`, one of the modules in the standard library:

```python
import math

print('pi is', math.pi)
print('cos(pi) is', math.cos(math.pi))
```


You can use `help` to learn about the contents of a library module. it works just like help for a function.
```python
help(math)
```
You can import specific items from a library module to shorten programs. You can use `from ... import ...` to load only specific items from a library module. Then refer to them directly without library name as prefix.

```python
from math import cos, pi

print('cos(pi) is', cos(pi))
```


You can create an alias for a library module when importing it to shorten programs. Use `import ... as ...` to give a library a short alias while importing it. Then refer to items in the library using that shortened name.

```python
import math as m

print('cos(pi) is', m.cos(m.pi))
```



# Programming style
A consistent coding style helps others (including our future selves) read and understand code more easily. Code is read much more often than it is written, and as the [Zen of Python](https://www.python.org/dev/peps/pep-0020/) states, “Readability counts”. Python proposed a standard style through one of its first Python Enhancement Proposals (PEP), [PEP8](https://www.python.org/dev/peps/pep-0008/).

Some points worth highlighting:

* document your code and ensure that assumptions, internal algorithms, expected inputs, expected outputs, etc., are clear
* use clear, semantically meaningful variable names
* use white-space, not tabs, to indent lines (tabs can cause problems across different text editors, operating systems, and version control systems)

# Python community

Python supports a large and diverse community across academia and industry.

* The [Python 3 documentation](https://docs.python.org/3/) covers the core language and the standard library.
* [PyCon](https://pycon.org/) is the largest annual conference for the Python community.
* [SciPy](https://scipy.org/) is a rich collection of scientific utilities. It is also the name of [a series of annual conferences](https://conference.scipy.org/).
* [Jupyter](https://jupyter.org/) is the home of Project Jupyter.
* [Pandas](https://pandas.pydata.org/) is the home of the Pandas data library.
* [Stack Overflow’s](https://stackoverflow.com/questions/tagged/python?tab=Votes) general Python section can be helpful, as well as the sections on [NumPy](https://stackoverflow.com/questions/tagged/numpy?tab=Votes), [SciPy](https://stackoverflow.com/questions/tagged/scipy?tab=Votes), and [Pandas](https://stackoverflow.com/questions/tagged/pandas?tab=Votes).


topic: data-science
tutorial: python-multiprocessing
title: Python - Multiprocessing
difficulty: advanced
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: c9f92599d9dd26f461427d60eb2936a3
last_modified: 2026-05-15T22:28:11.105612
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-multiprocessing/tutorial.html

content:
This tutorial will introduce you to the basics of Threads and Processes in
Python and how you can use them to parallelise your code.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Threads vs Processes

For many languages, threads can be extremely efficient, as they are rather light weight and don't require many resources to create new threads. "Threads are cheap". However Python has a major limitation with the {GIL}. Only one thread can be executing at once, whether it's the main thread, or one of the others you've spawned. So we look to alternative concurrency mechanisms like processes for sharing the load across multiple CPU cores.

For **threads**, if you are doing "lightweight" work like fetching data from the web where a lot of time is spent waiting for the server to respond, but very little computational work within Python is required, then threads are a great fit!

However if you are doing computationally intensive work (imagine calculating prime numbers or similar complex tasks) then each thread will be contending for CPU time. Python is a single processes and can only have one thread running at a time due to the {GIL}. So it will switch between multiple threads and try and make progress on each, but it cannot execute them truly simultaneously. Here we need to switch to processes.

**Processes** are relatively heavier weight as they essentially start a new python process, before executing the individual function. This is part of the reason for a "Pool", to amortise the expensive cost of setting up processes, before allowing them to do a lot of work quickly. Here each process can be using it's own CPU core fully, and thus make more sense for computationally expensive tasks.


Let's dive straight into an example: here we're using the `multiprocessing` library which uses processes and is relatively simple to understand:

```python
from multiprocessing import Pool
```

Importantly we should define a *pure* function, i.e. a function that only works on the inputs available to it, and doesn't use or affect global state (i.e.  without side effects). (printing is ok.)

```python
def f(x):
    return x * x
```

We spawn a process Pool, with 5 processes And then use the convenient map
function to send inputs multiply to the specified function `f`

```python
with Pool(5) as p:
    print(p.map(f, range(10)))
```

This will create 5 new python processes, and they will begin to consume tasks from the queue as quickly as they can. As soon as a process has finished handling one task, it will begin work on the next task.

## Pools & Paralellism

Let's do another example, the classic "print time", to see how the threads are actually executing across 4 processes.

```python
import time

def g(x):
    print(time.time())
    time.sleep(1)

with Pool(4) as p:
    print(p.map(g, range(12)))
```

> <question-title></question-title>
> What did you see here?
>
> Was it clear and easy to read?
> > <solution-title></solution-title>
> > It prints four timestamps immediately, all around the same time. Then one second later it prints 4 more, and one second later, a final four.
> > Our Pool of 4 processes processes the function the maximal amount of times possible concurrently. Once each of those functions returns, then it can move on to processing the next tasks.
> > This is precisely the situation of e.g. 4 queues at the grocery; as soon as one customer is processed, they immediately begin on the next, until no more remain.
> >
> > You might also see a situation where the numbers are interleaved in a completely unreadable way. Here they write immediately, but there is no synchronisation or limiting on who can write at one time, and the result is a mess of interleaved print statements.
> {: .solution}
{: .question}

## When to parallelise

Some guidelines:

- When you can isolate a pure function
- That do not require shared state, nor ordering
- That do not modify global state (pure!)
- That take a significant amount of time, relative to the time it takes to rewrite your code to support parallelising.

Some common examples of this are slow tasks like requesting data from multiple websites, e.g. using the `requests` library. Or doing some computationally expensive calculation. The last point however is especially important, as it is not always trivial to rearchitect your code to handle being spread across multiple threads or processes.

Let's parallelize this program which fetches some metadata from multiple Galaxy servers:

```python
import requests
import time

servers = [
    "https://usegalaxy.org",
    "https://usegalaxy.org.au",
    "https://usegalaxy.eu",
    "https://usegalaxy.fr",
    "https://usegalaxy.be",
]

data = {}
start = time.time()
for url in servers:
    print(url)
    try:
        response = requests.get(url + "/api/version", timeout=2).json()
        data[url] = response['version_major']
    except requests.exceptions.ConnectTimeout:
        data[url] = None
    except requests.exceptions.ReadTimeout:
        data[url] = None


# How long did it take to execute
print(time.time() - start)

for k, v in data.items():
    print(k, v)
```

If we look at this, we can see one hot spot in the code, where it's quite slow, requesting data from a remote server. If we want to speed this up we'll need to isolate it into a pure function. Here we can see a possibility for a function that requests the data, with the input of the server url, and output of the version.

```python
def fetch_version(server_url):
    try:
        response = requests.get(server_url + "/api/version", timeout=2).json()
        return response['version_major']
    except requests.exceptions.ConnectTimeout:
        return None
    except requests.exceptions.ReadTimeout:
        return None
```

This now lacks side effects (like modifying the `data` object), and can be used in a map statement.

```python
start = time.time()
with Pool(4) as p:
    versions = p.map(fetch_version, servers)
    data = dict(zip(servers, versions))
print(time.time() - start)

for k, v in data.items():
    print(k, v)
```

Same result, and now this is a lot more efficient!

## Sizing your Pool

This depends largely on profilling your code or knowing the performance characteristics of it. In the above example, there is very little computation work executed as part of this function, it's essentially all network I/O, no CPU or memory usage.

As such, we can probably set our pool size to be very large, a multiple of our systems' processor count. It will spawn many processes that do very little.

If however this were a more complicated task (e.g. calculating a large number, machine learning, etc.), then we might wish to set our pool size to the number of CPUs - 1, as each process will potentially consume it's CPU allocation completely, and we wish to have some left over for the managing program and any other work going on, on the system.

> <question-title></question-title>
> Try changing the pool size and see what effect it has on runtime. Start from a value of 1, and go up to 5, the number of servers in our list.
> > <solution-title></solution-title>
> > Given the small sample size (5 servers), and the variability of response times (in local testing between 3-8 seconds for the single-pool version), you can see varying results but *generally* it should decrease as the pool size increases. However, sometimes you will see the Pool=5 version take the same or longer than Pool=4.
> >
> > Especially in the case that 1 server (or 1 request one time) dominates the request time, this can "hide" the improvements as the others complete quickly on the remaining N-1 processes.
> {: .solution}
{: .question}

```python
results = []
for i in range(1, 6):
    start = time.time()
    with Pool(i) as p:
        versions = p.map(fetch_version, servers)
        data = dict(zip(servers, versions))
    duration = time.time() - start
    results.append(duration)

# Plot it
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.plot(range(1, 6), results)
ax.set(ylabel='time (s)', xlabel='Pool Size', title='Pool size vs runtime')
ax.grid()
# Uncomment if your notebook cannot display images inline.
# fig.savefig("pool-vs-runtime.png")
plt.show()
```

You might see a result similar to the following:

![graph of the above, showing a decrease in runtime from 2.6 seconds to approximately 1.1 seconds as the pool size increases from 1 to 5. As the pool size increases from 1-2 there is a large improvement, but as it increases from 4-5 the improvement is very small, on the order of milliseconds.](../../images/pool-vs-runtime.png)


## Threads

Let's convert our previous example from processes to threads, as processes aren't strictly necessary for such a light weight use case as fetching data from the internet where you're blocking on network rather than CPU resources.

```python
import multiprocessing.pool
import requests
import time

data = {}
start = time.time()

pool = multiprocessing.pool.ThreadPool(processes=4)
results = pool.map(fetch_version, servers, chunksize=1)
data = {k: v for (k, v) in zip(servers, results)}
pool.close()

print(time.time() - start)

for k, v in data.items():
    print(k, v)
```

This is a bit more complicated to write, but again, if you're not blocking on CPU resources, then this is potentially approximately as effecient as thread pools.

> <question-title></question-title>
> What result did you get? Was it slower, faster, or about the same as processes?
{: .question}


topic: data-science
tutorial: cli-bashcrawl
title: CLI Educational Game - Bashcrawl
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["game", "bash"]
content_hash: d1b8d76110b12d84dd6086c8088c1761
last_modified: 2026-05-15T22:28:10.897602
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-bashcrawl/tutorial.html

content:
This is not a tutorial like most GTN content but a fun exercise for you to play around and learn a bit about the command line, and hopefully re-inforce the skills you covered in Basic and Advanced CLI skills.

> <hands-on-title>Open a Jupyter Terminal</hands-on-title>
> To best follow this tutorial, you should open a terminal.
>
> Here are some instructions for how to do this on various environments.
>
> **Jupyter** on UseGalaxy.\* and **MyBinder.org**
>
> 1. Use the File → New → Terminal menu to launch a terminal.
>
>    ![screenshot of jupyterlab showing the File menu expanded to show new and terminal option.](../../images/carpentries-cli/mybinder-launch.png)
>
> 2. Disable "Simple" mode in the bottom left hand corner, if it activated.
>
>    ![screenshot of jupyterlab showing a toggle labelled simple](../../images/carpentries-cli/mybinder-simple.png)
>
> 3. Drag one of the terminal or notebook tabs to the side to have the training materials and terminal side-by-side
>
>    ![screenshot of jupyterlab with notebook and terminal side-by-side.](../../images/carpentries-cli/mybinder-final.png)
>
> **CoCalc**
>
> 1. Use the *Split View* functionality of cocalc to split your view into two portions.
>
>    ![screenshot of cocalc button to split views](../../images/carpentries-cli/cocalc-split.png)
>
> 2. Change the view of one panel to a terminal
>
>    ![screenshot of cocalc swapping view port to that of a terminal](../../images/carpentries-cli/cocalc-terminal.png)
>
{: .hands_on}

## Setup

```bash
git clone https://gitlab.com/slackermedia/bashcrawl
cd bashcrawl/entrance
# Run: cat scroll
```

Have fun, and good luck!


topic: data-science
tutorial: data-manipulation-olympics-jq
title: Data Manipulation Olympics - JQ
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa", "json"]
content_hash: e4ef880573819b91e6ee25f2405a3e0b
last_modified: 2026-05-15T22:28:10.900170
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics-jq/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->

Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

If you've opened this tutorial via the {% icon level %} icon in Galaxy (top menu bar), you can click on the tool names in the last column to quickly open them in Galaxy and start using them on your own!

Operation                | Description                                                         | Galaxy Tool
------------------------ | ------------------------------------                                | ----------------
Compute an expression    | Over each row, add it as a new column                               | `.key1 * .key2`
Concatenate datasets     | one after the other                                                 | `cat *.json | jq`
Convert format           | Change the file format                                              | Often xsv, other tools.
Counting                 | Count the number of objects in an array                             | `length`
Cut Columns              | By header name                                                      | `.key`
Filter                   | Remove objects based on values                                      | `select(.key == "value")`
Find and Replace         | in a specific column                                                | `gsub("(?<a>...)", "xyz\\(.a)abc")`
Group on a column        | And perform simple operations (count, mean, min, max etc)           | `group_by(.key1, -.key2)`
Join two Datasets        | side by side on a specified field                                   | `[JOIN(INDEX(input[]; .right); .[]; .left; add)]`
Paste                    | Two files side by side                                              | `-s '[.[][]]'`
Remove Beginning         | Good for removing header lines                                      | `last`, `index`
Select First lines       | Good for finding top 10s or saving header lines                     | `first`, `index`
Sort on a column         | Change the order of the rows based on values in one or more columns | `sort`, `sort_by(.key)`
Split file               | Based on values of a column                                         | n/a (requires bash/awk/etc.)
Unique                   | Remove duplicate rows                                               | `unique`

In this tutorial, these functions are explained in more detail, and we provide some exercises for you to practice.

# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"]({% link topics/introduction/tutorials/data-manipulation-olympics/images/cover.jpg %})


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!


# Download Data

Before we can do any manipulation, we will need some data. Let's download our table with Olympics results now.

We can use `jq` just by itself to pretty print the results:

```bash
cat olympics.json | jq -S . | head -n 20
```

> <question-title></question-title>
>
> 1. What is the format of the file?
> 2. How is it structured?
> 3. How many lines are in the file? (Hint: `wc`)
> 4. How many columns?
>
> > <solution-title></solution-title>
> >
> > 1. It is a JSON file, or a Javascript Object Notation
> > 2. It is structured as an array (we can tell by the leading `[`) with objects representing an athlete's participation in an event. If an athlete competes in multiple events, there is an entry for each event.
> > 3. `1`. The entire JSON file is on a single line, which makes it rather unpleasant to access via normal text editors. However it contains 234,522 records (`cat olympics.json | jq -c '.[]' | wc -l`)
> > 4. There are 17 columns in this file. There are multiple ways to find this answer:
> >    - Count the fields manually
> >    - Advanced JQ: `cat olympics.json| jq '.[0] | keys | length'`
> {: .solution}
{: .question}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.json` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.json` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.

# File Format Conversion

The file we uploaded is a `.json` file. This stands for *JavaScript Object Notation*. This means that this is a file containing arrays (`[]`) and objects (`{}`). 

First we need to understand a bit about file structure, and how JQ can convert back and forth between different structures:

The `.` operator is the identity operator, it gives you back exactly the same structure

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.'
```

You can use `.[]` to loop over the items in an array:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.[]'
```

And supply a number to access an individual array item:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.[0]'
```

And `.<key>` to access a specific key

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.[].a'
```

But that's not all! Sometimes you may want the resulting output to be back in an array, rather than by line:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '[.[].a]'
```

By wrapping the entire expression in `[]` we say we want the output of that statement in an array. We can actually do the same to get each entry as a new object:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '{"results": .[].a}'
```

Or an object with an array:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '{"results": [.[].a]}'
```

Or an array of objects:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '[{"results": .[].a}]'
```

We'll need to use these reshaping abilities as we go, to get data in the right shape to be worked with. Sometimes you'll want an array of objects, or objects with items in their keys.

## Pipes

Lastly, pipes are incredibly important in JQ, we'll use them to combine multiple filters, or make our pipelines more readable:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '.'
```

Here we pipe the entire incoming object (JQ operates on streams of JSON objects, so, there can be multiple), to the identity operator again but this time with `[]` to loop over every object in that item. (Really this is the same as `.[]` but we use a pipe for demonstration.)

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '. | .[]'
```

We can then pass that stream of individual items (no longer a single JSON object) on to another filter, one extracting an `.a`:

```bash
echo '[{"a": 1}, {"a": 2}]' | jq '. | .[] | .a'
```


# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

In `jq` we can use the `sort` or `sort_by` function. From the documentation:

```bash
echo '[8,3,null,6]' | jq '.sort'
```

`sort_by` however lets us sort multiple entries in an array, by a child key:

```bash
echo '[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]' | jq '.sort_by(.foo)'
echo '[{"foo":4, "bar":10}, {"foo":3, "bar":100}, {"foo":2, "bar":1}]' | jq '.sort_by(.bar)'
```

So let's sort our file in chronological order, based on the year of the Olympic games:

> <question-title></question-title>
>
> 1. Which key contains the year?
> 2. Do we want ascending or descending order if we want the oldest games at the top?
>
> > <solution-title></solution-title>
> >
> > 1. `.year`
> > 2. The file should be sorted in ascending (increasing) order
> >
> {: .solution}
{: .question}

```bash
cat olympics.json | jq -c 'sort_by(.year) | .[]'
```

> <tip-title>What does `.[]` do?</tip-title>
> We use this to first take the identity operator, taking exactly the array that was passed to us, and then `[]` lets us loop over the array, returning a stream of multiple individual objects. The `-c` flag lets us see that more easily, with one object per line.
{: .tip}

If we wanted to do it in reverse, we could just use the `reverse` function, or just use `-` prefixed to our sort factor.

```bash
cat olympics.json | jq -c 'sort_by(.year) | reverse | .[]'
cat olympics.json | jq -c 'sort_by(-.year) | .[]'
```

> <question-title></question-title>
>
> 1. Write a JQ command to access only the first entry.
> 2. Which athlete is listed at the top of the file now?
>
> > <solution-title></solution-title>
> > 1. We can simply access the 0th element:
> >    ```bash
> >    cat olympics.json | jq -c 'sort_by(.year) | .[0]'
> >    ```
> > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
> >
> {: .solution}
{: .question}

**Rename** the resulting file to somthing meaningful (e.g. `olympics-chronological.json`)

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, an then within each year, we sort again alphabetically by name. The `sort_by` function can do this too!

We will sort the file in chronological order based on the year of the Olympic games

```bash
cat olympics.json | jq -c 'sort_by(.year, .name) | .[]'
```

> <question-title></question-title>
>
> Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>
> > <solution-title></solution-title>
> >
> > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
> >
> {: .solution}
{: .question}


## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> > <br>
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <tip-title>Removing null values</tip-title>
> > This will be covered more during the Filtering section, but for now simply use this filter:
> > ```bash
> > cat olympics.json | jq '[.[] | select(.height != null)] | ...your sort_by here...'
> > ```
> {: .tip}
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We can use `.height`, and because we want the tallest on top, we will need to sort in *descending* (decreasing) order. Unfortunately you might discover there are null values.
> > 2. Rerun the same query as step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Lyton Mphande from Seol is the shortest at 127 cm.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > 1. `jq -c '[.[] | select(.height != null)] | sort_by(-.height) | .[0]' < olympics.json`
> > 2. `jq -c '[.[] | select(.height != null)] | sort_by(.height) | .[0]' < olympics.json`
> > 3. `jq -c '[.[] | select(.height != null)] | sort_by(-.year, -.height) | .[0]' < olympics.json`
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).


We will filter the file to show only winter Olympics
Look at the `olympics.json` file and answer the following questions

> <question-title></question-title>
>
> 1. Which key contains this information?
> 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>
> > <solution-title></solution-title>
> >
> > 1. `.season`
> > 2. The values can be `Summer` or `Winter` (`jq -c '[.[].season] | sort | unique' < olympics.json`)
> >
> {: .solution}
{: .question}

We'll be using the `select()` filter to select entries matching specific conditions:

> <question-title></question-title>
>
> 1. How would you write the expressions for the following conditions:
>    1. `enrolled` must be 'Yes'
>    2. `age` must be smaller than 75
>    3. `height` cannot be null
>    4. `birthplace` cannot be empty
>
> 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    How would you write expressions for the following filtering conditions:
>    1. `height` is larger than 200 or smaller than 160
>    2. `height` is larger than 200 and smaller than 210
>
> > <solution-title></solution-title>
> >
> > 1. The answers are:
> >    1. `select(.enrolled == "Yes")`
> >    2. `select(.age < 75)`
> >    3. `select(.height != null)``
> >    4. `select(.birthplace != "")`
> >
> > 2. The answers are:
> >    1. `select(.height > 200 or .height < 160)`
> >    2. `select(.height > 200 and .height < 210)`
> >
> {: .solution}
{: .question}



Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics. Make sure it is contained in an array, in case we want to do further sorting.

```bash
jq -c '[.[] | select(.season == "Winter")]' < olympics.json > winter.json
```

> <question-title></question-title>
>
> How many lines are in this file? (Hint: use the `length` filter.)
>
> > <solution-title></solution-title>
> >
> > 44,680
> >
> {: .solution}
{: .question}


**Repeat** the step for the Summer Olympics

```bash
jq -c '[.[] | select(.season == "Summer")]' < olympics.json > summer.json
```


> <question-title></question-title>
>
> 1. How many lines do you expect in the this file?
> 2. How many lines are in this file? Were you right?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Use the `length` filter, assuming your data is in an array, per the original prompt.
> > 2. Be careful to consider whether these counts include the header line of the file or not
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > 1. The original file has 234,522 lines, and the Winter Olympics had 44,680 lines. So we would expect 234,522 - 44,680 = 189,842 rows of data.
> > It is always useful to take a moment to think about the expected outcome, this makes it easier to spot mistakes and will save you time in the long run.
> >
> {: .solution}
{: .question}

## Exercises

Ok, time to train! let's see if you can use the `select` filter to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm? (inclusive)
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> > - You may want to `[ ... query ...] | length` to get a nice length
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `select(.medal == "Gold")`)
> >  2. 24,633  (Expression: `select(.medal == "Gold" or .medal == "Silver" or .medal == "Bronze")`, or `select(.medal != null)`)
> >  3. 131     (Expression: `select(.medal == "Gold" and .year == 2018)` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `select(.medal != null and .height >= 170 and .height <=180)`)
> >  5. 2,333   (Expression: `select(.medal != null and (.height < 160 or .height > 190))` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count.

Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)

We'll need to use the `group_by` filter which takes a key, and then emits arrays with objects with those matching keys:

```bash
echo '[{"foo":1, "bar":10}, {"foo":3, "bar":100}, {"foo":1, "bar":1}]' | jq 'group_by(.foo)
```

So let's try that with ours:

```bash
jq '. | group_by(.year)' < olympics.json
```

We now have a structure like

```
[
    [
        {"year" 1896, ...},
        {"year" 1896, ...},
        {"year" 1896, ...}
    ],
    [
        {"year" 1900, ...},
        {"year" 1900, ...},
        {"year" 1900, ...}
    ],
    ...
]
```

We'll re-structure to one "year group" per line:

```bash
jq '. | group_by(.year) | .[]' < olympics.json
```

So we can again use the `length` function to determine how many items are in each via

```bash
jq '. | group_by(.year) | .[] | [. | length]' < olympics.json
```

And all that's left is to include the game name:

```bash
jq '. | group_by(.year) | .[] | [. | length, .[0].games]' < olympics.json
```

But the output format is not so nice, so let's introduce the `@tsv` filter to turn the resulting data into a table:

```bash
jq '. | group_by(.year) | .[] | [. | length, .[0].games] | @tsv' < olympics.json
```

Where did the " and `\t` come from? We don't want those! That is a result of the resulting data being in a JSON string, so we can tell jq to use `raw` mode with `-r`

```bash
jq -r '. | group_by(.year) | .[] | [. | length, .[0].games] | @tsv' < olympics.json
```

> <question-title></question-title>
>
> 1. How many different Olympic games are in our file?
> 2. Which Olympic games had the most participations? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>
> > <solution-title></solution-title>
> >
> > 1. 52 games (`[. | group_by(.games) | .[] | [. | length, .[0].games]] | length`)
> >
> >    The resulting file looks something like:
> >
> >     ```
> >     615	1896 Summer Olympics
> >     2503	1900 Summer Olympics
> >     2643	1904 Summer Olympics
> >     3213	1908 Summer Olympics
> >     4610	1912 Summer Olympics
> >     3448	1920 Summer Olympics
> >     5242	1924 Summer Olympics
> >     358	1924 Winter Olympics
> >     4493	1928 Summer Olympics
> >     ...
> >     ```
> >
> > 2. 1996 Summer Olympics. (10501 participations)
> >
> {: .solution}
{: .question}

You may have guessed that like `sort_by`, that we could have selected multiple columns in the `group_by` step. This lets us count on combinations of columns.

Let's try counting the number of men and women in each olympic games.

```bash
jq '. | group_by(.games, .sex) | .[] | [. | length, .[0].games, .[0].sex] | @tsv' -r < olympics.json
```


> <question-title></question-title>
>
> You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>
> 1. How many women were in the first Olympic games?
>
> 2. Which Olympic games had the most women participants?
>
> > <solution-title></solution-title>
> >
> > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice).
> >    The file looks something like this:
> >    ```
> >    2	F	1896 Summer Olympics
> >    43	F	1900 Summer Olympics
> >    17	F	1904 Summer Olympics
> >    55	F	1908 Summer Olympics
> >    97	F	1912 Summer Olympics
> >    132	F	1920 Summer Olympics
> >    269	F	1924 Summer Olympics
> >    ```
> >
> > 2. 2020 Summer Olympics (4652)
> >
> {: .solution}
{: .question}


Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting tool above, we would get a results file for each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

We can use the `unique` filter in our pipeline to discover this. First let's do our group by and iterate over each resulting group:

```bash
jq '. | group_by(.games) | .[]' < olympics.json
```

For fun this time let's organise the output as an object. We know one value we want is the games:

```bash
jq '. | group_by(.games) | .[] | {"games": .[0].games}' < olympics.json
```

But the other we want is the list of sports. We can use our `.[]` to loop over every sport item in that group

```bash
jq '. | group_by(.games) | .[] | {"games": .[0].games, "sports": [.[].sport] }' < olympics.json
```

We're almost there! We just need to unique this list and determine it's length:

```bash
jq '. | group_by(.games) | .[] | {"games": .[0].games, "sports": [.[].sport] | unique | length }' < olympics.json
```

Or if you preferred as a table. But here it seems we can wrap it in `()` to make the parsing clearer:

```bash
jq '. | group_by(.games) | .[] | [.[0].games, ([.[].sport] | unique | length)] | @tsv' -r < olympics.json
```

> <question-title></question-title>
>
> 2. How many sport were in the first Olympics? How many in the latest?
> 3. Which Olympics had the most different sports?
>
> > <solution-title></solution-title>
> >
> > 2. 10 and 38.
> > 3. The 2020 Summer Olympics had the most different sports (38)
> >
> {: .solution}
{: .question}

Save the output as something descriptive.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting instances of a key, we can use `group_by(.team)` and then loop over that to print out the length, and the team name of each of those items.
> > 2. This is basically the same question as "how many women" participated, try modifying that query.
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations (`cat olympics.json | jq '. | group_by(.team) | .[] |  [. | length, .[0].team] | @tsv' -r`)
> >  2. 15 and 250. (`cat olympics.json | jq '. | group_by(.games) | .[] | [.[0].games, ([.[].team] | unique | length)] | @tsv' -r`)
> >
> {: .solution}
{: .question}

# Computing

Sometimes we want to use the data in our column to compute a new value, and add that to the table. For instance, for our dataset we could caluclate athtletes BMI (using height and weight columns), or their age at time of participation (from year of birth and year of the Olymics). By adding these computed values as a new colum to our datset, we can more easily query the dataset for these values. We can do these types of operations using the `+=` operation on a given object.

As an example, let's calculate the age of each athlete at the time of participation, and add this as a new column to our dataset.

First let's structure our query to loop over every object in the array:

```bash
jq '.[]' < olympics.json
```

Then let's try adding a static value to it:

```bash
jq '.[] | . += {"marco": "polo"}' < olympics.json
```

Do you see the resulting key in the new file? Ok, let's replace it with a computed value:

```bash
jq '.[] | . += {"age": .year - .birth_year}' < olympics.json
```

This errors, but JQ is a bit sensitive to syntax, so we can add some `()` to make things more explicit

```bash
jq '.[] | . += {"age": (.year - .birth_year)}' < olympics.json
```

Ah and then we discover null values!

```bash
jq '.[] | select(.birth_year != null) | . += {"age": (.year - .birth_year)}' < olympics.json
```

And there we go! But this is structured as a stream of objects, maybe we want it again as an array `[]` to ensure it remains in the same shape as our input dataset.

```bash
jq '[.[] | select(.birth_year != null) | . += {"age": (.year - .birth_year)}]' < olympics.json > olympics-with-age.json
```

> <question-title></question-title>
>
> 1. How old was Arnaud Boetsch during his Olympic tennis participation?
>
> > <solution-title></solution-title>
> >
> > 2. Arnaud Boetsch is listed on the first two lines, who turned 27 the year of their Olympics.
> >
> {: .solution}
{: .question}

This was a simple computation, but much more complex mathematical expressions can be computed with this tool. See the help section at the bottom of the tool for a list of all supported operations. In the exercise below, we will compute the BMI for each athlete as an example.


## Exercises

BMI stands for Body Mass Index, is a metric to provide a very crude measure of how healthy your weight is. The formula to compute BMI is:

$$ BMI = weight / (height^2) $$

(with weight in kilograms and height in meters).

> <tip-title>BMI is problematic</tip-title>
> [Adolphe Quetelet](https://en.wikipedia.org/wiki/Adolphe_Quetelet), who invented [BMI](https://en.wikipedia.org/wiki/Body_mass_index), was not a doctor of medicine, instead he was a Belgian astronomer, mathematician, statistician, and sociologist. His data consisted probably entirely of cisgender, white, european men and women {% cite Eknoyan_2007 %}.
> In his defense, he did argue it should not be used at individual levels, however it came to be used as such due to simplicity. (See [Wikipedia's History section](https://en.wikipedia.org/wiki/Body_mass_index#History)). BMI is a poor measure of health, especially for populations with high muscle content, and cannot be simply re-used as-is with anyone other than Anglo Saxons {% cite Caleyachetty_2021 %}, [Wikipedia:Limitations](https://en.wikipedia.org/wiki/Body_mass_index#Limitations).
>
> However as it is an easy to calculate metric, we include the calculation here.
{:.tip}


Let's compute this data for all athletes and add it as a new column!

> <question-title noprefix>Exercise: Calculating BMI</question-title>
>
> 1. How would you express this calculation in jq?
>    - Remember that our height is in cm, and the formula expects height in meters
>    - And that we have null values we must filter out
>
> 2. What is the BMI for Arnaud Boetsch?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - division is `/` and multiplication is ` * ` .
> > - Generally we cannot use `^`
> > - Parentheses may be required.
> > - use `select(.value != null)` to remove nulls.
> > - If that isn't eough, you can check that the type is a number with `(.value|type) == "number"` to ensure it really is a number and not e.g. a string.
> > - remember to wrap everything in `[...]` to retain the data shape
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> > 1. other variations are possible:
> >    ```bash
> >    jq '[.[] | select((.height|type) == "number" and (.weight|type) == "number") | . + {"bmi": (.weight / (.height/100 * .height/100))}]' < olympics.json > olympics-bmi.json
> >    ```
> > 2. 22.69
> >    ```bash
> >    jq '.[] | select(.name=="Arnaud Boetsch") | .bmi' < olympics-bmi.json
> >    ```
> {: .solution}
{: .question}


# Find and Replace

Often you may need to change the contents of a file a bit to fit the expectations of an analysis tool. For instance, our file uses `NA` for missing values, but other conventions included leaving the cell empty, or using `NaN` (not a number) instead. Or, when working with chromosomal data, you may need to add or remove the `chr` prefix from a column before using it as input to a certain tool. In such situations, we can find all occurrences of a certain pattern in our file, and replace it with another value.

If we want to perform such a replacement on a single column in our data, we can use the `gsub` function.

A few of the basics of regular expression, plus some links to further resources are given in the box below:

{% snippet faqs/galaxy/analysis_regular_expressions.md %}

Let's start with a simple example:
Our file uses a mix of `Athina` and `Athens` to indicate the Capital City of Greece in the `city` column.
Let's standardize this by replacing occurrences of `Athina` with `Athens`.

Let's start by filtering out the old spelling:

```bash
jq '.[] | select(.city == "Athina")' < olympics.json
```

We can access just the .city and try out our gsub next:

```bash
jq '.[] | select(.city == "Athina") | .city | gsub("Athina"; "Athens")' < olympics.json
```

Looks good! Let's use the technique we learned in [Computing](#computing) to update the key with a new key.

```bash
jq '.[] | select(.city == "Athina") | . += {"city": (.city | gsub("Athina"; "Athens"))}' < olympics.json
```

Also looks good, ok, that's it!

```bash
jq '[.[] | . += {"city": (.city | gsub("Athina"; "Athens"))}]' < olympics.json
```


Look at the file before and after. Athlete 7 (Patrick Chila) near the top of the `olympics.tsv` file, had a value of Athina in the city column. Verify that it has been changed to Athens.

This was rather simple example, so let's try a few more examples with slightly more complex expressions.


## Exercises

You may have noticed that our file has a lot of missing data. Especially for the earlier years, things like height, weight and birthday of athletes was not registered, or simply not known. In some columns you see these missing values have been replaced with an `NA` (not available) value. In other columns (for example birth place), the cells have simply been left empty.

Different tools may expect different ways of handling missing data. So you may have to change your missing data from empty to `NA`, `NaN`, or something else, between analysis steps

> <hands-on-title>Fill empty cells</hands-on-title>
>
> We will now replace empty cells in the `birth_place` column, to use `null` instead.
> Read the [documentation for PCRE and gsub](https://stedolan.github.io/jq/manual/#RegularexpressionsPCRE)
>
> > <question-title></question-title>
> >
> > 1. Should we use gsub?
> > 2. What is the expression for a line with just whitespace?
> >
> > > <solution-title noprefix>Answer</solution-title>
> > >
> > > 1. Yes, definitely
> > > 2. `^ *$` indicates an empty line, potentially with spaces. (`^` indicated the beginning, zero or more spaces, and `$` the end). The value in the column is treated as the line, since we are looking only in 1 column.
> > >
> > {: .solution}
> >
> > > <solution-title noprefix>Hints</solution-title>
> > > Note that you cannot use `gsub` to replace something with null. Instead you'll need to use `(if ... then "value" else "other" end)`.
> > > You might want to use the gsub instead to ensure lines with single blank characters, and empty lines, all map to the same null result.
> > {: .solution}
> >
> > > <solution-title noprefix>Full Solution</solution-title>
> > > other variations are possible:
> > > ```bash
> > > cat olympics.json | jq '.[] | (if (.birth_place|gsub("^ *$"; "")) == "" then null else .birth_place end)'
> > > ```
> > {: .solution}
> {: .question}
>
{: .hands_on}

Let's do another example, this one using capture groups.

Look at the `birth_day` column. It has values in a format like `12 December`. Suppose we have a tool that expects this data to be in the reverse format, `December 12`. The file is too big to change this manually in every column. But with regular expression tools we can make this replacement easily

We will now change the format in birthday column from `day month` to `month day`, as our boss is American and requested the silly format.

Read the [documentation for PCRE and gsub](https://stedolan.github.io/jq/manual/#RegularexpressionsPCRE)
- Read the text, what settings do you think we need to use?
- Read the Regular expressions 101 FAQ below

  {% snippet faqs/galaxy/analysis_regular_expressions.md %}

> <question-title></question-title>
>
> 1. How do we match on the birthday format? How strict/exact shoule we be here?
> 2. How do we captures both the day and the month?
> 3. How do we refer to the values we captured (for the replacement value)
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Birthday is one or more digits, followed by a space, followed by one or more letters.
> > 2. Remember that you can capture values using parentheses `(..)`
> > 3. We can name our capturing groups for easier referencing, by providing a name. `(?<day>...)` can be later referred to as `\(.day)`.
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > 1. There are multiple solutions here, depending on how strict you want to be
> >    - `\d+ ([a-zA-Z]+)` (not strict, would also match on `142 Septober`
> >    - `[0-9]{1,2} (January|February|March|April|May|June|July|August|September|October|November|December)` (this is much more strict, only matches on 1 or 2 digits, followed by a space, followed by one of the months. But this would still match on 42 April, and may miss it if one month names didn't start with a capital letter)
> >    - `[123]?[0-9] [a-zA-Z]+` this will only allow dates below 40
> >
> >    There are different ways to express this, and there is no one perfect solution. If you know your data is clean, and you do not have to worry about values like 42 Septober,
> >    then you can be less strict. If your data is less clean, you may be more worried about capturing things that aren't valid birthdays and might want to be a bit stricter.
> >    It all depends on your data and use case.
> >
> > 2. `(?<day>[\d]{1,2}) (?<mon>[a-zA-Z]+)` captures both the day and the month
> > 3. We can use `\(.day)` to refer to the day we captured, and `\(.mon)` for the month in our replacement.
> >
> > If you'd like to be fancier than just using `gsub`, you can also try the other capturing functions. E.g. `capture` returns an object with the named matches, which you can add back directly in to the original object:
> >
> > ```bash
> > jq '.[] | . += (.birth_day | capture("(?<birth_day>[0-9]+) (?<birth_month>[A-Z][a-z]+)\\s*"))' < olympics.json
> > ```
> >
> {: .solution}
{: .question}


# Removing Columns

We can remove columns from a table using either `del(.key)` if we just want to remove a few values, or otherwise by selecting, if we just want to keep a few values.

Suppose we want to simplify our file a bit. All we want is file with 4 columns: athlete name, sport, olympic games, and medals.

We can most easily do this by returning an array:

```bash
jq '.[] | [.name, .sport, .games, .medal]' < olympics.json
```

But perhaps we want something a bit more structured:

```bash
jq '.[] | {"name": .name, "sport": .sport, "games": .games, "medal": .medal}' < olympics.json
```

And of course we'll keep our list shape:

```bash
jq '[.[] | {"name": .name, "sport": .sport, "games": .games, "medal": .medal}]' < olympics.json > subset.json
```

Notice that during this step, we also changed the order of the keys. This tool can also be used to re-arrange keys, if you supply all key names but in a different order.

## Exercises

> <question-title noprefix>Exercise: Removing Columns</question-title>
>
> 1. Create a file that is similar to `olympics.json`, but without the first column (athlete_id column)
> 2. Which of the two methods would be easier to use and when? (this can be a personal preference, but think about whether you would subtract or add)
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Try the `del(.key)` function
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > 1. `jq '[.[] | del(.athlete_id)]' < olympics.json`
> > 2. The `del()` function is significantly easier if you are just removing keys, but may be cumbersome if you have a lot of key and only want a few. In that case it may be faster to type those out instead.
> {: .solution}
{: .question}


# Unique

Sometimes, in the course of our data manipulations, we may end up with a file that has duplicate values. In order to filter these out, we can use the `unique` filter.

Let's say we would like to create a list of all unique athletes (id and name).

First we will cut just the `athlete_id` and `name` columns from our dataset

```bash
jq '.[] | {"name": .name, "athlete_id": .athlete_id}' < olympics.json
```

> <question-title></question-title>
>
> 1. Do you see duplication? Why is that?
>
> > <solution-title></solution-title>
> >
> > 1. Yes. For all athletes who participated more than once, the row will be identical.
> >
> {: .solution}
{: .question}

Now let's remove those duplicates. For this filter to work, it needs to be passed an array, not individual values in a stream:

```bash
jq '[.[] | {"name": .name, "athlete_id": .athlete_id}] | unique' < olympics.json
```

> <question-title></question-title>
>
> How many unique athletes do we have?
>
> > <solution-title></solution-title>
> > 94,733
> >
> > ```bash
> > jq '[.[] | {"name": .name, "athlete_id": .athlete_id}] | unique | length' < olympics.json
> > ```
> >
> {: .solution}
{: .question}


# Joining Files

This file contains a lot of information, but we may want to add more information. For example if we had a file with information about each country (population, capital city, etc), we could join that information with our Olympics data, to get a single file with all information in every row.

For example, if we would like to be able to group by continent, to e.g. count athletes, medals etc per continent, we will have to add a `continent` column to our file. To do this we would need a second file that maps each country to the corresponding continent. This is what we will do in the next hands-on section.

We obtained country information data from [DataHub](https://datahub.io/core/country-codes). More information about this file can be found in the description there. It has 56 columns with a wide variety of data about each country (from country codes, to capital city, languages spoken, etc)

Download

```
{{page.zenodo_link}}/files/country-information.json
```

> <question-title></question-title>
>
> 1. How many keys does this file have?
> 2. Which keys(s) in this file are the same as in the `olympics.tsv` file?
>
> > <solution-title></solution-title>
> >
> > 1. The country information file has 56 columns (see [DataHub](https://datahub.io/core/country-codes) for more details).
> > 2. Both files have a `NOC` column with the 3-letter country code (`NOC` stands for National Olympic Committee). However, one is lowercase.
> {: .solution}
{: .question}

We would now like to take our Olympics dataset as the basis, and add columns to every row of this file with some information about the country. In order to join, we will need to have one column that is shared between the two files, on which we can match. The `NOC` column is perfect for this because it is a defined standard. Both files also contain a column with the country name in it, which is also a possible candidate to use for joining, but because it is less standardised, it is safer to use the NOC column. For example, if one file uses "Netherlands", while the other uses "The Netherlands" to indicate the same country, the joining will fail for these rows. So always make sure the columns you join on are compatible!

We can use the join and index commands, per [SQL Style Operators](https://stedolan.github.io/jq/manual/#SQL-StyleOperators) documentation and [this stack overflow answer](https://stackoverflow.com/a/72609410)

```bash
jq '[JOIN(INDEX(input[]; .NOC); .[]; .noc; add)]' olympics.json country-information.json
```

> <question-title></question-title>
>
> 1. What do you expect the output to look like? Were you right?
> 2. How many columns are in the resulting file? What about the NOC column?
> 3. What is a possible downside to this approach?
>
> > <solution-title></solution-title>
> >
> > 1. All the columns from the country information file are added to the end of each row of our olympics dataset
> > 2. Our olympics datset had 17 columns, the country information file has 56 columns. Therefore we have 17+56=73 columns columns in our resulting file. This also means the NOC column
> >    we joined on appears twice in our output.
> > 3. There is a lot of data duplication in this file now. The exact same country information is added to every line of every athlete from a certain country.
> >    This means much larger file size.
> >    If you do not need all these columns, it could save you a lot of space to remove unneeded columns from the `country-information.json` file, before joining.
> >
> {: .solution}
{: .question}

# Concatenating

Concatenation of two files simple means putting the contents of the two files together, one after the other. Our dataset was created in 2021, but since then we've had another Olympic event, the 2022 Winter Olympics in Beijing. If we have the same data for this latest Olympics, we could simply add the rows from the 2022 games to our current file with data, in order to create a single file with all data from 1896 to 2022.

First, let's get this data for the 2022 Olympics

```
wget {{page.zenodo_link}}/files/olympics_2022.json
```

View the new dataset, does it have the same structure as our original `olympics.json` file?

> <question-title></question-title>
>
> 1. Does the new file have the same structure?
> 2. Can we simply add the lines of the new files to the end of our existing olympics dataset?
>
> > <solution-title></solution-title>
> >
> > 1. Yes, this file has all the same columns, in the same order, so concatenation should be relatively straightforward.
> > 2. If we simply put the contents of this file after our existing dataset, we will have a second header line in the middle of our data rows. It is best to remove the header line from the second dataset after we have verified it is compatible. This way we will only add the real data rows to our dataset.
> >
> {: .solution}
{: .question}

Since this new dataset has the exact same structure (number and order of columns), we can simple add the lines from this file to the end of our existing `olympics.json` file.
For this, we'll need to use a new argument to `jq`, namely `-s` or `--slurp` which allows us to access all files as if they were one big indexed array.

```bash
cat olympics.json olympics_2022.json | jq -s '.[0] + .[1]' > olympics-combined.json
```

Alternatively we could loop over every array in the parent, and combine them. This would support more than two files:

```bash
cat olympics.json olympics_2022.json | jq -s '[.[][]]' > olympics-combined.json
```

Now this only works so simply because our two datasets had the same structure; two arrays. If your data comes from different sources, you may have to do some additional data manipulation before you can concatenate, e.g. to make sure the columns match, or how each file deals with missing data (empty cells, `NA`, `NaN` or something else).


# Splitting Files

This dataset contains a lot of data, we might want to split the data into one file per Olympic games, or have one file for all the winter games, and another file for all the summer games. In these situtations, where we want to create new, smaller files, based on the values in a column, we can use the tool.

> <tip-title>Not generically possible with only JQ</tip-title>
> It is not generically possible to e.g. split a file by value, with only jq. We need to use other tools, e.g. for loops, in addition to accomplish this.
{: .tip}

We would like to create a separate file for each Olympic event. First let's get a list of every event.

```bash
jq '[.[] | .games] | unique[]' -r < olympics.json
```

Easily done. Now we'll construct a loop over these results

```bash
jq '[.[] | .games] | unique[]' -r < olympics.json | while read -r game; do
    echo "Game: $game";
done
```

Here we can then use this to create a new file per the game identifier:

```bash
jq '[.[] | .games] | unique[]' -r < olympics.json | while read -r game; do
    echo "Selecting: $game";
    jq '[.[] | select(.games == "'$game'")]' < olympics.json > olympics-"$game".json
done
```


> <question-title></question-title>
>
> 1. How many different files do we get?
> 2. How many lines are in the file for the 1908 Summer Olympics?
>
> > <solution-title></solution-title>
> >
> > 1. We get 1 collection with 52 files in it, one per Olympic games:
> > 2. 61049 lines, 3213 entries.
> {: .solution}
{: .question}


## Exercises

Let's practice this a bit more, see if you can use the split file to answer the following questions:

> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. Create two files, one for Summer Olympics, one for Winter Olympics. Which has more lines?
> 2. Split the file by sport, how many sports have there been at the Olympics?
> 3. Split the file by medal, would you expect the output files to be equal sizes?
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The summer olympics file has more lines than the winter olympics file (the first winter Olympics wasn't held until 1924)
> >  2. Each file in the resulting collection represents a sport, there are 91 datasets, representing 91 different sports in the history of the Olympics.
> >  3. The files for gold silver and bronze are roughly equal, but not exactly (think of situations like shared second place, then 2 silver medals are handed out and no bronze medal)
> {: .solution}
>
{: .question}



# Conclusion

These tools and operations covered in the tutorial are just a few examples of some of the most common operations. There are many more tools available in Galaxy that perform other data manipulations. We encourage you to look around the `General Text Tools` section in the Galaxy toolbox (left panel) to find and explore more data manipulation tools. The more comfortable you are performing these kinds of steps, the more you can get out of Galaxy!


# Exercises: Putting it all together!

This section provides a number of exercises that require you to combine two or more of the techniques you learned in this tutorial. This is a great w


topic: data-science
tutorial: sql-python
title: SQL with Python
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["SQL", "Python"]
content_hash: 0c86c853e687219b51e2adfe5f918128
last_modified: 2026-05-15T22:28:11.121780
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-python/tutorial.html

content:
This tutorial will introduce you to accessing a SQL database from within Python. Experience with both SQL and Python is a pre-requisite.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

For this tutorial we need to download a database that we will use for the queries.

```python
!wget -c http://swcarpentry.github.io/sql-novice-survey/files/survey.db
```


# Programming with Databases - Python

Let's have a look at how to access a database from
a general-purpose programming language like Python.
Other languages use almost exactly the same model:
library and function names may differ,
but the concepts are the same.

Here's a short Python program that selects latitudes and longitudes
from an SQLite database stored in a file called `survey.db`:

```python
import sqlite3

connection = sqlite3.connect("survey.db")
cursor = connection.cursor()
cursor.execute("SELECT Site.lat, Site.long FROM Site;")
results = cursor.fetchall()
for r in results:
    print(r)
cursor.close()
connection.close()
```

The program starts by importing the `sqlite3` library.
If we were connecting to MySQL, DB2, or some other database,
we would import a different library,
but all of them provide the same functions,
so that the rest of our program does not have to change
(at least, not much)
if we switch from one database to another.

Line 2 establishes a connection to the database.
Since we're using SQLite,
all we need to specify is the name of the database file.
Other systems may require us to provide a username and password as well.
Line 3 then uses this connection to create a cursor.
Just like the cursor in an editor,
its role is to keep track of where we are in the database.

On line 4, we use that cursor to ask the database to execute a query for us.
The query is written in SQL,
and passed to `cursor.execute` as a string.
It's our job to make sure that SQL is properly formatted;
if it isn't,
or if something goes wrong when it is being executed,
the database will report an error.

The database returns the results of the query to us
in response to the `cursor.fetchall` call on line 5.
This result is a list with one entry for each record in the result set;
if we loop over that list (line 6) and print those list entries (line 7),
we can see that each one is a tuple
with one element for each field we asked for.

Finally, lines 8 and 9 close our cursor and our connection,
since the database can only keep a limited number of these open at one time.
Since establishing a connection takes time,
though,
we shouldn't open a connection,
do one operation,
then close the connection,
only to reopen it a few microseconds later to do another operation.
Instead,
it's normal to create one connection that stays open for the lifetime of the program.

Queries in real applications will often depend on values provided by users.
For example,
this function takes a user's ID as a parameter and returns their name:

```python
import sqlite3

def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id='" + person_id + "';"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query)
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

print("Full name for dyer:", get_name('survey.db', 'dyer'))
```

We use string concatenation on the first line of this function
to construct a query containing the user ID we have been given.
This seems simple enough,
but what happens if someone gives us this string as input?

```
dyer'; DROP TABLE Survey; SELECT '
```

It looks like there's garbage after the user's ID,
but it is very carefully chosen garbage.
If we insert this string into our query,
the result is:

```
SELECT personal || ' ' || family FROM Person WHERE id='dyer'; DROP TABLE Survey; SELECT '';
```

If we execute this,
it will erase one of the tables in our database.

This is called an SQL injection attack,
and it has been used to attack thousands of programs over the years.
In particular,
many web sites that take data from users insert values directly into queries
without checking them carefully first.
A very [relevant XKCD](https://xkcd.com/327/) that explains the
dangers of using raw input in queries a little more succinctly:

![A 4 panel comic, in the first panel a person is shown answering the phone, hearing that their son's school has some computer trouble. In panel 2 they apologises asking if their child broke something. In panel 3, the unseen person on the other end of the phone call asks if they really named their son Robert'); Drop table students;--? They respond saying 'oh yes. little bobby tables we call him.' In the 4th panel the caller says 'well we have lost this years student records, I hope you're happy.' They respond 'And I hope you've learned to sanitize your database inputs'.](../../images/xkcd/exploits_of_a_mom.png)

Since a villain might try to smuggle commands into our queries in many different ways,
the safest way to deal with this threat is
to replace characters like quotes with their escaped equivalents,
so that we can safely put whatever the user gives us inside a string.
We can do this by using a prepared statement
instead of formatting our statements as strings.
Here's what our example program looks like if we do this:

```python
import sqlite3

def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id=?;"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, [person_id])
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

print("Full name for dyer:", get_name('survey.db', 'dyer'))
```

The key changes are in the query string and the `execute` call.
Instead of formatting the query ourselves,
we put question marks in the query template where we want to insert values.
When we call `execute`,
we provide a list
that contains as many values as there are question marks in the query.
The library matches values to question marks in order,
and translates any special characters in the values
into their escaped equivalents
so that they are safe to use.

We can also use `sqlite3`'s cursor to make changes to our database,
such as inserting a new name.
For instance, we can define a new function called `add_name` like so:

```python
import sqlite3

def add_name(database_file, new_person):
    query = "INSERT INTO Person (id, personal, family) VALUES (?, ?, ?);"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, list(new_person))
    cursor.close()
    connection.close()


def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id=?;"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, [person_id])
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

# Insert a new name
add_name('survey.db', ('barrett', 'Mary', 'Barrett'))
# Check it exists
print("Full name for barrett:", get_name('survey.db', 'barrett'))
```


Note that in versions of sqlite3 >= 2.5, the `get_name` function described
above will fail with an `IndexError: list index out of range`,
even though we added Mary's
entry into the table using `add_name`.
This is because we must perform a `connection.commit()` before closing
the connection, in order to save our changes to the database.

```python
import sqlite3

def add_name(database_file, new_person):
    query = "INSERT INTO Person (id, personal, family) VALUES (?, ?, ?);"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, list(new_person))
    cursor.close()
    connection.commit()
    connection.close()


def get_name(database_file, person_id):
    query = "SELECT personal || ' ' || family FROM Person WHERE id=?;"

    connection = sqlite3.connect(database_file)
    cursor = connection.cursor()
    cursor.execute(query, [person_id])
    results = cursor.fetchall()
    cursor.close()
    connection.close()

    return results[0][0]

# Insert a new name
add_name('survey.db', ('barrett', 'Mary', 'Barrett'))
# Check it exists
print("Full name for barrett:", get_name('survey.db', 'barrett'))
```


> <question-title>Filling a Table vs. Printing Values</question-title>
>
> Write a Python program that creates a new database in a file called
> `original.db` containing a single table called `Pressure`, with a
> single field called `reading`, and inserts 100,000 random numbers
> between 10.0 and 25.0.  How long does it take this program to run?
> How long does it take to run a program that simply writes those
> random numbers to a file?
>
> > <solution-title></solution-title>
> > ```
> > import sqlite3
> > # import random number generator
> > from numpy.random import uniform
> >
> > random_numbers = uniform(low=10.0, high=25.0, size=100000)
> >
> > connection = sqlite3.connect("original.db")
> > cursor = connection.cursor()
> > cursor.execute("CREATE TABLE Pressure (reading float not null)")
> > query = "INSERT INTO Pressure (reading) VALUES (?);"
> >
> > for number in random_numbers:
> >     cursor.execute(query, [number])
> >
> > cursor.close()
> > connection.commit() # save changes to file for next exercise
> > connection.close()
> > ```
> > {: .python}
> >
> > For comparison, the following program writes the random numbers
> > into the file `random_numbers.txt`:
> >
> > ```
> > from numpy.random import uniform
> >
> > random_numbers = uniform(low=10.0, high=25.0, size=100000)
> > with open('random_numbers.txt', 'w') as outfile:
> >     for number in random_numbers:
> >         # need to add linebreak \n
> >         outfile.write("{}\n".format(number))
> > ```
> > {: .python}
> {: .solution}
{: .question}

> <question-title>Filtering in SQL vs. Filtering in Python</question-title>
>
> Write a Python program that creates a new database called
> `backup.db` with the same structure as `original.db` and copies all
> the values greater than 20.0 from `original.db` to `backup.db`.
> Which is faster: filtering values in the query, or reading
> everything into memory and filtering in Python?
>
> > <solution-title></solution-title>
> > The first example reads all the data into memory and filters the
> > numbers using the if statement in Python.
> >
> > ```
> > import sqlite3
> >
> > connection_original = sqlite3.connect("original.db")
> > cursor_original = connection_original.cursor()
> > cursor_original.execute("SELECT * FROM Pressure;")
> > results = cursor_original.fetchall()
> > cursor_original.close()
> > connection_original.close()
> >
> > connection_backup = sqlite3.connect("backup.db")
> > cursor_backup = connection_backup.cursor()
> > cursor_backup.execute("CREATE TABLE Pressure (reading float not null)")
> > query = "INSERT INTO Pressure (reading) VALUES (?);"
> >
> > for entry in results:
> >     # number is saved in first column of the table
> >     if entry[0] > 20.0:
> >         cursor_backup.execute(query, entry)
> >
> > cursor_backup.close()
> > connection_backup.commit()
> > connection_backup.close()
> > ```
> >
> > In contrast the following example uses the conditional ``SELECT`` statement
> > to filter the numbers in SQL.
> > The only lines that changed are in line 5, where the values are fetched
> > from `original.db` and the for loop starting in line 15 used to insert
> > the numbers into `backup.db`.
> > Note how this version does not require the use of Python's if statement.
> >
> > ```
> > import sqlite3
> >
> > connection_original = sqlite3.connect("original.db")
> > cursor_original = connection_original.cursor()
> > cursor_original.execute("SELECT * FROM Pressure WHERE reading > 20.0;")
> > results = cursor_original.fetchall()
> > cursor_original.close()
> > connection_original.close()
> >
> > connection_backup = sqlite3.connect("backup.db")
> > cursor_backup = connection_backup.cursor()
> > cursor_backup.execute("CREATE TABLE Pressure (reading float not null)")
> > query = "INSERT INTO Pressure (reading) VALUES (?);"
> >
> > for entry in results:
> >     cursor_backup.execute(query, entry)
> >
> > cursor_backup.close()
> > connection_backup.commit()
> > connection_backup.close()
> > ```
> >
> {: .solution}
{: .question}

> <question-title>Generating Insert Statements</question-title>
>
> One of our colleagues has sent us a
> CSV
> file containing
> temperature readings by Robert Olmstead, which is formatted like
> this:
>
> ```
> Taken,Temp
> 619,-21.5
> 622,-15.5
> ```
> {: .output}
>
> Write a small Python program that reads this file in and prints out
> the SQL `INSERT` statements needed to add these records to the
> survey database.  Note: you will need to add an entry for Olmstead
> to the `Person` table.  If you are testing your program repeatedly,
> you may want to investigate SQL's `INSERT or REPLACE` command.
{: .question}


topic: data-science
tutorial: sql-advanced
title: Advanced SQL
difficulty: introductory
time_estimation: 3H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["SQL"]
content_hash: b5d50610d3a32491f4d19a9d95aaadf4
last_modified: 2026-05-15T22:28:11.119713
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-advanced/tutorial.html

content:
> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


```sql
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c http://swcarpentry.github.io/sql-novice-survey/files/survey.db
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///survey.db")
%load_ext sql
%sql sqlite:///survey.db
%config SqlMagic.displaycon=False
```

# Aggregation

We now want to calculate ranges and averages for our data.
We know how to select all of the dates from the `Visited` table:

```sql
SELECT dated FROM Visited;
```


but to combine them,
we must use an aggregation function
such as `min` or `max`.
Each of these functions takes a set of records as input,
and produces a single record as output:

```sql
SELECT min(dated) FROM Visited;
```

![SQL Aggregation](../../images/carpentries-sql/sql-aggregation.svg)

```sql
SELECT max(dated) FROM Visited;
```

`min` and `max` are just two of
the aggregation functions built into SQL.
Three others are `avg`,
`count`,
and `sum`:

```sql
SELECT avg(reading) FROM Survey WHERE quant = 'sal';
```

```sql
SELECT count(reading) FROM Survey WHERE quant = 'sal';
```

```sql
SELECT sum(reading) FROM Survey WHERE quant = 'sal';
```

We used `count(reading)` here,
but we could just as easily have counted `quant`
or any other field in the table,
or even used `count(*)`,
since the function doesn't care about the values themselves,
just how many values there are.

SQL lets us do several aggregations at once.
We can,
for example,
find the range of sensible salinity measurements:

```sql
SELECT min(reading), max(reading) FROM Survey WHERE quant = 'sal' AND reading <= 1.0;
```

We can also combine aggregated results with raw results,
although the output might surprise you:

```sql
SELECT person, count(*) FROM Survey WHERE quant = 'sal' AND reading <= 1.0;
```

Why does Lake's name appear rather than Roerich's or Dyer's?
The answer is that when it has to aggregate a field,
but isn't told how to,
the database manager chooses an actual value from the input set.
It might use the first one processed,
the last one,
or something else entirely.

Another important fact is that when there are no values to aggregate ---
for example, where there are no rows satisfying the `WHERE` clause ---
aggregation's result is "don't know"
rather than zero or some other arbitrary value:

```sql
SELECT person, max(reading), sum(reading) FROM Survey WHERE quant = 'missing';
```

One final important feature of aggregation functions is that
they are inconsistent with the rest of SQL in a very useful way.
If we add two values,
and one of them is null,
the result is null.
By extension,
if we use `sum` to add all the values in a set,
and any of those values are null,
the result should also be null.
It's much more useful,
though,
for aggregation functions to ignore null values
and only combine those that are non-null.
This behavior lets us write our queries as:

```sql
SELECT min(dated) FROM Visited;
```

instead of always having to filter explicitly:

```sql
SELECT min(dated) FROM Visited WHERE dated IS NOT NULL;
```

Aggregating all records at once doesn't always make sense.
For example,
suppose we suspect that there is a systematic bias in our data,
and that some scientists' radiation readings are higher than others.
We know that this doesn't work:

```sql
SELECT person, count(reading), round(avg(reading), 2)
FROM  Survey
WHERE quant = 'rad';
```

because the database manager selects a single arbitrary scientist's name
rather than aggregating separately for each scientist.
Since there are only five scientists,
we could write five queries of the form:

```sql
SELECT person, count(reading), round(avg(reading), 2)
FROM  Survey
WHERE quant = 'rad'
AND   person = 'dyer';
```

but this would be tedious,
and if we ever had a data set with fifty or five hundred scientists,
the chances of us getting all of those queries right is small.

What we need to do is
tell the database manager to aggregate the hours for each scientist separately
using a `GROUP BY` clause:

```sql
SELECT   person, count(reading), round(avg(reading), 2)
FROM     Survey
WHERE    quant = 'rad'
GROUP BY person;
```

`GROUP BY` does exactly what its name implies:
groups all the records with the same value for the specified field together
so that aggregation can process each batch separately.
Since all the records in each batch have the same value for `person`,
it no longer matters that the database manager
is picking an arbitrary one to display
alongside the aggregated `reading` values.

> <tip-title>Know Excel? It's just a pivot table.</tip-title>
> `GROUP BY` is basically just a pivot table for Excel users, it lets you build
> nice summary tables which aggregate your results.
>
> And if you didn't already know the Excel equivalent, now you know what to
> look for when you need it!
{: .tip}

Just as we can sort by multiple criteria at once,
we can also group by multiple criteria.
To get the average reading by scientist and quantity measured,
for example,
we just add another field to the `GROUP BY` clause:

```sql
SELECT   person, quant, count(reading), round(avg(reading), 2)
FROM     Survey
GROUP BY person, quant;
```

Note that we have added `quant` to the list of fields displayed,
since the results wouldn't make much sense otherwise.

Let's go one step further and remove all the entries
where we don't know who took the measurement:

```sql
SELECT   person, quant, count(reading), round(avg(reading), 2)
FROM     Survey
WHERE    person IS NOT NULL
GROUP BY person, quant
ORDER BY person, quant;
```


Looking more closely,
this query:

1.  selected records from the `Survey` table where the `person` field was not null;
2.  grouped those records into subsets so that the `person` and `quant` values in each subset were the same;
3.  ordered those subsets first by `person`, and then within each sub-group by `quant`; and
4.  counted the number of records in each subset, calculated the average `reading` in each, and chose a `person` and `quant` value from each (it doesn't matter which ones, since they're all equal).

> <question-title>Counting Temperature Readings</question-title>
>
> How many temperature readings did Frank Pabodie record,
> and what was their average value?
>
> > <solution-title></solution-title>
> >
> > ```
> > SELECT count(reading), avg(reading) FROM Survey WHERE quant = 'temp' AND person = 'pb';
> > ```
> >
> > |count(reading)|avg(reading)|
> > |--------------|------------|
> > |2             |-20.0       |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Averaging with NULL</question-title>
>
> The average of a set of values is the sum of the values
> divided by the number of values.
> Does this mean that the `avg` function returns 2.0 or 3.0
> when given the values 1.0, `null`, and 5.0?
>
> > <solution-title></solution-title>
> > The answer is 3.0.
> > `NULL` is not a value; it is the absence of a value.
> > As such it is not included in the calculation.
> >
> > You can confirm this, by executing this code:
> >
> > ```
> > SELECT AVG(a) FROM (
> >     SELECT 1 AS a
> >     UNION ALL SELECT NULL
> >     UNION ALL SELECT 5);
> > ```
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>What Does This Query Do?</question-title>
>
> We want to calculate the difference between
> each individual radiation reading
> and the average of all the radiation readings.
> We write the query:
>
> ```
> SELECT reading - avg(reading) FROM Survey WHERE quant = 'rad';
> ```
>
> What does this actually produce, and can you think of why?
>
> > <solution-title></solution-title>
> > The query produces only one row of results when we what we really want is a result for each of the readings.
> > The `avg()` function produces only a single value, and because it is run first, the table is reduced to a single row.
> > The `reading` value is simply an arbitrary one.
> >
> > To achieve what we wanted, we would have to run two queries:
> >
> > ```
> > SELECT avg(reading) FROM Survey WHERE quant='rad';
> > ```
> >
> > This produces the average value (6.5625), which we can then insert into a second query:
> >
> > ```
> > SELECT reading - 6.5625 FROM Survey WHERE quant = 'rad';
> > ```
> >
> > This produces what we want, but we can combine this into a single query using subqueries.
> >
> > ```
> > SELECT reading - (SELECT avg(reading) FROM Survey WHERE quant='rad') FROM Survey WHERE quant = 'rad';
> > ```
> >
> > This way we don't have execute two queries.
> >
> > In summary what we have done is to replace `avg(reading)` with `(SELECT avg(reading) FROM Survey WHERE quant='rad')` in the original query.
> >
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Ordering When Concatenating</question-title>
>
> The function `group_concat(field, separator)`
> concatenates all the values in a field
> using the specified separator character
> (or ',' if the separator isn't specified).
> Use this to produce a one-line list of scientists' names,
> such as:
>
> ```
> William Dyer, Frank Pabodie, Anderson Lake, Valentina Roerich, Frank Danforth
> ```
>
> Can you find a way to order the list by surname?
{: .question}

```sql
-- Try solutions here!
```


# Combining Data

In order to submit our data to a web site
that aggregates historical meteorological data,
we might need to format it as
latitude, longitude, date, quantity, and reading.
However,
our latitudes and longitudes are in the `Site` table,
while the dates of measurements are in the `Visited` table
and the readings themselves are in the `Survey` table.
We need to combine these tables somehow.

This figure shows the relations between the tables:

![Survey Database Structure](../../images/carpentries-sql/sql-join-structure.svg)

The SQL command to do this is `JOIN`.
To see how it works,
let's start by joining the `Site` and `Visited` tables:

```sql
SELECT * FROM Site JOIN Visited;
```

`JOIN` creates
the cross product
of two tables,
i.e.,
it joins each record of one table with each record of the other table
to give all possible combinations.
Since there are three records in `Site`
and eight in `Visited`,
the join's output has 24 records (3 * 8 = 24) .
And since each table has three fields,
the output has six fields (3 + 3 = 6).

What the join *hasn't* done is
figure out if the records being joined have anything to do with each other.
It has no way of knowing whether they do or not until we tell it how.
To do that,
we add a clause specifying that
we're only interested in combinations that have the same site name,
thus we need to use a filter:

```sql
SELECT * FROM Site JOIN Visited ON Site.name = Visited.site;
```

`ON` is very similar to `WHERE`,
and for all the queries in this lesson you can use them interchangeably.
There are differences in how they affect [outer joins][outer],
but that's beyond the scope of this lesson.
Once we add this to our query,
the database manager throws away records
that combined information about two different sites,
leaving us with just the ones we want.

Notice that we used `Table.field` to specify field names
in the output of the join.
We do this because tables can have fields with the same name,
and we need to be specific which ones we're talking about.
For example,
if we joined the `Person` and `Visited` tables,
the result would inherit a field called `id`
from each of the original tables.

We can now use the same dotted notation
to select the three columns we actually want
out of our join:

```sql
SELECT Site.lat, Site.long, Visited.dated
FROM   Site JOIN Visited
ON     Site.name = Visited.site;
```


If joining two tables is good,
joining many tables must be better.
In fact,
we can join any number of tables
simply by adding more `JOIN` clauses to our query,
and more `ON` tests to filter out combinations of records
that don't make sense:

```sql
SELECT Site.lat, Site.long, Visited.dated, Survey.quant, Survey.reading
FROM   Site JOIN Visited JOIN Survey
ON     Site.name = Visited.site
AND    Visited.id = Survey.taken
AND    Visited.dated IS NOT NULL;
```

We can tell which records from `Site`, `Visited`, and `Survey`
correspond with each other
because those tables contain
primary keys
and foreign keys.
A primary key is a value,
or combination of values,
that uniquely identifies each record in a table.
A foreign key is a value (or combination of values) from one table
that identifies a unique record in another table.
Another way of saying this is that
a foreign key is the primary key of one table
that appears in some other table.
In our database,
`Person.id` is the primary key in the `Person` table,
while `Survey.person` is a foreign key
relating the `Survey` table's entries
to entries in `Person`.

Most database designers believe that
every table should have a well-defined primary key.
They also believe that this key should be separate from the data itself,
so that if we ever need to change the data,
we only need to make one change in one place.
One easy way to do this is
to create an arbitrary, unique ID for each record
as we add it to the database.
This is actually very common:
those IDs have names like "student numbers" and "patient numbers",
and they almost always turn out to have originally been
a unique record identifier in some database system or other.
As the query below demonstrates,
SQLite [automatically numbers records][rowid] as they're added to tables,
and we can use those record numbers in queries:

```sql
SELECT rowid, * FROM Person;
```

> <question-title>Listing Radiation Readings</question-title>
>
> Write a query that lists all radiation readings from the DR-1 site.
> > <solution-title></solution-title>
> >
> > ```
> > SELECT Survey.reading
> > FROM Site JOIN Visited JOIN Survey
> > ON Site.name = Visited.site
> > AND Visited.id = Survey.taken
> > WHERE Site.name = 'DR-1'
> > AND Survey.quant = 'rad';
> > ```
> >
> > |reading   |
> > |----------|
> > |9.82      |
> > |7.8       |
> > |11.25     |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Where's Frank?</question-title>
>
> Write a query that lists all sites visited by people named "Frank".
> > <solution-title></solution-title>
> >
> > ```
> > SELECT DISTINCT Site.name
> > FROM Site JOIN Visited JOIN Survey JOIN Person
> > ON Site.name = Visited.site
> > AND Visited.id = Survey.taken
> > AND Survey.person = Person.id
> > WHERE Person.personal = 'Frank';
> > ```
> >
> > |name   |
> > |-------|
> > |DR-3   |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Reading Queries</question-title>
>
> Describe in your own words what the following query produces:
>
> ```
> SELECT Site.name FROM Site JOIN Visited
> ON Site.lat < -49.0 AND Site.name = Visited.site AND Visited.dated >= '1932-01-01';
> ```
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Who Has Been Where?</question-title>
>
> Write a query that shows each site with exact location (lat, long) ordered by visited date,
> followed by personal name and family name of the person who visited the site
> and the type of measurement taken and its reading. Please avoid all null values.
> Tip: you should get 15 records with 8 fields.
> > <solution-title></solution-title>
> >
> > ```
> > SELECT Site.name, Site.lat, Site.long, Person.personal, Person.family, Survey.quant, Survey.reading, Visited.dated
> > FROM Site JOIN Visited JOIN Survey JOIN Person
> > ON Site.name = Visited.site
> > AND Visited.id = Survey.taken
> > AND Survey.person = Person.id
> > WHERE Survey.person IS NOT NULL
> > AND Visited.dated IS NOT NULL
> > ORDER BY Visited.dated;
> > ```
> >
> > name   |  lat        |  long       |  personal   | family   | quant     | reading   |     dated
> >--------|-------------|-------------|-------------|----------|-----------|-----------|-----------
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   rad     |    9.82   |   1927-02-08
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   sal     |    0.13   |   1927-02-08
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   rad     |    7.8    |   1927-02-10
> >DR-1    |    -49.85   |   -128.57   |  William    | Dyer     |   sal     |    0.09   |   1927-02-10
> >DR-3    |    -47.15   |   -126.72   |  Anderson   | Lake     |   sal     |    0.05   |   1930-01-07
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   rad     |    8.41   |   1930-01-07
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   temp    |    -21.5  |   1930-01-07
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   rad     |    7.22   |   1930-01-12
> >DR-3    |    -47.15   |   -126.72   |  Anderson   | Lake     |   sal     |    0.1    |   1930-02-26
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   rad     |    4.35   |   1930-02-26
> >DR-3    |    -47.15   |   -126.72   |  Frank      | Pabodie  |   temp    |    -18.5  |   1930-02-26
> >MSK-4   |    -48.87   |   -123.4    |  Anderson   | Lake     |   rad     |    1.46   |   1932-01-14
> >MSK-4   |    -48.87   |   -123.4    |  Anderson   | Lake     |   sal     |    0.21   |   1932-01-14
> >MSK-4   |    -48.87   |   -123.4    |  Valentina  | Roerich  |   sal     |    22.5   |   1932-01-14
> >DR-1    |    -49.85   |   -128.57   |  Valentina  | Roerich  |   rad     |    11.25  |   1932-03-22
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

A good visual explanation of joins can be found [in the SQL Join Visualizer][joinref]

[outer]: https://en.wikipedia.org/wiki/Join_%28SQL%29#Outer_join
[rowid]: https://www.sqlite.org/lang_createtable.html#rowid
[joinref]: https://sql-joins.leopard.in.ua/

# Data Hygiene

Now that we have seen how joins work, we can see why the relational
model is so useful and how best to use it.  The first rule is that
every value should be atomic, i.e., not
contain parts that we might want to work with separately.  We store
personal and family names in separate columns instead of putting the
entire name in one column so that we don't have to use substring
operations to get the name's components.  More importantly, we store
the two parts of the name separately because splitting on spaces is
unreliable: just think of a name like "Eloise St. Cyr" or "Jan Mikkel
Steubart".

The second rule is that every record should have a unique primary key.
This can be a serial number that has no intrinsic meaning,
one of the values in the record (like the `id` field in the `Person` table),
or even a combination of values:
the triple `(taken, person, quant)` from the `Survey` table uniquely identifies every measurement.

The third rule is that there should be no redundant information.
For example,
we could get rid of the `Site` table and rewrite the `Visited` table like this:

|id   |lat   |long   |dated      |
|-----|------|-------|-----------|
|619  |-49.85|-128.57| 1927-02-08|
|622  |-49.85|-128.57| 1927-02-10|
|734  |-47.15|-126.72| 1930-01-07|
|735  |-47.15|-126.72| 1930-01-12|
|751  |-47.15|-126.72| 1930-02-26|
|752  |-47.15|-126.72| None    |
|837  |-48.87|-123.40| 1932-01-14|
|844  |-49.85|-128.57| 1932-03-22|

In fact,
we could use a single table that recorded all the information about each reading in each row,
just as a spreadsheet would.
The problem is that it's very hard to keep data organized this way consistent:
if we realize that the date of a particular visit to a particular site is wrong,
we have to change multiple records in the database.
What's worse,
we may have to guess which records to change,
since other sites may also have been visited on that date.

The fourth rule is that the units for every value should be stored explicitly.
Our database doesn't do this,
and that's a problem:
Roerich's salinity measurements are several orders of magnitude larger than anyone else's,
but we don't know if that means she was using parts per million instead of parts per thousand,
or whether there actually was a saline anomaly at that site in 1932.

Stepping back,
data and the tools used to store it have a symbiotic relationship:
we use tables and joins because it's efficient,
provided our data is organized a certain way,
but organize our data that way because we have tools to manipulate it efficiently.
As anthropologists say,
the tool shapes the hand that shapes the tool.

> <question-title>Identifying Atomic Values</question-title>
>
> Which of the following are atomic values? Which are not? Why?
>
> *   New Zealand
> *   87 Turing Avenue
> *   January 25, 1971
> *   the XY coordinate (0.5, 3.3)
>
> > <solution-title></solution-title>
> > New Zealand is the only clear-cut atomic value.
> >
> > The address and the XY coordinate contain more than one piece of information
> > which should be stored separately:
> > - House number, street name
> > - X coordinate, Y coordinate
> >
> > The date entry is less clear cut, because it contains month, day, and year elements.
> > However, there is a `DATE` datatype in SQL, and dates should be stored using this format.
> > If we need to work with the month, day, or year separately, we can use the SQL functions available for our database software
> > (for example [`EXTRACT`](https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions050.htm) or [`STRFTIME`](http://www.sqlite.org/lang_datefunc.html) for SQLite).
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Identifying a Primary Key</question-title>
>
> What is the primary key in this table?
> I.e., what value or combination of values uniquely identifies a record?
>
> |latitude|longitude|date      |temperature|
> |--------|---------|----------|-----------|
> |57.3    |-22.5    |2015-01-09|-14.2      |
>
> > <solution-title></solution-title>
> > Latitude, longitude, and date are all required to uniquely identify the temperature record.
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

# Creating and Modifying Data

So far we have only looked at how to get information out of a database,
both because that is more frequent than adding information,
and because most other operations only make sense
once queries are understood.
If we want to create and modify data,
we need to know two other sets of commands.

The first pair are [`CREATE TABLE`][create-table] and [`DROP TABLE`][drop-table].
While they are written as two words,
they are actually single commands.
The first one creates a new table;
its arguments are the names and types of the table's columns.
For example,
the following statements create the four tables in our survey database:

```sql
CREATE TABLE Person(id text, personal text, family text);
CREATE TABLE Site(name text, lat real, long real);
CREATE TABLE Visited(id integer, site text, dated text);
CREATE TABLE Survey(taken integer, person text, quant text, reading real);
```

We can get rid of one of our tables using:

```sql
DROP TABLE Survey;
```

Be very careful when doing this:
if you drop the wrong table, hope that the person maintaining the database has a backup,
but it's better not to have to rely on it.

Different database systems support different data types for table columns,
but most provide the following:

|data type|  use                                       |
|---------|  ----------------------------------------- |
|INTEGER  |  a signed integer                          |
|REAL     |  a floating point number                   |
|TEXT     |  a character string                        |
|BLOB     |  a "binary large object", such as an image |

Most databases also support Booleans and date/time values;
SQLite uses the integers 0 and 1 for the former,
and represents the latter as text or numeric fields.

An increasing number of databases also support geographic data types,
such as latitude and longitude.
Keeping track of what particular systems do or do not offer,
and what names they give different data types,
is an unending portability headache.

> <tip-title>Which database should I use?</tip-title>
> SQLite is fantastic for small databases or embedded into applications where
> you want to be able to use SQL to query and process data.
>
> However for any real analysis PostgreSQL is usually the best choice, it
> scales incredibly well and can meet a wide range of use cases. It has good
> data type support.
{: .tip}

> <tip-title>Do you have geographic data?</tip-title>
> Use Postgres. The [PostGIS](https://postgis.net/) library is fantastic and industry standard for storing geographic data in a database.
{: .tip}

When we create a table,
we can specify several kinds of constraints on its columns.
For example,
a better definition for the `Survey` table would be:

```sql
CREATE TABLE Survey(
    taken   integer not null, -- where reading taken
    person  text,             -- may not know who took it
    quant   text not null,    -- the quantity measured
    reading real not null,    -- the actual reading
    primary key(taken, quant),
    foreign key(taken) references Visited(id),
    foreign key(person) references Person(id)
);
```

Once again,
exactly what constraints are available
and what they're called
depends on which database manager we are using.

Once tables have been created,
we can add, change, and remove records using our other set of commands,
`INSERT`, `UPDATE`, and `DELETE`.

Here is an example of inserting rows into the `Site` table:

```sql
INSERT INTO Site (name, lat, long) VALUES ('DR-1', -49.85, -128.57);
INSERT INTO Site (name, lat, long) VALUES ('DR-3', -47.15, -126.72);
INSERT INTO Site (name, lat, long) VALUES ('MSK-4', -48.87, -123.40);
```

We can also insert values into one table directly from another:

```sql
CREATE TABLE JustLatLong(lat real, long real);
INSERT INTO JustLatLong SELECT lat, long FROM Site;
```

Modifying existing records is done using the `UPDATE` statement.
To do this we tell the database which table we want to update,
what we want to change the values to for any or all of the fields,
and under what conditions we should update the values.

For example, if we made a mistake when entering the lat and long values
of the last `INSERT` statement above, we can correct it with an update:

```sql
UPDATE Site SET lat = -47.87, long = -122.40 WHERE name = 'MSK-4';
```

Be careful to not forget the `WHERE` clause or the update statement will
modify *all* of the records in the database.

Deleting records can be a bit trickier,
because we have to ensure that the database remains internally consistent.
If all we care about is a single table,
we can use the `DELETE` command with a `WHERE` clause
that matches the records we want to discard.
For example,
once we realize that Frank Danforth didn't take any measurements,
we can remove him from the `Person` table like this:

```sql
DELETE FROM Person WHERE id = 'danforth';
```

But what if we removed Anderson Lake instead?
Our `Survey` table would still contain seven records
of measurements he'd taken,
but that's never supposed to happen:
`Survey.person` is a foreign key into the `Person` table,
and all our queries assume there will be a row in the latter
matching every value in the former.

This problem is called referential integrity:
we need to ensure that all references between tables can always be resolved correctly.
One way to do this is to delete all the records
that use `'lake'` as a foreign key
before deleting the record that uses it as a primary key.
If our database manager supports it,
we can automate this
using cascading delete.
However,
this technique is outside the scope of this chapter.

> <tip-title>Hybrid Storage Models</tip-title>
>
> Many applications use a hybrid storage model
> instead of putting everything into a database:
> the actual data (such as astronomical images) is stored in files,
> while the database stores the files' names,
> their modification dates,
> the region of the sky they cover,
> their spectral characteristics,
> and so on.
> This is also how most music player software is built:
> the database inside the application keeps track of the MP3 files,
> but the files themselves live on disk.
{: .tip}

> <question-title>Replacing NULL</question-title>
>
> Write an SQL statement to replace all uses of `null` in
> `Survey.person` with the string `'unknown'`.
>
> > <solution-title></solution-title>
> > ```
> > UPDATE Survey SET person = 'unknown' WHERE person IS NULL;
> > ```
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Backing Up with SQL</question-title>
>
> SQLite has several administrative commands that aren't part of the
> SQL standard.  One of them is `.dump`, which prints the SQL commands
> needed to re-create the database.  Another is `.read`, which reads a
> file created by `.dump` and restores the database.  A colleague of
> yours thinks that storing dump files (which are text) in version
> control is a good way to track and manage changes to the database.
> What are the pros and cons of this approach?  (Hint: records aren't
> stored in any particular order.)
>
> > <solution-title></solution-title>
> > #### Advantages
> > - A version control system will be able to show differences between versions
> > of the dump file; something it can't do for binary files like databases
> > - A VCS only saves changes between versions, rather than a complete copy of
> > each version (save disk space)
> > - The version control log will explain the reason for the changes in each version
> > of the database
> >
> > #### Disadvantages
> > - Artificial differences between commits because records don't have a fixed order
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

{% include _includes/quiz.html id="recap.yml" %}

[create-table]: https://www.sqlite.org/lang_createtable.html
[drop-table]: https://www.sqlite.org/lang_droptable.html


topic: data-science
tutorial: git-cli
title: Basics of using Git from the Command Line
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: 30b096c4bd0bb6d173b67c83443638ad
last_modified: 2026-05-15T22:28:10.993207
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/git-cli/tutorial.html

content:
Version control is a way of tracking the change history of a project. Even if you have never used a version control tool, you've probably already done it manually: copying and renaming project folders ("paper-1.doc", "paper-2.doc", etc.) is a form of version control. Within bioinformatics (from research, to development, to sysadmin) a lot of us are using `git` as our primary method of source control for everything we do: notes, slides, tutorials, code, notebooks, ansible, system configuration, and more.

> <comment-title>Source</comment-title>
> This tutorial contains text from [this tutorial](https://github.com/rdadolf/git-tutorial/) by Robert Adolf ([@rdadolf](https://github.com/rdadolf)), which is licensed CC-BY.
{: .comment}

Git is a tool that automates and enhances a lot of the tasks that arise when dealing with larger, longer-living, and collaborative projects. It's also become the common underpinning to many popular online code repositories, GitHub being the most popular.

While it can be used collaboratively, this tutorial focuses on a single-user git repository for the most basic operations.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Why should you use version control?

If you ask 10 people, you'll get 10 different answers, but one of the commonalities is that most people don't realize how integral it is to their development process until they've started using it. Still, for the sake of argument, here are some highlights:

- **You can undo anything**: Git provides a complete history of every change that has ever been made to your project, timestamped, commented, and attributed. If something breaks, you always have the choice of going back to a previous tate.
- **You won't need to keep undo-ing things**: One of the advantages of using git properly is that by keeping new changes separate from a stable base, you tend to avoid the massive rollbacks associated with constantly tinkering with a single code.
- **You can identify exactly when and where changes were made** (and by whom!): Git allows you to pinpoint when a particular piece of code was changed, so finding what other pieces of code a bug might affect or figuring out why a certain expression was added is easy.
- **Git forces teams to face conflicts directly**: On a team-based project, many people are often working with the same code. By having a tool which understands when and where files were changed, it's easy to see when changes might conflict with each other. While it might seem troublesome sometimes to have to deal with conflicts, the alternative—not knowing there's a conflict—is much more insidious.

## Pre-requisites

[You will need to install git, if you have not done so already.](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)

## Setting up a Repository

Let's create a new repository.

> <hands-on-title>Create a Repository</hands-on-title>
> 1. Make a new directory where you will store your files, and navigate into it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > mkdir git-tutorial; cd git-tutorial;
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Create or "initialise" the `git` repository with the `git init` command.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git init
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > Initialized empty Git repository in /tmp/project/.git/
>    > ```
>    {: .code-out}
>
{: .hands_on}

This has created a folder `.git` in your project directory, here is where `git` stores all of it's data that it needs to track repository changes over time. It's not terribly interesting yet though!

> <hands-on-title>What's the status</hands-on-title>
> 1. You can always check the status of a repository with `git status`
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git status
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > On branch main
>    >
>    > No commits yet
>    >
>    > nothing to commit (create/copy files and use "git add" to track)
>    >
>    > ```
>    {: .code-out}
>
{: .hands_on}

## Adding Files

Let's add our first file, often a (pretty empty) readme file.

> <hands-on-title>What's the status</hands-on-title>
> 1. Create a new file, `readme.md` with some basic content
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "My Project" > readme.md
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Add a file with `git add`. This adds it to git's staging area to be committed.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git add readme.md
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. Commit the file! This will add it to git's log.
>
>    > <tip-title>What makes a good commit message?</tip-title>
>    > It depends a lot on the community, some have specific style guides they enforce, some don't, but in general
>    > - Keep the description short (<72 chars) and descriptive.
>    > - If you need, provide a long description as well, explaining your changes. (Use `git commit` without the `-m` flag!)
>    > [A lot has been written](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) about good commit messages, search the internet and find ideas for what you think makes a good commit message!
>    >
>    > And beware of the trap we all fall into sometimes, [unhelpful commit messages](https://xkcd.com/1296/)
>    > ![xkcd comic with commits in a table from 14h to 2h ago, starting with extremely useful commits like "created main loop & timing control", and becoming extremely unhelpful messages for the latest commits, like 'AAAAAA' or "asdfasdf".](../../images/xkcd/git_commit.png)
>    > Even your author is [very](https://github.com/galaxyproject/gxadmin/commit/568adb703189d98da701e35c3414be35ea106184), [very](https://github.com/galaxyproject/training-material/commit/611f82d267f1426aa09863c2ac9c9006f38e3c4c) guilty of this, but you can do better!
>    {: .tip}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git commit -m "Add readme"
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-out-title></code-out-title>
>    > ```bash
>    > [main (root-commit) f5ec14f] Add readme
>    >  1 file changed, 1 insertion(+)
>    >  create mode 100644 readme.md
>    > ```
>    {: .code-out}
>
> > <question-title>Is there anything left to do? Check the status</question-title>
> > Check `git status` to see if there's anything else left to resolve.
> > > <solution-title></solution-title>
> > > ```console
> > > $ git status
> > > On branch main
> > > nothing to commit, working tree clean
> > > ```
> > {: .solution}
> {: .question}
>
{: .hands_on}

Congratulations! You've made your first commit. The output of the commit command lists everything you've just done:

```
[main (root-commit) f5ec14f] Add readme
 1 file changed, 1 insertion(+)
 create mode 100644 readme.md
```

`f5ec14f` is the **commit id**, every commit you make is given a hash which uniquely refers to that specific commit. Next we see our commit message `Add readme`, a brief mention of how many files we've changed, and how many insertions or deletions we've made to the text, and lastly which files we've added.

### Exercise: Make some more commits

> <hands-on-title>Make some more commits</hands-on-title>
>
> 1. Add your name to the `readme.md` and commit your changes.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "Author: hexylena" >> readme.md
>    > git add readme.md
>    > git commit -m 'Add author name'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Make up a project description, add it to the readme, and commit.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "This project enables stakeholders to experience synergistic effects and increase link up opportunities to improve quarterly and YOY ROI.\n" >> readme.md
>    > git add readme.md
>    > git commit -m 'Add project description'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 3. [Pick a license](https://spdx.org/licenses/) for your project, and mention it in the `readme.md`, and commit.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "# License\nAGPL-3.0" >> readme.md
>    > git add readme.md
>    > git commit -m 'Add project license'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

After this step you should have ~3 commits to work with!

## Logs

One of the most helpful things about git is that, if you have written good commit messages, you can tell what you did and when!

> <hands-on-title>Check the Receipts</hands-on-title>
> 1. Check the `log` with `git log`. Notice that you can see each commit in reverse chronological order (newest at top), who made the commit, when, and what the commit message was.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git log
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```
>    > commit 5d05eb3ec22fd49282b585c60ef8f983d68c2fd7
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:13:21 2022 +0200
>    >
>    >     Add project license
>    >
>    > commit 62f974ec5f538232f65b016cf073815349364efa
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:13:16 2022 +0200
>    >
>    >     Add project description
>    >
>    > commit 10355c019c04052c15a95a817de04f9ea0ec336c
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:13:11 2022 +0200
>    >
>    >     Add author name
>    >
>    > commit f5ec14f05384d76812fc0576df5e4af79336f4e6
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 11:59:23 2022 +0200
>    >
>    >     Add readme
>    > ```
>    {: .code-in}
>
{: .hands_on}

The output of `git log` is a great way to help you remember what you were doing.

> <hands-on-title>`git log -p`</hands-on-title>
> 1. Use `git log -p` to see the log, along with which lines were changed in each commit.
{: .hands_on}

But currently this log is pretty boring, so let's replace a line and see how that looks.

> <hands-on-title>Replace a line</hands-on-title>
> 1. Update your project description in the `readme.md`, you've been told you need to support completely different features.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > sed -i s'/enables.*ROI/creates baking recipes/g' readme.md
>    > git add readme.md
>    > git commit -m 'Update project description'
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Check what happened with the `git log -p`:
>
>    > <code-out-title>Output</code-out-title>
>    > ```bash
>    > $ git log -p
>    > commit 416a121dfcda14de0c2cb181f298b2c08950475f (HEAD -> main)
>    > Author: Helena Rasche <hxr@hx42.org>
>    > Date:   Mon Jun 13 12:18:00 2022 +0200
>    >
>    >     Update project description
>    >
>    > diff --git a/readme.md b/readme.md
>    > index befc0c9..3b8899e 100644
>    > --- a/readme.md
>    > +++ b/readme.md
>    > @@ -1,6 +1,6 @@
>    >  My Project
>    >  Author: hexylena
>    > -This project enables stakeholders to experience synergistic effects and increase link up opportunities to improve quarterly and YOY ROI.
>    > +This project creates baking recipes.
>    >
>    >  # License
>    >  AGPL-3.0
>    >
>    > ```
>    {: .code-out}
>
>    This is a diff, a comparison between two versions of a file.
>
>    {% snippet topics/admin/faqs/diffs.md %}
>
{: .hands_on}

### Who did that? `git blame` to the rescue

If you want to know who changed a specific line of a file, you can use `git blame` to find out it was probably your fault (as most of us experience when we check the logs.)

> <code-in-title>Bash</code-in-title>
> ```bash
> git blame readme.md
> ```
> {: data-cmd="true"}
{: .code-in}

> <code-in-title>Output</code-in-title>
> ```bash
> ^f5ec14f (Helena Rasche 2022-06-13 11:59:23 +0200 1) My Project
> 10355c01 (Helena Rasche 2022-06-13 12:13:11 +0200 2) Author: hexylena
> 416a121d (Helena Rasche 2022-06-13 12:18:00 +0200 3) This project creates baking recipes.
> 62f974ec (Helena Rasche 2022-06-13 12:13:16 +0200 4)
> 5d05eb3e (Helena Rasche 2022-06-13 12:13:21 +0200 5) # License
> 5d05eb3e (Helena Rasche 2022-06-13 12:13:21 +0200 6) AGPL-3.0
> ```
> here we can see for every line: which commit last affected it, who made that commit, and when.
{: .code-in}

## Branching

Git has the concept of branches which are most often used to manage development over time, before it's considered final. Until now you've seen `main` in your commits and commit logs (or maybe `master` if your git installation is a bit older.)

Oftentimes you'll see this pattern:

1. There is a main branch with a lot of history
1. You want to test out a new option, new configuration, new script you're working on
	1. So you make a branch
	1. Work on that branch
1. And merge it back into the `main` branch, once it's done.

This is especially relevant for any project that is shared with others, has a public view, or a deployed version of the code. There you don't want to affect anyone else using the project, or you don't want to affect the production deployment, until you're done making your changes.

> <hands-on-title>Create a new branch</hands-on-title>
>
> 1. `git switch -c <branch>` is the command used to create a new branch and switch to it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch -c test
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    > Switched to a new branch 'test'
>    > ```
>    {: .code-in}
>
{: .hands_on}

If you look around, you'll notice everything looks exactly the same! But in fact we are now on a different branch:

> <hands-on-title>See available branches</hands-on-title>
>
> 1. `git branch` lists our available branches, and puts an asterisk next to the one we're currently on.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git branch
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    >   main
>    > * test
>    >	```
>    {: .code-in}
>
{: .hands_on}

We're now on the `test` branch, so let's make a commit.

> <hands-on-title>Add a new file</hands-on-title>
> 1. Add a new file, let's call it `docs.md`. Write something into it, it doesn't matter much what.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > echo "# Project Documentation" > docs.md
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Add it, commit it.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git add docs.md
>    > git commit -m "Added documentation"
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

This file now only exists on the testing branch.

> <hands-on-title>Try Switching Branches</hands-on-title>
> 1. Try switching back and forth between the `main` and `test` branches, and check what's available on each!
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git branch
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch main
>    > ls
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```console
>    > readme.md
>    > ```
>    {: .code-in}
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch test
>    > ls
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```console
>    > docs.md   readme.md
>    > ```
>    {: .code-in}
>
{: .hands_on}

Each branch has a different view of the repository, and might have different changes on it. Branches are really useful to keep track of work in progress, until it's done. In a single user environment however, most people often don't use them, but once you're collaborating with other's they're incredibly important!

### Merging

Once you're done with a branch, you can merge it into the main branch. This will take all of the work you did on that branch, and make it part of the main branch.

First, let's compare the two branches, to see what changed.

> <hands-on-title>Replacing argv.</hands-on-title>
>
> 1. Compare your current branch against the `main` branch with `git diff main`
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git diff main
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    > diff --git a/docs.md b/docs.md
>    > new file mode 100644
>    > index 0000000..384aaaa
>    > --- /dev/null
>    > +++ b/docs.md
>    > @@ -0,0 +1 @@
>    > +# Project Documentation
>    > ```
>    {: .code-in}
>
{: .hands_on}

We can see the output shows all of our changes compared to the main branch and it looks like what we want, so, let's merge it in.

> <hands-on-title>Merge the `test` branch into `main`</hands-on-title>
> 1. Switch to the main branch
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git switch main
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. Merge in the test branch
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git merge test
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
>    > <code-in-title>Output</code-in-title>
>    > ```bash
>    > Updating 416a121..9a3387d
>    > Fast-forward
>    >  docs.md | 1 +
>    >  1 file changed, 1 insertion(+)
>    >  create mode 100644 docs.md
>    > ```
>    {: .code-in}
>
{: .hands_on}

This has merged all of the changes you made on the `test` branch into the `main` branch.

> <hands-on-title>Check the history</hands-on-title>
> 1. Check `git log -p` again to see the history.
{: .hands_on}

## Undo! Revert!

Oh no, you've decided you liked your original project description better. Let's find that commit and revert it.

> <hands-on-title>Find and revert the bad commit</hands-on-title>
>
> 1. Find the commit you want to revert, e.g. with `git log`, find the one named "Update project description" (or similar.)
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git log
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
> 2. We can use the `git revert` command to undo this commit.
>
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > git revert 416a121dfcda14de0c2cb181f298b2c08950475f
>    > ```
>    > {: data-cmd="true"}
>    {: .code-in}
>
{: .hands_on}

This generates a *new* commit, which reverts the older commit (and probably puts you in a text editor to edit the commit message).
This is not the only way to undo mistakes, but probably the easiest.

If you check your `git log` you'll see the change was undone in a second commit, reverting the first. So if you just look at the current files it appears we never undid it, but within the logs we can see the undo step.

With that you've got enough skills to track your own data/code/etc with git!

# Further Reading

- [git - the simple guide](https://rogerdudler.github.io/git-guide/index.html)
- [giteveryday(7) manual page, the most common commands most folks use every day](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/giteveryday.html)
- [git flight rules, a convenient "how do I do X" guide](https://github.com/k88hudson/git-flight-rules)
- [do you prefer git tutorials as comics? (cw: language)](https://jvns.ca/blog/2018/10/27/new-zine--oh-shit--git-/)


topic: data-science
tutorial: cli-basics
title: CLI basics
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["[]"]
tags_json: ["bash"]
content_hash: b0fee311784b3865d53bdf932dad13b3
last_modified: 2026-05-15T22:28:10.897964
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/cli-basics/tutorial.html

content:
This tutorial will walk you through the basics of how to use the Unix command line.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) ["The Unix Shell"](https://swcarpentry.github.io/shell-novice/) lesson, which is licensed CC-BY 4.0. Adaptations have been made to make this work better in a GTN/Galaxy environment.
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Background


Humans and computers commonly interact in many different ways, such as through a keyboard and mouse,
touch screen interfaces, or using speech recognition systems.
The most widely used way to interact with personal computers is called a
**graphical user interface** (GUI).
With a GUI, we give instructions by clicking a mouse and using menu-driven interactions.

While the visual aid of a GUI makes it intuitive to learn,
this way of delivering instructions to a computer scales very poorly.
Imagine the following task:
for a literature search, you have to copy the third line of one thousand text files in one thousand
different directories and paste it into a single file.
Using a GUI, you would not only be clicking at your desk for several hours,
but you could potentially also commit an error in the process of completing this repetitive task.
This is where we take advantage of the Unix shell.
The Unix shell is both a **command-line interface** (CLI) and a scripting language,
allowing such repetitive tasks to be done automatically and fast.
With the proper commands, the shell can repeat tasks with or without some modification
as many times as we want.
Using the shell, the task in the literature example can be accomplished in seconds.

## Setup

Before we do anything, we'll get you setup with some test data which can help
guide your exploration of the CLI.

```bash
cd ~/
mkdir -p Desktop/
cd Desktop/
wget -c https://github.com/swcarpentry/shell-novice/raw/2929ba2cbb1bcb5ff0d1b4100c6e58b96e155fd1/data/shell-lesson-data.zip
unzip -u shell-lesson-data.zip
```


## The Shell

The shell is a program where users can type commands.
With the shell, it's possible to invoke complicated programs like climate modeling software
or simple commands that create an empty directory with only one line of code.
The most popular Unix shell is Bash (the Bourne Again SHell ---
so-called because it's derived from a shell written by Stephen Bourne).
Bash is the default shell on most modern implementations of Unix and in most packages that provide
Unix-like tools for Windows.

Using the shell will take some effort and some time to learn.
While a GUI presents you with choices to select, CLI choices are not automatically presented to you,
so you must learn a few commands like new vocabulary in a language you're studying.
However, unlike a spoken language, a small number of "words" (i.e. commands) gets you a long way,
and we'll cover those essential few today.

The grammar of a shell allows you to combine existing tools into powerful
pipelines and handle large volumes of data automatically. Sequences of
commands can be written into a *script*, improving the reproducibility of
workflows.

In addition, the command line is often the easiest way to interact with remote machines
and supercomputers.
Familiarity with the shell is near essential to run a variety of specialized tools and resources
including high-performance computing systems.
As clusters and cloud computing systems become more popular for scientific data crunching,
being able to interact with the shell is becoming a necessary skill.
We can build on the command-line skills covered here
to tackle a wide range of scientific questions and computational challenges.

Let's get started.

When the shell is first opened, you are presented with a **prompt**,
indicating that the shell is waiting for input.

```
$
```

The shell typically uses `$ ` as the prompt, but may use a different symbol.
In the examples for this lesson, we will **not** include this prompt!

Most importantly:
when typing commands, either from these lessons or from other sources,
*do not type the prompt*, only the commands that follow it.
Also note that after you type a command, you have to press the <kbd>Enter</kbd> key to execute it.

The prompt is followed by a **text cursor**, a character that indicates the position where your
typing will appear.
The cursor is usually a flashing or solid block, but it can also be an underscore or a pipe.
You may have seen it in a text editor program, for example.

So let's try our first command, `ls` which is short for listing.


> <hands-on-title>Open a Jupyter Terminal</hands-on-title>
> This tutorial will let you accomplish almost everything from this view, running code in the cells below directly in the training material. You can choose between running the code here, or opening up a terminal tab in which to run it.
>
> Here are some instructions for how to do this on various environments.
>
> **Jupyter** on UseGalaxy.\* and **MyBinder.org**
>
> 1. Use the File → New → Terminal menu to launch a terminal.
>
>    ![screenshot of jupyterlab showing the File menu expanded to show new and terminal option.](../../images/carpentries-cli/mybinder-launch.png)
>
> 2. Disable "Simple" mode in the bottom left hand corner, if it activated.
>
>    ![screenshot of jupyterlab showing a toggle labelled simple](../../images/carpentries-cli/mybinder-simple.png)
>
> 3. Drag one of the terminal or notebook tabs to the side to have the training materials and terminal side-by-side
>
>    ![screenshot of jupyterlab with notebook and terminal side-by-side.](../../images/carpentries-cli/mybinder-final.png)
>
> **CoCalc**
>
> 1. Use the *Split View* functionality of cocalc to split your view into two portions.
>
>    ![screenshot of cocalc button to split views](../../images/carpentries-cli/cocalc-split.png)
>
> 2. Change the view of one panel to a terminal
>
>    ![screenshot of cocalc swapping view port to that of a terminal](../../images/carpentries-cli/cocalc-terminal.png)
>
{: .hands_on}

This command will list the contents of the current directory:

```bash
ls
```

> <tip-title>Command not found</tip-title>
> If the shell can't find a program whose name is the command you typed, it
> will print an error message such as:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > $ ks
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > ks: command not found
> > ```
> {: .code-out}
>
> This might happen if the command was mis-typed or if the program corresponding to that command
> is not installed.
{: .tip }


## Nelle's Pipeline: A Typical Problem

Nelle Nemo, a marine biologist,
has just returned from a six-month survey of the
[North Pacific Gyre](http://en.wikipedia.org/wiki/North_Pacific_Gyre),
where she has been sampling gelatinous marine life in the
[Great Pacific Garbage Patch](http://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch).
She has 1520 samples that she's run through an assay machine to measure the relative abundance
of 300 proteins.
She needs to run these 1520 files through an imaginary program called `goostats.sh` she inherited.
On top of this huge task, she has to write up results by the end of the month so her paper
can appear in a special issue of *Aquatic Goo Letters*.

The bad news is that if she has to run `goostats.sh` by hand using a GUI,
she'll have to select and open a file 1520 times.
If `goostats.sh` takes 30 seconds to run each file, the whole process will take more than 12 hours
of Nelle's attention.
With the shell, Nelle can instead assign her computer this mundane task while she focuses
her attention on writing her paper.

The next few lessons will explore the ways Nelle can achieve this.
More specifically,
they explain how she can use a command shell to run the `goostats.sh` program,
using loops to automate the repetitive steps of entering file names,
so that her computer can work while she writes her paper.

As a bonus,
once she has put a processing pipeline together,
she will be able to use it again whenever she collects more data.

In order to achieve her task, Nelle needs to know how to:
- navigate to a file/directory
- create a file/directory
- check the length of a file
- chain commands together
- retrieve a set of files
- iterate over files
- run a shell script containing her pipeline

# Navigating Files and Directories

The part of the operating system responsible for managing files and directories
is called the **file system**.
It organizes our data into files,
which hold information,
and directories (also called 'folders'),
which hold files or other directories.

Several commands are frequently used to create, inspect, rename, and delete files and directories.
To start exploring them, we'll go to our open shell window.

First, let's find out where we are by running a command called `pwd`
(which stands for 'print working directory'). Directories are like *places*, at any time
while we are using the shell, we are in exactly one place called
our **current working directory**. Commands mostly read and write files in the
current working directory, i.e. 'here', so knowing where you are before running
a command is important. `pwd` shows you where you are:

```bash
pwd
```

Try running it now.

> <tip-title>Home Directory Variation</tip-title>
>
> The home directory path will look different on different operating systems.
>
> Linux         | OSX            | Jupyter (UseGalaxy/Binder) | CoCalc
> ---           | --             | ---                        | ---
> `/home/nelle` | `/Users/nelle` | `/home/joyvan`             | `/projects/<id>`
>
> On Windows, it will be similar to `C:\Documents and Settings\nelle` or `C:\Users\nelle`.
> (Note that it may look slightly different for different versions of Windows.)
> In future examples, we've used Mac output as the default - Linux and Windows
> output may differ slightly but should be generally similar.
>
> We will also assume that your `pwd` command returns your user's home directory.
> If `pwd` returns something different, you may need to navigate there using `cd`
> or some commands in this lesson will not work as written.
> See [Exploring Other Directories](#exploring-other-directories) for more details
> on the `cd` command.
{: .tip}

To understand what a 'home directory' is,
let's have a look at how the file system as a whole is organized.  For the
sake of this example, we'll be
illustrating the filesystem on our scientist Nelle's computer.  After this
illustration, you'll be learning commands to explore your own filesystem,
which will be constructed in a similar way, but not be exactly identical.

On Nelle's computer, the filesystem looks like this:

![The file system is made up of a root directory that contains sub-directories titled bin, data, users, and tmp](../../images/carpentries-cli/filesystem.svg)

At the top is the **root directory**
that holds everything else.
We refer to it using a slash character, `/`, on its own;
this character is the leading slash in `/Users/nelle`.

Inside that directory are several other directories:
`bin` (which is where some built-in programs are stored),
`data` (for miscellaneous data files),
`Users` (where users' personal directories are located),
`tmp` (for temporary files that don't need to be stored long-term),
and so on.

We know that our current working directory `/Users/nelle` is stored inside `/Users`
because `/Users` is the first part of its name.
Similarly,
we know that `/Users` is stored inside the root directory `/`
because its name begins with `/`.

> <tip-title>Slashes</tip-title>
>
> Notice that there are two meanings for the `/` character.
> When it appears at the front of a file or directory name,
> it refers to the root directory. When it appears *inside* a path,
> it's just a separator.
{: .tip}

Underneath `/Users`,
we find one directory for each user with an account on Nelle's machine,
her colleagues *imhotep* and *larry*.

![Like other directories, home directories are sub-directories underneath '/Users' like '/Users/imhotep', '/Users/larry' or '/Users/nelle'](../../images/carpentries-cli/home-directories.svg)

The user *imhotep*'s files are stored in `/Users/imhotep`,
user *larry*'s in `/Users/larry`,
and Nelle's in `/Users/nelle`.  Because Nelle is the user in our
examples here, therefore we get `/Users/nelle` as our home directory.
Typically, when you open a new command prompt, you will be in
your home directory to start.

Now let's learn the command that will let us see the contents of our
own filesystem.  We can see what's in our home directory by running `ls`:

```bash
ls
```

`ls` prints the names of the files and directories in the current directory.
We can make its output more comprehensible by using the `-F` **option**
(also known as a **switch** or a **flag**),
which tells `ls` to classify the output
by adding a marker to file and directory names to indicate what they are:
- a trailing `/` indicates that this is a directory
- `@` indicates a link
- `*` indicates an executable

Depending on your default options,
the shell might also use colors to indicate whether each entry is a file or
directory.

```bash
ls -F
```

Here,
we can see that our home directory contains only **sub-directories**.
Any names in our output that don't have a classification symbol
are plain old **files**.

> <tip-title>Real Terminal Tip: Clearing your terminal</tip-title>
>
> If your screen gets too cluttered, you can clear your terminal using the
> `clear` command. You can still access previous commands using <kbd>↑</kbd>
> and <kbd>↓</kbd> to move line-by-line, or by scrolling in your terminal.
{: .tip}


## Getting help

`ls` has lots of other **options**. There are two common ways to find out how
to use a command and what options it accepts:

1. We can pass a `--help` option to the command,
2. We can read its manual with `man`

### The `--help` option

Many bash commands, and programs that people have written that can be
run from within bash, support a `--help` option to display more
information on how to use the command or program.

```bash
ls --help
```

> <tip-title>Unsupported command-line options</tip-title>
> If you try to use an option (flag) that is not supported, `ls` and other commands
> will usually print an error message similar to:
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > ls -j
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > ls: invalid option -- 'j'
> > Try 'ls --help' for more information.
> > ```
> {: .code-out}
>
{: .tip}

### The `man` command

The other way to learn about `ls` is to type

> <code-in-title>Bash</code-in-title>
> ```
> $ man ls
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> LS(1)                                                          User Commands                                                          LS(1)
>
> NAME
>        ls - list directory contents
>
> SYNOPSIS
>        ls [OPTION]... [FILE]...
>
> DESCRIPTION
>        List information about the FILEs (the current directory by default).  Sort entries alphabetically if none of -cftuvSUX nor --sort is
>        specified.
>
>        Mandatory arguments to long options are mandatory for short options too.
>
>        -a, --all
>               do not ignore entries starting with .
>
>        -A, --almost-all
>               do not list implied . and ..
>
>        --author
>               with -l, print the author of each file
>
>        -b, --escape
>               print C-style escapes for nongraphic characters
>
>        --block-size=SIZE
>               with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below
>
>        -B, --ignore-backups
>               do not list implied entries ending with ~
>
>        -c     with -lt: sort by, and show, ctime (time of last modification of file status information); with -l: show ctime  and  sort  by
>               name; otherwise: sort by ctime, newest first
>
>        -C     list entries by columns
>
>        --color[=WHEN]
>               colorize the output; WHEN can be 'always' (default if omitted), 'auto', or 'never'; more info below
>
> ```
{: .code-out}

This command will turn your terminal into a page with a description
of the `ls` command and its options.

To navigate through the `man` pages,
you may use <kbd>↑</kbd> and <kbd>↓</kbd> to move line-by-line,
or try <kbd>B</kbd> and <kbd>Spacebar</kbd> to skip up and down by a full page.
To search for a character or word in the `man` pages,
use <kbd>/</kbd> followed by the character or word you are searching for.
Sometimes a search will result in multiple hits.
If so, you can move between hits using <kbd>N</kbd> (for moving forward) and
<kbd>Shift</kbd>+<kbd>N</kbd> (for moving backward).

To **quit** the `man` pages, press <kbd>Q</kbd>.

> <tip-title>Manual pages on the web</tip-title>
>
> Of course, there is a third way to access help for commands:
> searching the internet via your web browser.
> When using internet search, including the phrase `unix man page` in your search
> query will help to find relevant results.
>
> GNU provides links to its
> [manuals](http://www.gnu.org/manual/manual.html) including the
> [core GNU utilities](http://www.gnu.org/software/coreutils/manual/coreutils.html),
> which covers many commands introduced within this lesson.
{: .tip}

> <question-title>Exploring More `ls` Flags</question-title>
>
> You can also use two options at the same time. What does the command `ls` do when used
> with the `-l` option? What about if you use both the `-l` and the `-h` option?
>
> Some of its output is about properties that we do not cover in this lesson (such
> as file permissions and ownership), but the rest should be useful
> nevertheless.
>
> > <solution-title></solution-title>
> > The `-l` option makes `ls` use a **l**ong listing format, showing not only
> > the file/directory names but also additional information, such as the file size
> > and the time of its last modification. If you use both the `-h` option and the `-l` option,
> > this makes the file size '**h**uman readable', i.e. displaying something like `5.3K`
> > instead of `5369`.
> {: .solution}
{: .question}

> <question-title>Listing in Reverse Chronological Order</question-title>
>
> By default, `ls` lists the contents of a directory in alphabetical
> order by name. The command `ls -t` lists items by time of last
> change instead of alphabetically. The command `ls -r` lists the
> contents of a directory in reverse order.
> Which file is displayed last when you combine the `-t` and `-r` flags?
> Hint: You may need to use the `-l` flag to see the
> last changed dates.
>
> > <solution-title></solution-title>
> > The most recently changed file is listed last when using `-rt`. This
> > can be very useful for finding your most recent edits or checking to
> > see if a new output file was written.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

## Exploring Other Directories

Not only can we use `ls` on the current working directory,
but we can use it to list the contents of a different directory.
Let's take a look at our current directory by running `ls -F ~/`,
i.e.,
the command `ls` with the `-F` **option** and the [**argument**][Arguments]  `/`.
The argument `~/` tells `ls` that
we want a listing of something other than our current working directory:

```bash
ls -F ~/
```

As you may now see, using a bash shell is strongly dependent on the idea that
your files are organized in a hierarchical file system.
Organizing things hierarchically in this way helps us keep track of our work:
it's possible to put hundreds of files in our home directory,
just as it's possible to pile hundreds of printed papers on our desk,
but it's a self-defeating strategy.

Now that we know the `shell-lesson-data` directory is located in current working directory, we
can do two things.

First, we can look at its contents, using the same strategy as before, passing
a directory name to `ls`:

```bash
ls -F shell-lesson-data
```

Second, we can actually change our location to a different directory, so
we are no longer located in
our home directory.

The command to change locations is `cd` followed by a
directory name to change our working directory.
`cd` stands for 'change directory',
which is a bit misleading:
the command doesn't change the directory;
it changes the shell's idea of what directory we are in.
The `cd` command is akin to double clicking a folder in a graphical interface to get into a folder.

Let's say we want to move to the `data` directory we saw above. We can
use the following series of commands to get there:

```bash
cd shell-lesson-data
```

These commands will move us into
the `shell-lesson-data` directory, then into the `data` directory.
You will notice that `cd` doesn't print anything. This is normal.
Many shell commands will not output anything to the screen when successfully executed.
But if we run `pwd` after it, we can see that we are now
in `/Users/nelle/Desktop/shell-lesson-data/data`.
If we run `ls -F` without arguments now,
it lists the contents of `/Users/nelle/Desktop/shell-lesson-data/data`,
because that's where we now are:

```bash
pwd
```

```bash
ls -F data/
```

We now know how to go down the directory tree (i.e. how to go into a subdirectory),
but how do we go up (i.e. how do we leave a directory and go into its parent directory)?
We might try the following:

```bash
cd shell-lesson-data
```

But we get an error! Why is this?

With our methods so far,
`cd` can only see sub-directories inside your current directory. There are
different ways to see directories above your current location; we'll start
with the simplest.

There is a shortcut in the shell to move up one directory level
that looks like this:

```bash
cd ..
```

`..` is a special directory name meaning
"the directory containing this one",
or more succinctly,
the **parent** of the current directory.
Sure enough,
if we run `pwd` after running `cd ..`, we're back in `/Users/nelle/Desktop/shell-lesson-data`:

```bash
pwd
```

The special directory `..` doesn't usually show up when we run `ls`. If we want
to display it, we can add the `-a` option to `ls -F`:

```bash
ls -F -a
```

`-a` stands for 'show all';
it forces `ls` to show us file and directory names that begin with `.`,
such as `..` (which, if we're in `/Users/nelle`, refers to the `/Users` directory).
As you can see,
it also displays another special directory that's just called `.`,
which means 'the current working directory'.
It may seem redundant to have a name for it,
but we'll see some uses for it soon.

Note that in most command line tools, multiple options can be combined
with a single `-` and no spaces between the options: `ls -F -a` is
equivalent to `ls -Fa`.

> <tip-title>Other Hidden Files</tip-title>
>
> In addition to the hidden directories `..` and `.`, you may also see a file
> called `.bash_profile`. This file usually contains shell configuration
> settings. You may also see other files and directories beginning
> with `.`. These are usually files and directories that are used to configure
> different programs on your computer. The prefix `.` is used to prevent these
> configuration files from cluttering the terminal when a standard `ls` command
> is used.
{: .tip}

These three commands are the basic commands for navigating the filesystem on your computer:
`pwd`, `ls`, and `cd`. Let's explore some variations on those commands. What happens
if you type `cd` on its own, without giving
a directory?

```bash
cd
```

How can you check what happened? `pwd` gives us the answer!

```bash
pwd
```

It turns out that `cd` without an argument will return you to your home directory,
which is great if you've gotten lost in your own filesystem.

Let's try returning to the `data` directory from before. Last time, we used
three commands, but we can actually string together the list of directories
to move to `data` in one step:

```bash
cd ~/Desktop/shell-lesson-data/data
```

Check that we've moved to the right place by running `pwd` and `ls -F`.

If we want to move up one level from the data directory, we could use `cd ..`.  But
there is another way to move to any directory, regardless of your
current location.

So far, when specifying directory names, or even a directory path (as above),
we have been using **relative paths**.  When you use a relative path with a command
like `ls` or `cd`, it tries to find that location from where we are,
rather than from the root of the file system.

However, it is possible to specify the **absolute path** to a directory by
including its entire path from the root directory, which is indicated by a
leading slash. The leading `/` tells the computer to follow the path from
the root of the file system, so it always refers to exactly one directory,
no matter where we are when we run the command.

This allows us to move to our `shell-lesson-data` directory from anywhere on
the filesystem (including from inside `data`). To find the absolute path
we're looking for, we can use `pwd` and then extract the piece we need
to move to `shell-lesson-data`.

> <code-in-title>Bash</code-in-title>
> ```
> $ pwd
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```bash
> /Users/nelle/
> ```
{: .code-out}

Then we could run something like:

```
$ cd /Users/nelle/shell-lesson-data
```

Run `pwd` and `ls -F` to ensure that we're in the directory we expect. If we're not, edit the command block below to make sure we're in the correct location (`~/Desktop/shell-lesson-data/`)

```
pwd
ls -F
```

> <tip-title>Two More Shortcuts</tip-title>
>
> The shell interprets a tilde (`~`) character at the start of a path to
> mean "the current user's home directory". For example, if Nelle's home
> directory is `/Users/nelle`, then `~/data` is equivalent to
> `/Users/nelle/data`. This only works if it is the first character in the
> path: `here/there/~/elsewhere` is *not* `here/there/Users/nelle/elsewhere`.
>
> Another shortcut is the `-` (dash) character. `cd` will translate `-` into
> *the previous directory I was in*, which is faster than having to remember,
> then type, the full path.  This is a *very* efficient way of moving
> *back and forth between two directories* -- i.e. if you execute `cd -` twice,
> you end up back in the starting directory.
>
> The difference between `cd ..` and `cd -` is
> that the former brings you *up*, while the latter brings you *back*.
>
> ----
> Try it out below!
{: .tip}

First navigate to `~/Desktop/shell-lesson-data` (you should already be there).

```bash
cd ~/Desktop/shell-lesson-data
```

Then `cd` into the `creatures` directory
```bash
cd creatures
```

Now if you run
```bash
cd -
```
you'll see you're back in `~/Desktop/shell-lesson-data`.
Run `cd -` again and you're back in `~/Desktop/shell-lesson-data/creatures`

> <question-title>Absolute vs Relative Paths</question-title>
>
> Starting from `/Users/amanda/data`,
> which of the following commands could Amanda use to navigate to her home directory,
> which is `/Users/amanda`?
>
> 1. `cd .`
> 2. `cd /`
> 3. `cd /home/amanda`
> 4. `cd ../..`
> 5. `cd ~`
> 6. `cd home`
> 7. `cd ~/data/..`
> 8. `cd`
> 9. `cd ..`
>
> > <solution-title></solution-title>
> > 1. No: `.` stands for the current directory.
> > 2. No: `/` stands for the root directory.
> > 3. No: Amanda's home directory is `/Users/amanda`.
> > 4. No: this command goes up two levels, i.e. ends in `/Users`.
> > 5. Yes: `~` stands for the user's home directory, in this case `/Users/amanda`.
> > 6. No: this command would navigate into a directory `home` in the current directory, if it exists.
> > 7. Yes: unnecessarily complicated, but correct.
> > 8. Yes: shortcut to go back to the user's home directory.
> > 9. Yes: goes up one level.
> {: .solution}
{: .question}

> <question-title>Relative Path Resolution</question-title>
>
> Using the filesystem diagram below, if `pwd` displays `/Users/thing`,
> what will `ls -F ../backup` display?
>
> 1.  `../backup: No such file or directory`
> 2.  `2012-12-01 2013-01-08 2013-01-27`
> 3.  `2012-12-01/ 2013-01-08/ 2013-01-27/`
> 4.  `original/ pnas_final/ pnas_sub/`
>
> ![A directory tree below the Users directory where '/Users' contains the directories 'backup' and 'thing'; '/Users/backup' contains 'original', 'pnas_final' and 'pnas_sub'; '/Users/thing' contains 'backup'; and '/Users/thing/backup' contains '2012-12-01', '2013-01-08' and '2013-01-27'](../../images/carpentries-cli/filesystem-challenge.svg)
>
> > <solution-title></solution-title>
> > 1. No: there *is* a directory `backup` in `/Users`.
> > 2. No: this is the content of `Users/thing/backup`, but with `..`, we asked for one level further up.
> > 3. No: see previous explanation.
> > 4. Yes: `../backup/` refers to `/Users/backup/`.
> {: .solution}
{: .question}

> <question-title>`ls` Reading Comprehension</question-title>
>
> Using the filesystem diagram below,
> if `pwd` displays `/Users/backup`,
> and `-r` tells `ls` to display things in reverse order,
> what command(s) will result in the following output:
>
> ```
> pnas_sub/ pnas_final/ original/
> ```
> {: .output}
>
> ![A directory tree below the Users directory where '/Users' contains the directories 'backup' and 'thing'; '/Users/backup' contains 'original', 'pnas_final' and 'pnas_sub'; '/Users/thing' contains 'backup'; and'/Users/thing/backup' contains '2012-12-01', '2013-01-08' and '2013-01-27'](../../images/carpentries-cli/filesystem-challenge.svg)
>
> 1.  `ls pwd`
> 2.  `ls -r -F`
> 3.  `ls -r -F /Users/backup`
>
> > <solution-title></solution-title>
> >  1. No: `pwd` is not the name of a directory.
> >  2. Yes: `ls` without directory argument lists files and directories in the current directory.
> >  3. Yes: uses the absolute path explicitly.
> {: .solution}
{: .question}


## General Syntax of a Shell Command
We have now encountered commands, options, and arguments,
but it is perhaps useful to formalise some terminology.

Consider the command below as a general example of a command,
which we will dissect into its component parts:

```
$ ls -F ~/Desktop/shell-lesson-data/
```

`ls` is the **command**, with an **option** `-F` and an
**argument** `/`.
We've already encountered options (also called **switches** or **flags**) which
either start with a single dash (`-`) or two dashes (`--`),
and they change the behavior of a command.
[Arguments] tell the command what to operate on (e.g. files and directories).
Sometimes options and arguments are referred to as **parameters**.
A command can be called with more than one option and more than one argument, but a
command doesn't always require an argument or an option.

Each part is separated by spaces: if you omit the space
between `ls` and `-F` the shell will look for a command called `ls-F`, which
doesn't exist. Also, capitalization can be important.
For example, `ls -s` will display the size of files and directories alongside the names,
while `ls -S` will sort the files and directories by size, as shown below:

```
$ ls -s Desktop/shell-lesson-data/data
total 116
 4 amino-acids.txt   4 animals.txt   4 morse.txt  12 planets.txt  76 sunspot.txt
 4 animal-counts     4 elements      4 pdb         4 salmon.txt
$ ls -S Desktop/shell-lesson-data/data
sunspot.txt  animal-counts  pdb        amino-acids.txt  salmon.txt
planets.txt  elements       morse.txt  animals.txt
```

Putting all that together, our command above gives us a listing
of files and directories in our shell-lesson-data folder.


### Nelle's Pipeline: Organizing Files

Knowing this much about files and directories,
Nelle is ready to organize the files that the protein assay machine will create.
First,
she creates a directory called `north-pacific-gyre`
(to remind herself where the data came from).
Inside that,
she creates a directory called `2012-07-03`,
which is the date she started processing the samples.
She used to use names like `conference-paper` and `revised-results`,
but she found them hard to understand after a couple of years.
(The final straw was when she found herself creating
a directory called `revised-revised-results-3`.)

> <tip-title>Sorting Output</tip-title>
>
> Nelle names her directories 'year-month-day',
> with leading zeroes for months and days,
> because the shell displays file and directory names in alphabetical order.
> If she used month names,
> December would come before July;
> if she didn't use leading zeroes,
> November ('11') would come before July ('7'). Similarly, putting the year first
> means that June 2012 will come before June 2013.
{: .tip}

Each of her physical samples is labelled according to her lab's convention
with a unique ten-character ID,
such as 'NENE01729A'.
This ID is what she used in her collection log
to record the location, time, depth, and other characteristics of the sample,
so she decides to use it as part of each data file's name.
Since the assay machine's output is plain text,
she will call her files `NENE01729A.txt`, `NENE01812A.txt`, and so on.
All 1520 files will go into the same directory.

Now in her current directory `shell-lesson-data`,
Nelle can see what files she has using the command:

```bash
ls north-pacific-gyre/2012-07-03/
```

This command is a lot to type,
but she can let the shell do most of the work through what is called **tab completion**.
If she types:

```bash
ls nor
```

and then presses <kbd>Tab</kbd> (the tab key on her keyboard),
the shell automatically completes the directory name for her:

```
$ ls north-pacific-gyre/
```
{: .language-bash}

If she presses <kbd>Tab</kbd> again,
Bash will add `2012-07-03/` to the command,
since it's the only possible completion.
Pressing <kbd>Tab</kbd> again does nothing,
since there are 19 possibilities;
pressing <kbd>Tab</kbd> twice brings up a list of all the files,
and so on.
This is called **tab completion**,
and we will see it in many other tools as we go on.


# Working with Files and Directories

We now know how to explore files and directories,
but how do we create them in the first place?

## Step one: see where we are and what we already have

Let's go back to our `shell-lesson-data` directory on the Desktop
and use `ls -F` to see what it contains:

```bash
pwd
```

```bash
ls -F
```

You should see folders like the following, if not, you're in the wrong place!

```
creatures/  data/  molecules/  north-pacific-gyre/  notes.txt  pizza.cfg  solar.pdf  writing/
```

## Create a directory

Let's create a new directory called `thesis` using the command `mkdir thesis`
(which has no output):

```bash
mkdir thesis
```

As you might guess from its name,
`mkdir` means 'make directory'.
Since `thesis` is a relative path
(i.e., does not have a leading slash, like `/what/ever/thesis`),
the new directory is created in the current working directory:

```bash
ls -F
```

Since we've just created the `thesis` directory, there's nothing in it yet:

```bash
ls -F thesis
```

Note that `mkdir` is not limited to creating single directories one at a time.
The `-p` option allows `mkdir` to create a directory with nested subdirectories
in a single operation:

```bash
mkdir -p project/data project/results
```

The `-R` option to the `ls` command will list all nested subdirectories within a directory.
Let's use `ls -FR` to recursively list the new directory hierarchy we just created in the
`project` directory:

```bash
ls -FR project
```

It should look like:

```
project/:
data/  results/

project/data:

project/results:
```

> <tip-title>Two ways of doing the same thing</tip-title>
> Using the shell to create a directory is no different than using a file explorer.
> If you open the current directory using your operating system's graphical file explorer,
> the `thesis` directory will appear there too.
> While the shell and the file explorer are two different ways of interacting with the files,
> the files and directories themselves are the same.
{: .tip}

> <tip-title>Good names for files and directories</tip-title>
>
> Complicated names of files and directories can make your life painful
> when working on the command line. Here we provide a few useful
> tips for the names of your files and directories.
>
> 1. Don't use spaces.
>
>    Spaces can make a name more meaningful,
>    but since spaces are used to separate arguments on the command line
>    it is better to avoid them in names of files and directories.
>    You can use `-` or `_` instead (e.g. `north-pacific-gyre/` rather than `north pacific gyre/`).
>    To test this out, try typing `mkdir north pacific gyre`and see what directory (or directories!)
>    are made when you check with `ls -F`.
>
> 2. Don't begin the name with `-` (dash).
>
>    Commands treat names starting with `-` as options.
>
> 3. Stick with letters, numbers, `.` (period or 'full stop'), `-` (dash) and `_` (underscore).
>
>    Many other characters have special meanings on the command line.
>    We will learn about some of these during this lesson.
>    There are special characters that can cause your command to not work as
>    expected and can even result in data loss.
>
> If you need to refer to names of files or directories that have spaces
> or other special characters, you should surround the name in quotes (`""`).
{: .tip}

## Creating a text file

Let's change our working directory to `thesis` using `cd`,
then run a text editor called Nano to create a file called `draft.txt`:

```
$ cd thesis
$ nano draft.txt
```

> <warning-title>Nano won't work in a Jupyter Notebook!</warning-title>
> Nano is an interactive, full screen editor. You'll need to switch to a proper
> console to do this step, if you haven't already.
>
> In JupyterLab you can find this under the top menu by clicking **File -> New -> Terminal**
{: .warning}

> <tip-title>Which Editor?</tip-title>
>
> When we say, "`nano` is a text editor" we really do mean "text": it can
> only work with plain character data, not tables, images, or any other
> human-friendly media. We use it in examples because it is one of the
> least complex text editors. However, because of this trait, it may
> not be powerful enough or flexible enough for the work you need to do
> after this workshop. On Unix systems (such as Linux and macOS),
> many programmers use [Emacs](http://www.gnu.org/software/emacs/) or
> [Vim](http://www.vim.org/) (both of which require more time to learn),
> or a graphical editor such as
> [Gedit](http://projects.gnome.org/gedit/). On Windows, you may wish to
> use [Notepad++](http://notepad-plus-plus.org/).  Windows also has a built-in
> editor called `notepad` that can be run from the command line in the same
> way as `nano` for the purposes of this lesson.
>
> No matter what editor you use, you will need to know where it searches
> for and saves files. If you start it from the shell, it will (probably)
> use your current working directory as its default location. If you use
> your computer's start menu, it may want to save files in your desktop or
> documents directory instead. You can change this by navigating to
> another directory the first time you "Save As..."
{: .tip}

Let's type in a few lines of text.
Once we're happy with our text, we can press <kbd>Ctrl</kbd>+<kbd>O</kbd>
(press the <kbd>Ctrl</kbd> or <kbd>Control</kbd> key and, while
holding it down, press the <kbd>O</kbd> key) to write our data to disk
(we'll be asked what file we want to save this to:
press <kbd>Return</kbd> to accept the suggested default of `draft.txt`).

![screenshot of nano text editor in action](../../images/carpentries-cli/nano-screenshot.png)

Once our file is saved, we can use <kbd>Ctrl</kbd>+<kbd>X</kbd> to quit the editor and
return to the shell.

> <tip-title>Control, Ctrl, or ^ Key</tip-title>
>
> The Control key is also called the 'Ctrl' key. There are various ways
> in which using the Control key may be described. For example, you may
> see an instruction to press the <kbd>Control</kbd> key and, while holding it down,
> press the <kbd>X</kbd> key, described as any of:
>
> * `Control-X`
> * `Control+X`
> * `Ctrl-X`
> * `Ctrl+X`
> * `^X`
> * `C-x`
>
> In nano, along the bottom of the screen you'll see `^G Get Help ^O WriteOut`.
> This means that you can use `Control-G` to get help and `Control-O` to save your
> file.
{: .tip}

`nano` doesn't leave any output on the screen after it exits,
but `ls` now shows that we have created a file called `draft.txt`:

```bash
ls
```

> <question-title>Creating Files a Different Way</question-title>
>
> We have seen how to create text files using the `nano` editor.
> Now, try the following command:
>
> ```bash
> touch my_file.txt
> ```
>
> 1.  What did the `touch` command do?
>     When you look at your current directory using the GUI file explorer,
>     does the file show up?
>
> 2.  Use `ls -l` to inspect the files.  How large is `my_file.txt`?
>
> 3.  When might you want to create a file this way?
>
> > <solution-title></solution-title>
> > 1.  The `touch` command generates a new file called `my_file.txt` in
> >     your current directory.  You
> >     can observe this newly generated file by typing `ls` at the
> >     command line prompt.  `my_file.txt` can also be viewed in your
> >     GUI file explorer.
> >
> > 2.  When you inspect the file with `ls -l`, note that the size of
> >     `my_file.txt` is 0 bytes.  In other words, it contains no data.
> >     If you open `my_file.txt` using your text editor it is blank.
> >
> > 3.  Some programs do not generate output files themselves, but
> >     instead require that empty files have already been generated.
> >     When the program is run, it searches for an existing file to
> >     populate with its output.  The touch command allows you to
> >     efficiently generate a blank text file to be used by such
> >     programs.
> {: .solution}
{: .question}

```bash
# Explore the possible solutions here!
```

> <tip-title>What's In A Name?</tip-title>
>
> You may have noticed that all of Nelle's files are named 'something dot
> something', and in this part of the lesson, we always used the extension
> `.txt`.  This is just a convention: we can call a file `mythesis` or
> almost anything else we want. However, most people use two-part names
> most of the time to help them (and their programs) tell different kinds
> of files apart. The second part of such a name is called the
> **filename extension** and indicates
> what type of data the file holds: `.txt` signals a plain text file, `.pdf`
> indicates a PDF document, `.cfg` is a configuration file full of parameters
> for some program or other, `.png` is a PNG image, and so on.
>
> This is *just a convention*, albeit an important one. Files contain
> bytes: it's up to us and our programs to interpret those bytes
> according to the rules for plain text files, PDF documents, configuration
> files, images, and so on.
>
> Naming a PNG image of a whale as `whale.mp3` doesn't somehow
> magically turn it into a recording of whale song, though it *might*
> cause the operating system to try to open it with a music player
> when someone double-clicks it.
{: .tip}

## Moving files and directories
Returning to the `shell-lesson-data` directory,

```bash
cd ~/Desktop/shell-lesson-data/
```

In our `thesis` directory we have a file `draft.txt`
which isn't a particularly informative name,
so let's change the file's name using `mv`,
which is short for 'move':

```bash
mv thesis/draft.txt thesis/quotes.txt
```

The first argument tells `mv` what we're 'moving',
while the second is where it's to go.
In this case,
we're moving `thesis/draft.txt` to `thesis/quotes.txt`,
which has the same effect as renaming the file.
Sure enough,
`ls` shows us that `thesis` now contains one file called `quotes.txt`:

```bash
ls thesis
```

One must be careful when specifying the target file name, since `mv` will
silently overwrite any existing file with the same name, which could
lead to data loss. An additional option, `mv -i` (or `mv --interactive`),
can be used to make `mv` ask you for confirmation before overwriting.

Note that `mv` also works on directories.

Let's move `quotes.txt` into the current working directory.
We use `mv` once again,
but this time we'll use just the name of a directory as the second argument
to tell `mv` that we want to keep the filename
but put the file somewhere new.
(This is why the command is called 'move'.)
In this case,
the directory name we use is the special directory name `.` that we mentioned earlier.

```bash
mv thesis/quotes.txt .
```

The effect is to move the file from the directory it was in to the current working directory.
`ls` now shows us that `thesis` is empty:

```bash
ls thesis
```

Alternatively, we can confirm the file `quotes.txt` is no longer present in the `thesis` directory
by explicitly trying to list it:

```bash
ls thesis/quotes.txt
```

`ls` with a filename or directory as an argument only lists the requested file or directory.
If the file given as the argument doesn't exist, the shell returns an error as we saw above.
We can use this to see that `quotes.txt` is now present in our current directory:

```bash
ls quotes.txt
```

> <question-title>Moving Files to a new folder</question-title>
>
> After running the following commands,
> Jamie realizes that she put the files `sucrose.dat` and `maltose.dat` into the wrong folder.
> The files should have been placed in the `raw` folder.
>
> ```
> $ ls -F
>  analyzed/ raw/
> $ ls -F analyzed
> fructose.dat glucose.dat maltose.dat sucrose.dat
> $ cd analyzed
> ```
>
> Fill in the blanks to move these files to the `raw/` folder
> (i.e. the one she forgot to put them in)
>
> ```bash
> mv sucrose.dat maltose.dat ____/____
> ```
>
> > <solution-title></solution-title>
> > ```
> > $ mv sucrose.dat maltose.dat ../raw
> > ```
> > {: .language-bash}
> > Recall that `..` refers to the parent directory (i.e. one above the current directory)
> > and that `.` refers to the current directory.
> {: .solution}
{: .question}

## Copying files and directories

The `cp` command works very much like `mv`,
except it copies a file instead of moving it.
We can check that it did the right thing using `ls`
with two paths as arguments --- like most Unix commands,
`ls` can be given multiple paths at once:

```bash
cp quotes.txt thesis/quotations.txt
ls quotes.txt thesis/quotations.txt
```

We can also copy a directory and all its contents by using the
[recursive](https://en.wikipedia.org/wiki/Recursion) option `-r`,
e.g. to back up a directory:

```bash
cp -r thesis thesis_backup
```

We can check the result by listing the contents of both the `thesis` and `thesis_backup` directory:

```bash
ls thesis thesis_backup
```

> <question-title>Renaming Files</question-title>
>
> Suppose that you created a plain-text file in your current directory to contain a list of the
> statistical tests you will need to do to analyze your data, and named it: `statstics.txt`
>
> After creating and saving this file you realize you misspelled the filename! You want to
> correct the mistake, which of the following commands could you use to do so?
>
> 1. `cp statstics.txt statistics.txt`
> 2. `mv statstics.txt statistics.txt`
> 3. `mv statstics.txt .`
> 4. `cp statstics.txt .`
>
> > <solution-title></solution-title>
> > 1. No.  While this would create a file with the correct name,
> > the incorrectly named file still exists in the directory
> > and would need to be deleted.
> > 2. Yes, this would work to rename the file.
> > 3. No, the period(.) indicates where to move the file, but does not provide a new file name;
> > identical file names
> > cannot be created.
> > 4. No, the period(.) indicates where to copy the file, but does not provide a new file name;
> > identical file names cannot be created.
> {: .solution}
{: .question}

> <question-title>Moving and Copying</question-title>
>
> What is the output of the closing `ls` command in the sequence shown below?
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > pwd
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > /Users/jamie/data
> > ```
> {: .code-out}
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > ls
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > proteins.dat
> > ```
> {: .code-out}
>
>
> > <code-in-title>Bash</code-in-title>
> > ```
> > mkdir recombined
> > mv proteins.dat recombined/
> > cp recombined/proteins.dat ../proteins-saved.dat
> > ls
> > ```
> {: .code-in}
>
> 1.   `proteins-saved.dat recombined`
> 2.   `recombined`
> 3.   `proteins.dat recombined`
> 4.   `proteins-saved.dat`
>
> > <solution-title></solution-title>
> > We start in the `/Users/jamie/data` directory, and create a new folder called `recombined`.
> > The second line moves (`mv`) the file `proteins.dat` to the new folder (`recombined`).
> > The third line makes a copy of the file we just moved.
> > The tricky part here is where the file was copied to.
> > Recall that `..` means 'go up a level', so the copied file is now in `/Users/jamie`.
> > Notice that `..` is interpreted with respect to the current working
> > directory, **not** with respect to the location of the file being copied.
> > So, the only thing that will show using ls (in `/Users/jamie/data`) is the recombined folder.
> >
> > 1. No, see explanation above. `proteins-saved.dat` is located at `/Users/jamie`
> > 2. Yes
> > 3. No, see explanation above. `proteins.dat` is located at `/Users/jamie/data/recombined`
> > 4. No, see explanation above. `proteins-saved.dat` is located at `/Users/jamie`
> >
> {: .solution}
{: .question}

## Removing files and directories

Returning to the `shell-lesson-data` directory,
let's tidy up this directory by removing the `quotes.txt` file we created.
The Unix command we'll use for this is `rm` (short for 'remove'):

```bash
rm quotes.txt
```

We can confirm the file has gone using `ls`:

```bash
ls quotes.txt
```

>


topic: data-science
tutorial: data-manipulation-olympics-sql
title: Data Manipulation Olympics - SQL
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa", "sql"]
content_hash: 6ace3676a678bd26e21ca46b29b0cfbb
last_modified: 2026-05-15T22:28:10.991204
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/data-manipulation-olympics-sql/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->


Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

| Operation                | Description                                                         | Galaxy Tool                                   |
| ------------------------ | ------------------------------------------------------------------- | --------------------------------------------- |
| Compute on rows          | to derive new column values from existing ones                      | `SELECT x * 2 FROM y`                         |
| Concatenate datasets     | one after the other                                                 | `SELECT * FROM x; union all; SELECT * FROM y` |
| Counting                 | Count occurrences of values in a column                             | `SELECT count(x) FROM y where x = 'value'`    |
| Cut Columns              | By header name                                                      | `SELECT x, y, z FROM a`                       |
| Filter                   | Remove rows based on values in one or more columns                  | `... WHERE x = 'value'`                       |
| Find and Replace         | in a specific column                                                | `REPLACE()`, `regexp_replace` in postgresql   |
| Group on a column        | And perform simple operations (count, mean, min, max etc)           | `... GROUP BY x ...`                          |
| Join two Datasets        | side by side on a specified field                                   | `SELECT * FROM x, y JOIN x.id = y.id`         |
| Select First lines       | Good for finding top 10s or saving header lines                     | `... LIMIT 10`                                |
| Sort on a column         | Change the order of the rows based on values in one or more columns | `... ORDER BY x ASC`                          |
| Unique                   | Remove duplicate rows                                               | `SELECT DISTINCT x FROM y`                    |


In this tutorial, these functions are explained in more detail, and we provide some exercises for you to practice.

# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"]({% link topics/introduction/tutorials/data-manipulation-olympics/images/cover.jpg %})


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!

# Preamble
```sql
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c {{page.zenodo_link}}/files/olympics.db
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///olympics.db")
%load_ext sql
%sql sqlite:///olympics.db
%config SqlMagic.displaycon=False
```

# Download Data

Before we can do any manipulation, we will need some data. Let's download our table with Olympics results now.

```sql
SELECT * FROM olympics LIMIT 10;
```

And now we can start querying the database:

```sql
SELECT
    name
FROM
    sqlite_schema
```

> <question-title></question-title>
>
> 1. What tables are available?
> 2. How are they structured?
>
> > <solution-title></solution-title>
> >
> > 1. `countries`, `olympics`, `olympics_2022`
> > 2. Each are tables with 10 or more columns.
> {: .solution}
{: .question}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The `olympics` table contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.db` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.

# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

In `SQL` we can use the `ORDER BY` clause. We'll start by limiting our results, as every table in this dataset is quite large.

```sql
SELECT NOC, `CLDR display name` FROM countries LIMIT 30;
```

You can use `ORDER BY column-name ASC` or `ORDER BY column-name DESC` to sort the data ascending or descending.

```sql
SELECT NOC, `CLDR display name` FROM countries ORDER BY NOC ASC LIMIT 30;
```

```sql
SELECT NOC, `CLDR display name` FROM countries ORDER BY NOC DESC LIMIT 30;
```

So let's sort our file in chronological order, based on the year of the Olympic games:

> <question-title></question-title>
>
> 1. Which column contains the year?
>
> > <solution-title></solution-title>
> >
> > 1. `year`
> >
> {: .solution}
{: .question}

```sql
SELECT * FROM olympics ORDER BY year LIMIT 30;
```

If we wanted to do it in reverse, we could just use `order by year desc`

```sql
SELECT * FROM olympics ORDER BY year DESC LIMIT 30;
```

> <question-title></question-title>
>
> 1. Write a query to access only the first entry.
> 2. Which athlete is listed at the top of the file now?
>
> > <solution-title></solution-title>
> > 1. We can use `LIMIT` for this.
> >    ```sql
> >    select * from olympics order by year limit 1;
> >    ```
> > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
> >
> {: .solution}
{: .question}

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, an then within each year, we sort again alphabetically by name.

We will sort the file in chronological order based on the year of the Olympic games

```sql
SELECT * FROM olympics ORDER BY year, name LIMIT 30;
```

> <question-title></question-title>
>
> Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>
> > <solution-title></solution-title>
> >
> > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
> >
> {: .solution}
{: .question}


## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> >
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <tip-title>Removing null values</tip-title>
> > This will be covered more during the Filtering section, but for now simply use this filter:
> > ```sql
> > SELECT * from olympics where height is not null ... ;
> > ```
> {: .tip}
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We can use `.height`, and because we want the tallest on top, we will need to sort in *descending* (decreasing) order. Unfortunately you might discover there are null values.
> > 2. Rerun the same query as step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Lyton Mphande from Seol is the shortest at 127 cm.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > 1. `select * from olympics  where height is not null order by height desc limit 1;`
> > 2. `select * from olympics  where height is not null order by height asc limit 1`
> > 3. `select * from olympics  where height is not null order by year desc, height desc limit 1`
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).

We will filter the file to show only winter Olympics
Look at the `olympics` table and answer the following questions

> <question-title></question-title>
>
> 1. Which key contains this information?
> 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>
> > <solution-title></solution-title>
> >
> > 1. `season`
> > 2. The values can be `Summer` or `Winter` (`select distinct season from olympics`)
> >
> {: .solution}
{: .question}

We'll be using the `WHERE` filter to select entries matching specific conditions:

> <question-title></question-title>
>
> 1. How would you write the expressions for the following conditions:
>    1. `enrolled` must be 'Yes'
>    2. `age` must be smaller than 75
>    3. `height` cannot be null
>    4. `birthplace` cannot be empty
>
> 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    How would you write expressions for the following filtering conditions:
>    1. `height` is larger than 200 or smaller than 160
>    2. `height` is larger than 200 and smaller than 210
>
> > <solution-title></solution-title>
> >
> > 1. The answers are:
> >    1. `select * from olympics where enrolled = 'Yes'`
> >    2. `select * from olympics where age < 75`
> >    3. `select * from olympics where height is not null``
> >    4. `select * from olympics where birthplace != ""`
> >
> > 2. The answers are:
> >    1. `select * from olympics where height > 200 or height < 160`
> >    2. `select * from olympics where height > 200 and height < 210`
> >
> {: .solution}
{: .question}



Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics. Make sure it is contained in an array, in case we want to do further sorting.

```sql
CREATE TABLE winter AS SELECT * FROM olympics WHERE season = 'Winter'
```

> <question-title></question-title>
>
> How many entries are in this file? (Hint: use `count(*)`)
>
> > <solution-title></solution-title>
> >
> > 44,680
> >
> {: .solution}
{: .question}


**Repeat** the step for the Summer Olympics

```sql
CREATE TABLE summer AS SELECT * FROM olympics WHERE season = 'Summer'
```

> <question-title></question-title>
>
> 1. How many lines do you expect in the this file?
> 2. How many lines are in this file? Were you right?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Use the `count(*)` select
> > 2. Be careful to consider whether these counts include the header line of the file or not
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > 1. The original file has 234,522 entries, and the Winter Olympics had 44,680 entries. So we would expect 234,522 - 44,680 = 189,842 rows of data.
> > It is always useful to take a moment to think about the expected outcome, this makes it easier to spot mistakes and will save you time in the long run.
> >
> {: .solution}
{: .question}

## Exercises

Ok, time to train! let's see if you can use the `select` filter to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm? (inclusive)
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `SELECT count(*) FROM olympics WHERE medal == "Gold"`)
> >  2. 24,633  (Expression: `SELECT count(*) FROM olympics WHERE medal == "Gold" or medal == "Silver" or medal == "Bronze")`, or `medal != null`)
> >  3. 131     (Expression: `SELECT count(*) FROM olympics WHERE medal == "Gold" and year == 2018` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `SELECT count(*) FROM olympics WHERE medal is not null and height >= 170 and height <=180`)
> >  5. 2,333   (Expression: `SELECT count(*) FROM olympics WHERE medal is not null and (height < 160 or height > 190)` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count.

Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)

We'll need to use the `group by` syntax which takes a key, and then groups by those values.

> <question-title></question-title>
>
> 1. How many different Olympic games are in our file?
> 2. Which Olympic games had the most participations? (Tip: use order by)
>
> > <solution-title></solution-title>
> >
> > 1. 52 games (`select count(*), games from olympics group by games`)
> >
> >    The resulting file looks something like:
> >
> >     ```
> >     615	1896 Summer Olympics
> >     2503	1900 Summer Olympics
> >     2643	1904 Summer Olympics
> >     3213	1908 Summer Olympics
> >     4610	1912 Summer Olympics
> >     3448	1920 Summer Olympics
> >     5242	1924 Summer Olympics
> >     358	1924 Winter Olympics
> >     4493	1928 Summer Olympics
> >     ...
> >     ```
> >
> > 2. 1996 Summer Olympics. (10501 participations)
> >
> {: .solution}
{: .question}

You may have guessed that like `order by`, that we could have selected multiple columns in the `group by` step. This lets us count on combinations of columns.

Let's try counting the number of men and women in each olympic games.

```sql
select count(*), games, sex from olympics group by games, sex
```

> <question-title></question-title>
>
> You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>
> 1. How many women were in the first Olympic games?
>
> 2. Which Olympic games had the most women participants?
>
> > <solution-title></solution-title>
> >
> > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice, in this query. Do you know any way we could query that? Try it out!)
> >    The results looks something like this:
> >    ```
> >    2	F	1896 Summer Olympics
> >    43	F	1900 Summer Olympics
> >    17	F	1904 Summer Olympics
> >    55	F	1908 Summer Olympics
> >    97	F	1912 Summer Olympics
> >    132	F	1920 Summer Olympics
> >    269	F	1924 Summer Olympics
> >    ```
> >
> > 2. 2020 Summer Olympics (4652)
> >
> {: .solution}
{: .question}

Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting query above, we would get resultsfor each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

We can use the `distinct` filter in our pipeline to discover this. First let's do our group by and iterate over each resulting group:

```sql
select games, sport from olympics group by games;
```

And let's count all of their appearances

```sql
select games, count(sport) as sports from olympics group by games;
```

But those results still aren't distinct, those numbers are far too high. So let's use distinct:

```sql
select games, count(distinct sport) as sports from olympics group by games;
```

We're almost there! Let's sort this

```sql
select games, count(distinct sport) as sports from olympics group by games order by sports asc;
```

> <question-title></question-title>
>
> 2. How many sport were in the first Olympics? How many in the latest?
> 3. Which Olympics had the most different sports?
>
> > <solution-title></solution-title>
> >
> > 2. 10 and 38.
> > 3. The 2020 Summer Olympics had the most different sports (38)
> >
> {: .solution}
{: .question}

Save the output as something descriptive.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting instances of a key, we can use `group by team` and then loop over that to print out the length, and the team name of each of those items.
> > 2. This is basically the same question as "how many women" participated, try modifying that query.
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations (`select team, count(team) as count from olympics group by team order by count desc;`)
> >  2. 15 and 250. (`select games, count(distinct team) as teams from olympics group by games;`)
> >
> {: .solution}
{: .question}


# Grouping

Often we may want to group rows based on a value in a column, and perform some operation on the resulting rows. For example we would like to group the olympics data by one value (e.g. year, country, sport), and determine some value for each group (e.g. number of medals won, average age of athletes, etc).

In the [counting](#counting) section of this tutorial we show how to get answers that require a count (e.g. number of medals won), but sometimes we want to do something more complex, like calculating the average height of athletes in a group, say per country or per sport. This section will show some example of these types of questions.

We can use continue to use group by for this, but now we'll need the max and min aggregate operations. Essentially every time we use `group by` we need to use an aggregation like finding the maximum, minimum, or counting the number of results.

> <hands-on-title>Tallest athlete per sport</hands-on-title>
>
> We would like to answer the following question: *How tall was the tallest athlete of each sport?*
>
> > <question-title></question-title>
> >
> > 1. How tall was the tallest athlete in basketball? And what about karate?
> > 2. Why do some sports have null values?
> >
> > > <solution-title></solution-title>
> > > ```sql
> > > select max(height), min(height),sport from olympics group by sport
> > > ```
> > >
> > > 1. Basketball's tallest athlete was 192cm. For Karate it is 163.
> > > 2. Our dataset had quite a number of `null` (unknown) values in the height column, especially for the earlier Olympics. These are preserved in the outputs.
> > {: .solution}
> {: .question}
{: .hands_on}


## Grouping on multiple columns

You may have noticed that we could also provide multiple columns to group on. If we do this, we can compute values for combinations of groups, such as sex and sport, to find e.g. the tallest woman in basketball or the shortest man per Olympics. There are also many more options for the computation we perform, so perhaps we are more interested not in the tallest athlete, but the average height. Let's perform some of these slightly more advanced queries now.


> <hands-on-title>Average height of men and women per sport</hands-on-title>
>
> The question we would like to answer here, is what is the average height for men and women per sport?
>
> ```sql
> select avg(height), sex, sport from olympics group by sex, sport;
> ```
>
> See if you can answer the following questions based on the output file.
>
> > <question-title></question-title>
> >
> > 1. What is the average height of women participating in archery?
> > 2. What is the average height of men participating in [ballooning](https://en.wikipedia.org/wiki/Ballooning_at_the_1900_Summer_Olympics)?
> > 3. Why do some values have `null` instead of a height?
> > 4. Why do some sports not have a value for one of the sexes?
> > 5. Can you find a sport where women were taller than the men? (Hint: it starts with the letter A)
> >
> > > <solution-title></solution-title>
> > >
> > > 1. 167.25677031093 cm
> > > 2. 170 cm
> > > 3. If none of the rows in the group had height data available, it will output `nan` (not a number) instead. This is most common for sports that were only featured a few times in the early years of the Olympics.
> > > 4. Sports such as artistic swimming only exist for women, so no M appears in the data for that group, so there simply is no row for the mean height of men doing artistic swimming in our output.
> > > 5. [Art Competitions](https://en.wikipedia.org/wiki/Art_competitions_at_the_Summer_Olympics)
> > >
> > > If all went well, your output file should look something like:
> > >
> > > ```
> > > GroupBy(sport)	     GroupBy(sex)  mean(height)
> > > Aeronautics         M             nan
> > > Alpine Skiing       F             167.38324708926
> > > Alpine Skiing       M             178.18747142204
> > > Alpinism            M             nan
> > > Archery             F             167.25677031093
> > > Archery             M             178.5865470852
> > > Art Competitions    F             175.33333333333
> > > Art Competitions    M             173.97260273973
> > > Artistic Gymnastics F             156.15316901408
> > > ```
> > {: .solution}
> {: .question}
>
{: .hands_on}


## Exercises

> <question-title noprefix>Exercise: Grouping and computing</question-title>
>
> 1. How tall is the shortest woman Badminton player to win a gold medal?
> 2. What is the average height of athletes from team Denmark in the 1964 Olympics? (Note: 1964 has summer and winter olympics)
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We need to group on 3 columns: medal, sport and sex, and then select the `min`.
> > 2. We need to group on 2 columns: country (team) and year, then compute the average over height.
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 161 cm.
> >  2. mean height: 175.91304347826, standard deviation: 7.0335410308672`
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> > ```sql
> > select min(height), medal, sport, sex from olympics group by medal, sport, sex;
> > select avg(height), team, games from olympics where team = "Denmark" group by team, games;
> > ```
> >
> {: .solution}
>
{: .question}


# Computing

Sometimes we want to use the data in our column to compute a new value, and add that to the table. For instance, for our dataset we could caluclate athtletes BMI (using height and weight columns), or their age at time of participation (from year of birth and year of the Olymics). By adding these computed values as a new colum to our datset, we can more easily query the dataset for these values. We can do these types of operations on the fly, and then if we like, store them as (temporary) tables.

As an example, let's calculate the age of each athlete at the time of participation, and add this as a new column to our dataset.

```sql
select year - birth_year as age, games from olympics LIMIT 30;
```

If we want to save that result to make it easier to query, then we have a couple options.

1. Don't store it, calculate on demand. Very storage efficient.

   ```sql
   select noc, name, ..., year - birth_year as age from olympics
   ```

1. Create a temporary table with this data, great if we only need it temporarily

   ```sql
   create temporary table olympics_ages as select *, year - birth_year as age from olympics
   ```

2. Create a new permanent table with this data

   ```sql
   create table olympics_ages as select *, year - birth_year as age from olympics
   ```

3. Update the existing table by adding it as a new column

   ```sql
   alter table olympics add column age int;
   update olympics set age = year - birth_year;
   ```

> <question-title></question-title>
>
> 1. How old was Arnaud Boetsch during his Olympic tennis participation?
>
> > <solution-title></solution-title>
> >
> > 1. Arnaud Boetsch is listed on the first two lines, who turned 27 the year of their Olympics.
> >
> {: .solution}
{: .question}

This was a simple computation, but much more complex mathematical expressions can be computed with this tool. In the exercise below, we will compute the BMI for each athlete as an example.


## Exercises

BMI stands for Body Mass Index, is a metric to provide a very crude measure of how healthy your weight is. The formula to compute BMI is:

$$ BMI = weight / (height^2) $$

(with weight in kilograms and height in meters).

> <tip-title>BMI is problematic</tip-title>
> [Adolphe Quetelet](https://en.wikipedia.org/wiki/Adolphe_Quetelet), who invented [BMI](https://en.wikipedia.org/wiki/Body_mass_index), was not a doctor of medicine, instead he was a Belgian astronomer, mathematician, statistician, and sociologist. His data consisted probably entirely of cisgender, white, european men and women {% cite Eknoyan_2007 %}.
> In his defense, he did argue it should not be used at individual levels, however it came to be used as such due to simplicity. (See [Wikipedia's History section](https://en.wikipedia.org/wiki/Body_mass_index#History)). BMI is a poor measure of health, especially for populations with high muscle content, and cannot be simply re-used as-is with anyone other than Anglo Saxons {% cite Caleyachetty_2021 %}, [Wikipedia:Limitations](https://en.wikipedia.org/wiki/Body_mass_index#Limitations).
>
> However as it is an easy to calculate metric, we include the calculation here.
{:.tip}

Let's compute this data for all athletes and add it as a new column!

> <question-title noprefix>Exercise: Calculating BMI</question-title>
>
> 1. How would you express this calculation in SQL?
>    - Remember that our height is in cm, and the formula expects height in meters
>    - And that we have null values, but we can ignore those as they will make the final calculation null as well
>
> 2. What is the BMI for Arnaud Boetsch?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - division is `/` and multiplication is ` * ` .
> > - Generally we cannot use `^`
> > - Parentheses may be required.
> > - use `select(.value != null)` to remove nulls.
> > - If that isn't eough, you can check that the type is a number with `(.value|type) == "number"` to ensure it really is a number and not e.g. a string.
> > - remember to wrap everything in `[...]` to retain the data shape
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> > 1. other variations are possible:
> >
> >    ```sql
> >    create temporary table olympics_bmi as select *, weight / (height / 100 * height / 100) as bmi from olympics;
> >    ```
> >
> > 2. 22.69
> >
> >    ```sql
> >    select * from olympics_bmi where name like 'Arnaud Boetsch';
> >    ```
> {: .solution}
{: .question}


# Find and Replace

Often you may need to change the contents of a file a bit to fit the expectations of an analysis tool. For instance, our database uses `null` for missing values, but other conventions included leaving the cell empty instead. Or, when working with chromosomal data, you may need to add or remove the `chr` prefix from a column before using it as input to a certain tool. In such situations, we can find all occurrences of a certain pattern in our file, and replace it with another value.

If we want to perform such a replacement on a single column in our data, we can use an update statement.

A few of the basics of regular expression, plus some links to further resources are given in the box below:

{% snippet faqs/galaxy/analysis_regular_expressions.md %}

Let's start with a simple example:
Our file uses a mix of `Athina` and `Athens` to indicate the Capital City of Greece in the `city` column.
Let's standardize this by replacing occurrences of `Athina` with `Athens`.

Let's start by filtering out the old spelling:

```sql
select * from olympics where city = 'Athina' limit 30;
```

Let's try replacing it:

```sql
update olympics
set city = 'Athens'
where city = 'Athina';
```

Look at the file before and after. Athlete 7 (Patrick Chila) near the top of the `olympics.tsv` file, had a value of Athina in the city column. Verify that it has been changed to Athens.

This was rather simple example, so let's try a few more examples with slightly more complex expressions.

## Exercises

You may have noticed that our file has a lot of missing data. Especially for the earlier years, things like height, weight and birthday of athletes was not registered, or simply not known. In some columns you see these missing values have been replaced with an `NA` (not available) value. In other columns (for example birth place), the cells have simply been left empty.

Different tools may expect different ways of handling missing data. So you may have to change your missing data from empty to `NA`, `NaN`, or something else, between analysis steps

> <hands-on-title>Fill empty cells</hands-on-title>
>
> We will now replace empty cells in the `birth_place` column, to use `null` instead.
>
> > <solution-title noprefix>Hints</solution-title>
> > Remember that comparison to nulls is done with `is` instead of `=`
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> > other variations are possible:
> > ```sql
> > update olympics set birth_place = null where birth_place = '';
> > ```
> {: .solution}
>
{: .hands_on}

Let's do another example, this one splitting and re-constructing strings.

Look at the `birth_day` column. It has values in a format like `12 December`. Suppose we have a tool that expects this data to be in the reverse format, `December 12`. We would not want to do this manually, but with sql we can make this replacement easily

We will now change the format in birthday column from `day month` to `month day`, as our boss is American and requested the silly format.

First we need to understand that sqlite does not ship a regex engine, thus we cannot use familiar regular expressions. Instead we can make use of `instr(string, search)` to find the location of a substring like ` `, identifying where the day stops and the month starts. Then we can use `substr(string, start)` and `substr(string, start, end)` to chop up our date string.

In SQL, concatenation is done with `||`.

> <question-title></question-title>
>
> 1. How do we captures both the day and the month?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. We should use something like `substr(birth_day, instr(birth_day, ' '))`
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> > ```sql
> > select
> >   birth_day,
> >   instr(birth_day, ' ') as idx,
> >   substr(birth_day, instr(birth_day, ' ')) as month,
> >   substr(birth_day, 0, instr(birth_day, ' ')) as day
> > from olympics
> > limit 10
> > ```
> > 
> > We can make our final query:
> > 
> > ```sql
> > select
> >   substr(birth_day, instr(birth_day, ' ')) || substr(birth_day, 0, instr(birth_day, ' ')) as birth_day_new
> > from olympics
> > limit 10
> > ```
> > And then store this as a new column (using `update`) or as a new table (using `create [temporary] table`)
> >
> {: .solution}
{: .question}


# Removing Columns

In sqlite you cannot remove columns, it is not supported. Proper databases like Postgres and MySQL support this operation.
Instead for sqlite you could select the columns you want to keep, create a new table from that, and then delete the original table.

Other databases:

```
alter table delete column NAME from olympics
```

# Unique

Sometimes, in the course of our data manipulations, we may end up with a file that has duplicate values. In order to filter these out, we can use the `distinct` filter.

Let's say we would like to create a list of all unique athletes (id and name).

First we will just select the `athlete_id` and `name` columns from our dataset

```sql
select name, athlete_id from olympics LIMIT 30
```

> <question-title></question-title>
>
> 1. Do you see duplication? Why is that?
>
> > <solution-title></solution-title>
> >
> > 1. Yes. For all athletes who participated more than once, the row will be identical.
> >
> {: .solution}
{: .question}

Now let's remove those duplicates.

```sql
select distinct name, athlete_id from olympics limit 30
```

> <question-title></question-title>
>
> How many unique athletes do we have? Note that you cannot count multiple columns, so choose one that is correct.
>
> > <solution-title></solution-title>
> > 94,733
> >
> > ```sql
> > select count(distinct athlete_id) from olympics;
> > ```
> >
> {: .solution}
{: .question}


# Joining Datasets

This database contains a lot of information, but we may want to add more information. For example if we had a file with information about each country (population, capital city, etc), we could join that information with our Olympics data, to get a single result with all information in every row.

For example, if we would like to be able to group by continent, to e.g. count athletes, medals etc per continent, we will have to add a `continent` column to our file. To do this we would need a second file that maps each country to the corresponding continent. This is what we will do in the next hands-on section.

We obtained country information data from [DataHub](https://datahub.io/core/country-codes). More information about this file can be found in the description there. It has 56 columns with a wide variety of data about each country (from country codes, to capital city, languages spoken, etc)

It is available in the countries table.

> <question-title></question-title>
>
> 2. Which keys(s) in this file are the same as in the `olympics.tsv` file?
>
> > <solution-title></solution-title>
> >
> > 2. Both files have a `NOC` column with the 3-letter country code (`NOC` stands for National Olympic Committee). However, one is lowercase.
> {: .solution}
{: .question}

We would now like to take our Olympics dataset as the basis, and add columns to every row of this file with some information about the country. In order to join, we will need to have one column that is shared between the two files, on which we can match. The `NOC` column is perfect for this because it is a defined standard. Both files also contain a column with the country name in it, which is also a possible candidate to use for joining, but because it is less standardised, it is safer to use the NOC column. For example, if one file uses "Netherlands", while the other uses "The Netherlands" to indicate the same country, the joining will fail for these rows. So always make sure the columns you join on are compatible!

We can use the join commands.

```sql
select * from olympics left join countries on olympics.noc = countries.NOC LIMIT 30
```

> <question-title></question-title>
>
> 1. What do you expect the output to look like? Were you right?
> 2. How many columns are in the resulting file? What about the NOC column?
> 3. What is a possible downside to this approach?
>
> > <solution-title></solution-title>
> >
> > 1. All the columns from the country information file are added to the end of each row of our olympics dataset
> > 2. Our olympics datset had 17 columns, the country information file has 56 columns. Therefore we have 17+56=73 columns columns in our resulting file. This also means the NOC column
> >    we joined on appears twice in our output.
> > 3. There is a lot of data duplication in the output now. The exact same country information is added to every line of every athlete from a certain country.
> >    This means much larger response size.
> >    If you do not need all these columns, it could save you a lot of space to select only specific columns that you require.
> >
> {: .solution}
{: .question}

# Concatenating

Concatenation of two files simple means putting the contents of the two files together, one after the other. Our dataset was created in 2021, but since then we've had another Olympic event, the 2022 Winter Olympics in Beijing. If we have the same data for this latest Olympics, we could simply add the rows from the 2022 games to our current file with data, in order to create a single file with all data from 1896 to 2022.

View the table `olympics_2022`, does it have the same structure as our original `olympics` table?

> <question-title></question-title>
>
> 1. Does the new table have the same structure?
> 2. Can we simply add the lines of the new table to the end of our existing olympics dataset?
>
> > <solution-title></solution-title>
> > 1. Yes, this file has all the same columns, in the same order, so concatenation should be relatively straightforward.
> > 2. Yes.
> >
> {: .solution}
{: .question}

Since this new dataset has the exact same structure (number and order of columns), we can simple add the lines from this file to the end of our existing `olympic` table.
For this, we'll need to use the `union all` which takes two separate sql queries and unifies the results.

```sql
select * from olympics
union all
select * from olympics_2022
LIMIT 30;
```

(Note: We are limiting the outputs to ensure your browser does not crash loading all of the data.)

Now this only works so simply because our two datasets had the same structure. If your data comes from different sources, you may have to do some additional data manipulation before you can union, e.g. to make sure the columns match, or how each file deals with missing data (empty cells, `NA`, `NaN` or something else).

# Conclusion

These operations covered in the tutorial are just a few examples of some of the most common operations. There are many more available. We encourage you to look around the documentation of sqlite or your database. The more comfortable you are performing these kinds of steps, the more you can get out of SQL!

# Exercises: Putting it all together!

This section provides a number of exercises that require you to combine two or more of the techniques you learned in this tutorial. This is a great way to practice your data manipulation skills. Full solutions are provided for every exercise (i.e. all tools and settings), but for many of these exercises there will be multiple solutions, so if you obtained the same results in a different way, that is correct too!

> <question-title noprefix>Exercise 1: Finding shortest/lightest athlete</question-title>
>
> If you have done exercises in the [sorting](#sorting) section, you noticed that finding the shortest athlete ever to compete was not easy,
> because all the rows with missing height data (`NA`) in the column were sorted to the top. We need to filter out these values first, then
> perform our sort, so that our answer is on top.
>
> 1. Find the shortest athlete ever to compete in the Olympics
> 2. Find the shortest athlete of the Winter Olympics
> 2. Find the lightest athlete of the *most recent* Summer Olympics
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. You will need to filter out the columns with (`NA`) in the height column first
> > 2. You will need to filter by season as well
> > 3. You will need to filter out missing data in the weight column, filter out Summer Olympics, then sort (by 2 columns)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Lyton Mphande and  Rosario Briones were both 127 cm tall, competing in boxing and gymnastics respectively
> >  2. Carolyn Krau was a 137 cm tall figure skater.
> >  3. Flávia Saraiva was the lightest athlete (31kg), she was a Artistic Gymnast from Brazil.
> >
> {: .solution}
>
> > <solution-title noprefix>Full solution</solution-title>
> >
> > 1. First we filter out the NA values from the height column:
> >
> >    ```sql
> >    select * from olympics where height is not null
> >    ```
> >
> >    Then we can sort by height, in ascending order to get the shortest athletes on top:
> >
> >    ```sql
> >    select * from olympics where height is not null order by height desc limit 10;
> >    ```
> >
> > 2. We can take the output from the first exercise, and filter for only Winter Olympics:
> >
> >    ```sql
> >    select * from olympics where height is not null and season = 'Winter' order by height desc limit 10;
> >    ```
> >
> > 3. First we filter out the NA values from the weight column:
> >
> >    ```sql
> >    select * from olympics where weight is not null and games = '2020 Summer Olympics' order by weight asc limit 10;
> >    ```
> >
> {: .solution}
>
{: .question}

Congratulations! You have now mastered the basics of data manipulation! There are a lot more data manipulation operations available that you may need. Please explore the tools for yourself, and check back with this tutorial often, we plan to add more sections and exercises over time!


