# tutorials

topic: data-science
tutorial: sql-r
title: SQL with R
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["SQL", "R"]
content_hash: 9b4f97eaf5782a17f5f3c23b6e159b94
last_modified: 2026-02-26T21:12:14.912813
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-r/tutorial.html

content:
In this tutorial you'll learn to use SQL via R. Some R and SQL experience is a pre-requisite.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

For this tutorial we need to download a database that we will use for the queries.

```r
download.file("http://swcarpentry.github.io/sql-novice-survey/files/survey.db", destfile="survey.db")
```

# Programming with Databases - R

Let's have a look at how to access a database from
a data analysis language like R.
Other languages use almost exactly the same model:
library and function names may differ,
but the concepts are the same.

Here's a short R program that selects latitudes and longitudes
from an SQLite database stored in a file called `survey.db`:

```r
library(RSQLite)
connection <- dbConnect(SQLite(), "survey.db")
results <- dbGetQuery(connection, "SELECT Site.lat, Site.long FROM Site;")
print(results)
dbDisconnect(connection)
```

The program starts by importing the `RSQLite` library.
If we were connecting to MySQL, DB2, or some other database,
we would import a different library,
but all of them provide the same functions,
so that the rest of our program does not have to change
(at least, not much)
if we switch from one database to another.

Line 2 establishes a connection to the database.
Since we're using SQLite,
all we need to specify is the name of the database file.
Other systems may require us to provide a username and password as well.

On line 3, we retrieve the results from an SQL query.
It's our job to make sure that SQL is properly formatted;
if it isn't,
or if something goes wrong when it is being executed,
the database will report an error.
This result is a dataframe with one row for each entry and one column for each column in the database.

Finally, the last line closes our connection,
since the database can only keep a limited number of these open at one time.
Since establishing a connection takes time,
though,
we shouldn't open a connection,
do one operation,
then close the connection,
only to reopen it a few microseconds later to do another operation.
Instead,
it's normal to create one connection that stays open for the lifetime of the program.

Queries in real applications will often depend on values provided by users.
For example,
this function takes a user's ID as a parameter and returns their name:

```r
library(RSQLite)

connection <- dbConnect(SQLite(), "survey.db")

getName <- function(personID) {
  query <- paste0("SELECT personal || ' ' || family FROM Person WHERE id =='",
                  personID, "';")
  return(dbGetQuery(connection, query))
}

print(paste("full name for dyer:", getName('dyer')))

dbDisconnect(connection)
```

We use string concatenation on the first line of this function
to construct a query containing the user ID we have been given.
This seems simple enough,
but what happens if someone gives us this string as input?

```
dyer'; DROP TABLE Survey; SELECT '
```

It looks like there's garbage after the user's ID,
but it is very carefully chosen garbage.
If we insert this string into our query,
the result is:

```
SELECT personal || ' ' || family FROM Person WHERE id='dyer'; DROP TABLE Survey; SELECT '';
```

If we execute this,
it will erase one of the tables in our database.

This is called an SQL injection attack,
and it has been used to attack thousands of programs over the years.
In particular,
many web sites that take data from users insert values directly into queries
without checking them carefully first.
A very [relevant XKCD](https://xkcd.com/327/) that explains the
dangers of using raw input in queries a little more succinctly:

![A 4 panel comic, in the first panel a person is shown answering the phone, hearing that their son's school has some computer trouble. In panel 2 they apologises asking if their child broke something. In panel 3, the unseen person on the other end of the phone call asks if they really named their son Robert'); Drop table students;--? They respond saying 'oh yes. little bobby tables we call him.' In the 4th panel the caller says 'well we have lost this years student records, I hope you're happy.' They respond 'And I hope you've learned to sanitize your database inputs'.](../../images/xkcd/exploits_of_a_mom.png)

Since an unscrupulous parent might try to smuggle commands into our queries in many different ways,
the safest way to deal with this threat is
to replace characters like quotes with their escaped equivalents,
so that we can safely put whatever the user gives us inside a string.
We can do this by using a prepared statement
instead of formatting our statements as strings.
Here's what our example program looks like if we do this:

```r
library(RSQLite)
connection <- dbConnect(SQLite(), "survey.db")

getName <- function(personID) {
  query <- "SELECT personal || ' ' || family FROM Person WHERE id == ?"
  return(dbGetPreparedQuery(connection, query, data.frame(personID)))
}

print(paste("full name for dyer:", getName('dyer')))

dbDisconnect(connection)
```

The key changes are in the query string and the `dbGetQuery` call (we use dbGetPreparedQuery instead).
Instead of formatting the query ourselves,
we put question marks in the query template where we want to insert values.
When we call `dbGetPreparedQuery`,
we provide a dataframe
that contains as many values as there are question marks in the query.
The library matches values to question marks in order,
and translates any special characters in the values
into their escaped equivalents
so that they are safe to use.

> <question-title>Filling a Table vs. Printing Values</question-title>
>
> Write an R program that creates a new database in a file called
> `original.db` containing a single table called `Pressure`, with a
> single field called `reading`, and inserts 100,000 random numbers
> between 10.0 and 25.0.  How long does it take this program to run?
> How long does it take to run a program that simply writes those
> random numbers to a file?
{: .question}

> <question-title>Filtering in SQL vs. Filtering in R</question-title>
>
> Write an R program that creates a new database called
> `backup.db` with the same structure as `original.db` and copies all
> the values greater than 20.0 from `original.db` to `backup.db`.
> Which is faster: filtering values in the query, or reading
> everything into memory and filtering in R?
{: .question}

## Database helper functions in R

R's database interface packages (like `RSQLite`) all share
a common set of helper functions useful for exploring databases and
reading/writing entire tables at once.

To view all tables in a database, we can use `dbListTables()`:

```r
connection <- dbConnect(SQLite(), "survey.db")
dbListTables(connection)
```


To view all column names of a table, use `dbListFields()`:

```r
dbListFields(connection, "Survey")
```


To read an entire table as a dataframe, use `dbReadTable()`:

```r
dbReadTable(connection, "Person")
```


Finally to write an entire table to a database, you can use `dbWriteTable()`.
Note that we will always want to use the `row.names = FALSE` argument or R
will write the row names as a separate column.
In this example we will write R's built-in `iris` dataset as a table in `survey.db`.

```r
dbWriteTable(connection, "iris", iris, row.names = FALSE)
head(dbReadTable(connection, "iris"))
```

And as always, remember to close the database connection when done!

```r
dbDisconnect(connection)
```


topic: data-science
tutorial: python-plotting
title: Plotting in Python
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: a9990919fdf3f448ccd5b1095029fce0
last_modified: 2026-02-26T21:12:14.904445
zenodo_link: https://zenodo.org/record/3477564
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-plotting/tutorial.html

content:
In this lesson, we will be using Python 3 with some of its most popular scientific libraries. This tutorial assumes that the reader is familiar with the fundamentals of data analysis using the Python programming language, as well as, how to run Python programs using Galaxy. Otherwise, it is advised to follow the "Introduction to Python" and "Advanced Python" tutorials available in the same platform. We will be using JupyterNotebook, a Python interpreter that comes with everything we need for the lesson.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which is licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Plot data using matplotlib

For the purposes of this tutorial, we will use a file with the annotated differentially expressed genes that was produced in the [Reference-based RNA-Seq data analysis]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial.

Firstly, we read the file with the data.

```python
data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
print(data)
```

We can now use the `DataFrame.info()` method to find out more about a dataframe.

```python
data.info()
```

We learn that this is a DataFrame. It consists of 130 rows and 12 columns. None of the columns contains any missing values. 6 columns contain 64-bit floating point `float64` values, 2 contain 64-bit integer `int64` values and 4 contain character `object` values. It uses 13.2KB of memory.

We now have a basic understanding of the dataset and we can move on to creating a few plots and further explore the data. `matplotlib` is the most widely used scientific plotting library in Python, especially the `matplotlib.pyplot` module.

```python
import matplotlib.pyplot as plt
```
Simple plots are then (fairly) simple to create. You can use the `plot()` method and simply specify the data to be displayed in the x and y axis, by passing the data as the first and second argument. In the following example, we select a subset of the dataset and plot the P-value of each gene, using a lineplot.

```python
subset = data.iloc[121:, :]

x = subset['P-value']
y = subset['Gene name']

plt.plot(x, y)
plt.xlabel('P-value')
plt.ylabel('Gene name')
```

![A line chart is shown with a y axis of gene name and an x axis of p-value. The specific content of the graph is not important other than that it is produced with the expected x and y labels.](../../images/python-plotting/Figure9_Lineplot.png)

We use Jupyter Notebook and so running the cell generates the figure directly below the code. The figure is also included in the Notebook document for future viewing. However, other Python environments like an interactive Python session started from a terminal or a Python script executed via the command line require an additional command to display the figure.

Instruct matplotlib to show a figure:

```python
plt.show()
```
This command can also be used within a Notebook - for instance, to display multiple figures if several are created by a single cell.

If you want to save and download the image to your local machine, you can use the `plt.savefig()` command with the name of the file (png, pdf etc) as the argument. The file is saved in the Jupyter Notebook session and then you can download it. For example:

```python
plt.tight_layout()
plt.savefig('foo.png')
```

`plt.tight_layout()` is used to make sure that no part of the image is cut off during saving.

When using dataframes, data is often generated and plotted to screen in one line, and `plt.savefig()` seems not to be a possible approach. One possibility to save the figure to file is then to save a reference to the current figure in a local variable (with `plt.gcf()`) and then call the savefig class method from that variable. For example, the previous plot:

```python
subset = data.iloc[121:, :]

x = subset['P-value']
y = subset['Gene name']

fig = plt.gcf()
plt.plot(x, y)
fig.savefig('my_figure.png')
```



## More about plots

You can use the `plot()` method directly on a dataframe. You can plot multiple lines in the same plot. Just specify more columns in the x or y axis argument. For example:

```python
new_subset = data.iloc[0:10, :]
new_subset.loc[:, ['P-value', 'P-adj']].plot()
plt.xticks(range(0,len(new_subset.index)), new_subset['Gene name'], rotation=60)
plt.xlabel('Gene name')
```

![The graph is similar to the last image, now the x axis is the gene name, and the y axis runs from 0 to 7 with an annotation of 1e-65 above. Now there are two lines, one in red labelled p-adj, and one in blue labelled p-value.](../../images/python-plotting/Figure10_Multiple_lines_plot.png)

In this example, we select a new subset of the dataset, but plot only the two columns `P-value` and `P-adj`. Then we use the `plt.xticks()` method to change the text and the rotation of the x axis.

Another useful plot type is the barplot. In the following example we plot the number of genes that belong to the different chromosomes of the dataset.

```python
bar_data = data.groupby('Chromosome').size()
bar_data.plot(kind='bar')
plt.xticks(rotation=60)
plt.ylabel('N')
```
![It is now a bar plot with chromosome as an X axis and some values of N for the y axis for a few chromosomes.](../../images/python-plotting/Figure11_Barplot.png)

`matplotlib` supports also different plot styles from ather popular plotting libraries such as ggplot and seaborn. For example, the previous plot in ggplot style.

```python
plt.style.use('ggplot')
bar_data = data.groupby('Chromosome').size()
bar_data.plot(kind='bar')
plt.xticks(rotation=60)
plt.ylabel('N')
```
![The same graph as the previous, but with a different aethetic, the background is now light grey instead of white and the bars are red instead of blue to be a bit more like ggplot2 outputs.](../../images/python-plotting/Figure12_ggplot_Barplot.png)

You can also change different parameters and customize the plot.

```python
plt.style.use('default')
bar_data = data.groupby('Chromosome').size()
bar_data.plot(kind='bar', color = 'red', edgecolor = 'black')
plt.xticks(rotation=60)
plt.ylabel('N')
```

![The same graph again, but now the bars are red with a black border.](../../images/python-plotting/Figure13_Barplot_2.png)

Another useful type of plot is a scatter plot. In the following example we plot the Base mean of a subset of genes.

```python
scatter_data = data[['Base mean', 'Gene name']].head(n = 15)

plt.scatter(scatter_data['Gene name'], scatter_data['Base mean'])
plt.xticks(rotation = 60)
plt.ylabel('Base mean')
plt.xlabel('Gene name')
```

![A scatterplot is shown comparing gene name to base mean.](../../images/python-plotting/Figure14_Scatterplot.png)

> <question-title>Plotting</question-title>
>
>  Using the same dataset, create a scatterplot of the average P-value for every chromosome for the "+" and the "-" strand.
>
> > <solution-title></solution-title>
> >
> > First find the data and save it in a new dataframe. Then create the scatterplot. You can even go one step further and assign different colors for the different strands.Note the use of the `map` method that assigns the different colors using a dictionary as an input.
> >
> > ```
> > exercise_data = data.groupby(['Chromosome', 'Strand']).agg(mean_pvalue = ('P-value', 'mean')).reset_index()
> >
> > colors = {'+':'red', '-':'blue'}
> > plt.scatter(x = exercise_data['Chromosome'], y = exercise_data['mean_pvalue'], c = exercise_data['Strand'].map(colors))
> > plt.ylabel('Average P-value')
> > plt.xlabel('Chromosome')
> > ```
> > ![Another scatterplot showing chromosome vs average p-value, but every column has both a blue and red point, presumably showing the values for different strands.](../../images/python-plotting/Figure15_Exercise_plot.png)
> >
> {: .solution}
{: .question}


## Making your plots accessible
Whenever you are generating plots to go into a paper or a presentation, there are a few things you can do to make sure that everyone can understand your plots.

Always make sure your text is large enough to read. Use the `fontsize` parameter in `xlabel`, `ylabel`, `title`, and `legend`, and `tick_params` with `labelsize` to increase the text size of the numbers on your axes.
Similarly, you should make your graph elements easy to see. Use `s` to increase the size of your scatterplot markers and `linewidth` to increase the sizes of your plot lines.
Using `color` (and nothing else) to distinguish between different plot elements will make your plots unreadable to anyone who is colorblind, or who happens to have a black-and-white office printer. For lines, the `linestyle` parameter lets you use different types of lines. For scatterplots, `marker` lets you change the shape of your points.


topic: data-science
tutorial: python-types
title: Python - Basic Types & Type Conversion
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: eb3a45e44672bce5a29d089878c1bac6
last_modified: 2026-02-26T21:12:14.905161
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-types/tutorial.html

content:
Python is a typed language, data has a type, and different types of data cannot always be connected immediately and might need some conversion step before they can be used together. For instance if you add a number to a number, what should happen? If you add a number to a message, what do you expect will happen?

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Types

Every value in a program has a specific type.

Name                  | Python Code | Represents
---                   | ---         | ---
Integer               | `int`       | represents positive or negative whole numbers like 3 or -512.
Floating point number | `float`     | represents real numbers like 3.14159 or -2.5.
Character string      | `str`       | text, written with either `'` or `"` quotes (they must match)

### Checking the Type

Use the built-in function `type` to find out what type a value has. This works on values as well as variables. But remember: the *value* has the type --- the *variable* is just a label.

Check the type of values with the [`type()`](https://docs.python.org/3.8/library/functions.html#type) function:

```python
print(type(52))
print(type(3.14159))
```

You can also check the types of variables

```python
fitness = 'average'
print(type(fitness))
```

### Methods

A value's type determines what the program can do to it. Some operations may work

```python
print(5 - 3)
```

And some operations may not work:

```python
print('hello' - 'h')
```

For instance, you can use the `+` and `*` operators on strings.

```python
full_name = 'Ahmed' + ' ' + 'Walsh'
print(full_name)
separator = '=' * 10
print(separator)
```

Some methods only accept specific types, or only work on specific types.

The built-in function `len` returns the length of your data. Which of the following would you expect to work? `len(string)`? `len(int)`?

```python
print(len(full_name))
print(len(52))
```

### Matching Types

Not all types support all operations, adding an integer to a string doesn't make much sense:

```python
print(1 + '2')
```

This does not work because it's ambiguous: should `1 + '2'` be `3` (a number) or `'12'` (a string)? Some types can be converted to other types by using the type name as a function.

```python
print(1 + int('2'))
print(str(1) + '2')
```

### Operation Support

Here is a quick chart showing which operations are allowed for each pair:

Left\Right | `int`  | `float` | `str`
---------- | ---    | -----   | ---
`int`      | `+-*/` | `+-*/`  | `*`
`float`    | `+-*/` | `+-*/`  | ``
`str`      | `*`    | ``      | `+`

As you can see you can do `3 * "test"` and `"test" * 3`, but it doesn't work with floats.

## Can mix integers and floats freely in operations.

Integers and floating-point numbers can be mixed in arithmetic. Python 3 (which we use) automatically converts integers to floats as needed.

```python
print(f'half is {1 / 2.0}')
print(f'three squared is {3.0 ** 2}')
```


## Variables only change value when something is assigned to them.

If we make one cell in a spreadsheet depend on another, and update the latter,
the former updates automatically. However, this does **not** happen in programming languages.

```python
variable_one = 1
variable_two = 5 * variable_one
variable_one = 2
print(f'first is {variable_one} and second is {variable_two}')
```

The computer reads the value of `first` when doing the multiplication, creates
a new value, and assigns it to `second`. After that, `second` does not remember
where it came from. Every computation happens line-by-line.

> <question-title>Fractions</question-title>
>
> What type of value is 3.14159?
> How can you find out?
>
> > <solution-title></solution-title>
> >
> > It is a floating-point number (often abbreviated "float").
> > It is possible to find out by using the built-in function `type()`.
> >
> > ```
> > print(type(3.14159))
> > <class 'float'>
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!

```

> <question-title>Automatic Type Conversion</question-title>
>
> What type of value is the result of (3.25 + 4)?
>
> > <solution-title></solution-title>
> >
> > It is a float:
> > integers are automatically converted to floats as necessary.
> >
> > ```
> > result = 3.25 + 4
> > print(f'result is {type(result)}')
> > ```
> >
> > ```
> > 7.25 is <class 'float'>
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!

```

> <question-title>Choose a Type</question-title>
>
> What type of value (integer, floating point number, or character string)
> would you use to represent each of the following?  Try to come up with more than one good answer for each problem.  For example, in  # 1, when would counting days with a floating point variable make more sense than using an integer?
>
> 1. Number of days since the start of the year.
> 2. Time elapsed from the start of the year until now in days.
> 3. Serial number of a piece of lab equipment.
> 4. A lab specimen's age
> 5. Current population of a city.
> 6. Average population of a city over time.
>
> > <solution-title></solution-title>
> >
> > The answers to the questions are:
> > 1. Integer, since the number of days would lie between 1 and 365.
> > 2. Floating point, since fractional days are required
> > 3. Character string if serial number contains letters and numbers, otherwise integer if the serial number consists only of numerals
> > 4. This will vary! How do you define a specimen's age? whole days since collection (integer)? date and time (string)?
> > 5. Choose floating point to represent population as large aggregates (eg millions), or integer to represent population in units of individuals.
> > 6. Floating point number, since an average is likely to have a fractional part.
> {: .solution}
{: .question}

> <question-title>Division Types</question-title>
>
> In Python 3, the `//` operator performs integer (whole-number) floor division, the `/` operator performs floating-point
> division, and the `%` (or *modulo*) operator calculates and returns the remainder from integer division:
>
> ```python
> print(f'5 // 3: {5 // 3}')
> print(f'5 / 3: {5 / 3}')
> print(f'5 % 3: {5 % 3}')
> ```
>
> ```
> 5 // 3: 1
> 5 / 3: 1.6666666666666667
> 5 % 3: 2
> ```
>
> If `num_subjects` is the number of subjects taking part in a study,
> and `num_per_survey` is the number that can take part in a single survey,
> write an expression that calculates the number of surveys needed
> to reach everyone once.
>
> > <solution-title></solution-title>
> > We want the minimum number of surveys that reaches everyone once, which is
> > the rounded up value of `num_subjects/ num_per_survey`. This is
> > equivalent to performing a floor division with `//` and adding 1. Before
> > the division we need to subtract 1 from the number of subjects to deal with
> > the case where `num_subjects` is evenly divisible by `num_per_survey`.
> > ```
> > num_subjects = 600
> > num_per_survey = 42
> > num_surveys = (num_subjects - 1) // num_per_survey + 1
> >
> > print(num_subjects, 'subjects,', num_per_survey, 'per survey:', num_surveys)
> > ```
> >
> > ```
> > 600 subjects, 42 per survey: 15
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!
```

> <question-title>Strings to Numbers</question-title>
>
> Where reasonable, `float()` will convert a string to a floating point number,
> and `int()` will convert a floating point number to an integer:
>
> ```
> print("string to float:", float("3.4"))
> print("float to int:", int(3.4))
> ```
>
> ```
> string to float: 3.4
> float to int: 3
> ```
>
> If the conversion doesn't make sense, however, an error message will occur.
>
> > <code-in-title>Python</code-in-title>
> > ```
> > print("string to float:", float("Hello world!"))
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: could not convert string to float: 'Hello world!'
> > ```
> {: .code-out}
>
>
> Given this information, what do you expect the following program to do?
>
> What does it actually do?
>
> Why do you think it does that?
>
> ```
> print("fractional string to int:", int("3.4"))
> ```
>
> > <solution-title></solution-title>
> > What do you expect this program to do? It would not be so unreasonable to expect the Python 3 `int` command to
> > convert the string "3.4" to 3.4 and an additional type conversion to 3. After all, Python 3 performs a lot of other
> > magic - isn't that part of its charm?
> >
> > ```
> > int("3.4")
> > ```
> > However, Python 3 throws an error.
> > ```
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: invalid literal for int() with base 10: '3.4'
> > ```
> >
> > Why? To be consistent, possibly. If you ask Python to perform two consecutive
> > typecasts, you must convert it explicitly in code.
> >
> > ```
> > int(float("3.4"))
> > ```
> >
> > ```
> > 3
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!
```

> <question-title>Arithmetic with Different Types</question-title>
>
> Which of the following will return the floating point number `2.0`?
> Note: there may be more than one right answer.
>
> ```
> first = 1.0
> second = "1"
> third = "1.1"
> ```
>
> 1. `first + float(second)`
> 2. `float(second) + float(third)`
> 3. `first + int(third)`
> 4. `first + int(float(third))`
> 5. `int(first) + int(float(third))`
> 6. `2.0 * second`
>
> > <solution-title></solution-title>
> >
> > Answer: 1 and 4 give exactly 2.0.
> > Answer 5 gives the value `2` which may be considered equivalent, but is not returning a float specifically.
> {: .solution}
{: .question}

```python
# Test out solutions here!
```


topic: data-science
tutorial: gnmx-lecture7
title: Versioning your code and data with git
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: baf2c61f1a15b82981842d5757111d23
last_modified: 2026-02-26T21:12:14.882295
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture7/tutorial.html

content:
[![XKCD1597](https://imgs.xkcd.com/comics/git.png)](https://xkcd.com/1597/)

# Setup and Introduction

## Lecture setup

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Start terminal within JupyterLab instance

## Git resources

- [The Git Book](https://git-scm.com/book/en/v2)
- Troubleshooting [PG13](https://dangitgit.com/en) [R](https://ohshitgit.com/)

## Version control

- Manages changes over time
- Enables collaboration
- Provides complete history

## Git versus GitHub

- Git - version control system
- GitHub - hosting service

## Initial fundamentals

## Logic

Why do we need version control? Well ... to control versions and avoid mess:

![Why version control?](https://i.imgur.com/vVGsuDQ.png "From tutorials by <a href='https://www.kylebradbury.org/index.html'>Kyle Bradbury</a>")

## Main commands

Here are some of the most fundamental commands in `git` repertoire:

![Main commands](https://i.imgur.com/Xyhswin.png "From tutorials by <a href='https://www.kylebradbury.org/index.html'>Kyle Bradbury</a>")

## Branches

A repository may have multiple branches:

![Branch flow](https://i.imgur.com/vtXzgGO.png "From <a href='https://gitbetter.substack.com/p/how-to-work-in-multiple-git-branches'>GitBetter</a>")

# Let's do it!

## Clone a repo

Let's clone a [sample repo](https://github.com/nekrut/git_foo_bar) from GitHub

```bash
git clone https://github.com/nekrut/git_foo_bar.git
```

## Check history of this repo using `git log`

```bash
$git log
commit ee99d64890f3b1cf0637a14269ab8738357e8dd8 (HEAD -> main, origin/main, origin/HEAD)
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Mon Mar 11 17:38:42 2024 -0400

    Update README.md

commit de89f51d8e124665713f6fd94cd46447d172033b
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 08:02:50 2023 -0500

    Create file2.txt

commit bc3e5c4a9d54203739bb90f29d92e20082b9e5d4
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 08:02:25 2023 -0500

    Create file1.txt

commit 99dd43783be37b08c1ca80cdc881eae537526396
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 08:00:34 2023 -0500

    Updated readme

commit 18ebdabfe6f92f33ea626994ce6c9998cbe63522
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 07:59:46 2023 -0500

    Initial commit
```

## Check status of the repo

```bash
$git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
```

## Let's change and stage a file

Use editor to modify `file1.txt`. Once it is saved, we can see what is happening by using `git status` again:

```bash
$git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")
```

You can see, that the file is modified, but it is *not staged*. To stage it for commit you need to explicitly add it to staging:

```bash
git add file1.txt 
```

If you run `git status` now you will get:

```bash
$git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/
```

## Oh &#128169; `get reset`

Running `git reset` will restore the *status quo* as it was prior to `git add`:

```bash
$git reset
Unstaged changes after reset:
M       file1.txt
```

and `git status` will look as it did prior to `git add`:

```bash
git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")
```

## Let's stage and commit

If you are indeed ready to go ahead let's `add`:

```bash
git add file1.txt
```

and `commit`

```bash
$git commit -m 'modified file1'
[main 476adb7] modified file1
 1 file changed, 1 insertion(+), 1 deletion(-)
```

If run `git log` you will this addition commit:

```bash
$git log
commit 476adb7b64a9330199fea13ca2e7523f7fe90189 (HEAD -> main)
Author: nekrut <anekrut@gmail.com>
Date:   Tue Feb 21 13:23:24 2023 +0000

    modified file1

```

or alternatively you can use `--oneline` flag:

```bash
$git log --oneline
476adb7 (HEAD -> main) modified file1
de89f51 (origin/main, origin/HEAD) Create file2.txt
bc3e5c4 Create file1.txt
99dd437 Updated readme
18ebdab Initial commit
```

## Oh &#128169; `get revert`

To roll everything back you can do this:

```bash
$git revert 476adb7
[main 1f13d5f] Revert "modified file1"
 1 file changed, 1 insertion(+), 1 deletion(-)
```

This will bring `vim` editor with a pre-filled rollback message:

![vim image](https://i.imgur.com/sEZNPN8.png)

To save this and get out:

- press <kbd>ESC</kbd>
- type `:wq`
- hit <kbd>Enter</kbd>

Now, let's look at the log (showing just two lat commits):

```bash
 git log
commit 1f13d5fcf38057db9a90066b99aa475a6eeb1bae (HEAD -> main)
Author: nekrut <anekrut@gmail.com>
Date:   Tue Feb 21 13:31:24 2023 +0000

    Revert "modified file1"
    
    This reverts commit 476adb7b64a9330199fea13ca2e7523f7fe90189.

commit 476adb7b64a9330199fea13ca2e7523f7fe90189
Author: nekrut <anekrut@gmail.com>
Date:   Tue Feb 21 13:23:24 2023 +0000

    modified file1
```

## Let's actually do make changes and highlight them with diff

First, let's modify, say, `file2.txt` by adding a line. In this example I modified it from:

```
This
is
another
file
I've
made
```

to 

```
This
is
file2
I've
made
```

Now `add` and `commit`:

```bash
$ git status
On branch main
Your branch is ahead of 'origin/main' by 2 commits.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file2.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")

$ git add file2.txt 

$ git commit -m 'changed file2'
[main a947686] changed file2
 1 file changed, 1 insertion(+), 2 deletions(-)
```

Let's now compare the changes between two commits:

```bash
$ git diff a947686 1f13d5f
diff --git a/file2.txt b/file2.txt
index 1f383ac..e13e461 100644
--- a/file2.txt
+++ b/file2.txt
@@ -1,5 +1,6 @@
 This
 is
-file2
+another
+file
 I've
 made
```
 
 here:
 
 - `a` and `b` = tags of files being compared
 - `---` and `+++` marked to indiciate differences between `a` and `b`
 - `@@ -1,5 +1,6 @@` file chunk header which has the following format:
     - `@@ [file a range][file b range] @@`
     - File ranges are: `<start line><number of lines>`

## Branches

![git branches](https://i.imgur.com/cLCQ650.png)

> From [GitBetter](https://gitbetter.substack.com/p/how-to-work-in-multiple-git-branches)

To enable collaborations and to give the ability to develop major features without disrupting production (`master` or `main`) branch Git allows creation of multiple branches.  

### Create a new branch and switch to it

Let's create branch dev:

```bash
$ git branch dev
```

to see existing branches:

```bash
$ git branch
  dev
* main
```

The `main` branch is active (tagged with `*`). To actually switch branches:

```bash
$ git checkout dev
Switched to branch 'dev'
```

### Make changes

Let's make some changes, say, to `file1.txt` from this:

```
This
is
a 
file
```

to this:

```
This
is
a 
file1
```

and then get `status`, `add`, and `commit`:

```bash
$ git status
On branch dev
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")

$ git add file1.txt 

$ git commit -m 'changes to file1.txt'
[dev 05ce64d] changes to file1.txt
 1 file changed, 1 insertion(+), 1 deletion(-)
 ```

### Switch to another branch and look at the modified file

If we switch back to `main` and look at the content of `file1.txt` we will see the "old" content:

```
This
is
a 
file
```

### Merge branches

To incorporate changes from `dev` to `master` we need to do a merge:

```bash
$ git merge dev main
Updating a947686..05ce64d
Fast-forward
 file1.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
```

and if we look at `file1.txt` again we will see the change:

$ more file1.txt 
This
is
a 
file1

We can now delete the branch:

```bash
$ git branch -d dev
Deleted branch dev (was 05ce64d).
```

### Merging with conflicts

To simulate a merging conflict we need to make different changes to the same line of a file in different branches. 

#### Modify `file1.txt` in `main`

First, let's checkout `main` and make the following change to `file1.txt`, say, from 

```
This
is
a 
file1
```

to 

```
This
is
a 
file1 - the first file we created
```

now we add and commit:

```bash
$ git add file1.txt 

$ git commit -m 'main changes to file1'
[main dfd0598] main changes to file1
 1 file changed, 1 insertion(+), 1 deletion(-)
```

> <tip-title>Showing differences between branches</tip-title>
>
>You can use `git diff` to see the differences between branches such as, for example:
>
> ```bash
> $ git diff main dev
> diff --git a/file1.txt b/file1.txt
> index f8e8191..e82ee85 100644
> --- a/file1.txt
> +++ b/file1.txt
> @@ -1,4 +1,4 @@
>  This
>  is
>  a 
> -file1 - the first file we created
> +file1
> ```
{: .tip}

#### Modify `file1.txt` in `dev`

Checkout `dev`:

```bash
$ git checkout dev
Switched to branch 'dev'
```

Change `file1.txt`, say, from:


```
This
is
a 
file1
```

to

```
This
is
a 
file1 - the first file
```

now we add and commit:

```bash
$ git add file1.txt 

$ git commit -m 'dev changes to file1'
[dev b2b8b9e] dev changes to file1
 1 file changed, 1 insertion(+), 1 deletion(-)
 ```

#### Checkout `main` and try to `merge`


```bash
 $ git merge main dev
 Auto-merging file1.txt
 CONFLICT (content): Merge conflict in file1.txt
 Automatic merge failed; fix conflicts and then commit the result.
```

If then actually look at the content of the file, you will see this:

![file1](https://i.imgur.com/s1zuCN6.png)


so here you need to decide which version you will keep and edit the file correspondingly.  For example, I edited it to this form and saved:

![file1_edited](https://i.imgur.com/1IbIQmD.png)

Now I need to `add` and `commit`:

```bash
 $ git add file1.txt
 jovyan@jupyter-jupyterlab-2djupyterlab-2ddemo-2dryet2dsd:~/bmmb554_foo_bar$ git commit -m 'merged dev into main'
```


topic: data-science
tutorial: python-subprocess
title: Python - Subprocess
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: ef5c726c7ee961fdbbbc85b7eb26f45b
last_modified: 2026-02-26T21:12:14.904629
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-subprocess/tutorial.html

content:
Sometimes you need to run other tools in Python, like maybe you want to
Here we'll give a quick tutorial on how to read and write files within Python.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Subprocesses

Programs can run other programs, and in Python we do this via the `subprocess` module. It lets you run any other command on the system, just like you could at the terminal.

The first step is importing the module

```python
import subprocess
```

You'll primarily use two functions:

```python
help(subprocess.check_call)
```

Which executes a command and checks if it was successful (or it raises an exception), and

```python
help(subprocess.check_output)
```


# Check Call: Downloading Files

Which executes a command returns the output of that command. This is really useful if you're running a subprocess that writes something to stdout, like a report you need to parse. We'll learn how to use these by running two gene callers, augustus and glimmer. You can install both from Conda if you do not have them already.

```
conda create -n subprocess augustus glimmer3
```

Additionally you'll need two files, you *generally should not do this*, but you can use a subprocess to download the file! We'll use `subprocess.check_call` for this which simply executes the program, and continues on. If there is an error in the execution, it will raise an exception and stop execution.

```python
url = "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/836/945/GCF_000836945.1_ViralProj14044/GCF_000836945.1_ViralProj14044_genomic.fna.gz"
genome = 'Escherichia virus T4.fna.gz'
subprocess.check_call(['wget', url, '-O', genome])
subprocess.check_call(['gzip', '-d', genome])
```

> <tip-title>What do these commands look like on the CLI?</tip-title>
> ```
> wget https://ftp.ncbi.nlm.nih.gov/.... -O "Escherichia virus T4.fna.gz"
> gzip -d "Escherichia virus T4.fna.gz"
> ```
{: .tip}

The above segment
- sets a url variable
- sets an output filename, `Escherichia virus T4.fna.gz`
- runs `check_call` with a single argument: a list
    - `wget` a tool we use to download files
    - the URL
    - `-O` indicating the next argument will be the 'output name'
    - what we want the output filename to be called
- runs `check_call` with a single argument: a list
    - `gzip` a tool to decompress files
    -`-d` indicating we want to decompress
    - and the filename.

This list is especially important. When you run commands on the command line, normally you just type in a really bit of text by yourself. It's one big string, and you're responsible for making sure quotation marks appear in the right place. For instance, if you have spaces in your filenames, you have to quote the filename. Python requires you specify a list of arguments, and then handles the quoting for you! Which, honestly, is easier and safer.

> > <code-in-title>Terminal</code-in-title>
> > Here we manually quote the argument
> > ```
> > glimmer3 "bow genome.txt"
> > ```
> {: .code-in}
> > <code-out-title>Python</code-out-title>
> > Here python handles that for us
> > ```
> > subprocess.check_call(['glimmer3', 'bow genome.txt'])
> > ```
> {: .code-out}
{: .code-2col}

> <tip-title>Exploitation!</tip-title>
> This is one of the major reasons we don't use `os.system` or older Python interfaces for running commands.
> If you're processing files, and a user supplies a file with a space, if your program isn't expecting that space in that filename, then it could do something dangerous!
> Like exploit your system!
>
> So, **always** use `subprocess` if you run to commands, never any other module, *despite what you see on the internet!*
{: .tip}


There are more functions in the module, but the vast majority of the time, those are sufficient.

# Check Output: Gene Calling with Augustus

```python
gff3 = subprocess.check_output([
    'augustus', # Our command
    '--species=E_coli_K12', # the first argument, the species, we're using a phage so we call genes  based on it's host organism
    'Escherichia virus T4.fna', # The path to our genome file, without the .gz because we decompressed it.
    '--gff3=on' # We would like gff3 formatted output (it's easier to parse!)
])

gff3 = gff3.decode('utf-8')
gff3 = gff3.split('\n')
```

> <tip-title>What does this commands look like on the CLI?</tip-title>
> ```
> augustus --species=E_coli_K12 'Escherichia virus T4.fna' --gff3=on
> ```
{: .tip}

If you're using `subprocess.check_output()` python doesn't return plain text `str` to you, instead it returns a `bytes` object. We can decode that into text with `.decode('utf-8')`, a phrase you should memorise as going next to `check_output()`, for 99% of use cases.

Let's look at the results!

```python
print(gff3[0:20])
```

It's a lot of comment lines, starting with `#`. Let's remove those

```python
cleaned_gff3 = []
for line in gff3:
    if line.startswith('#'):
        continue
    cleaned_gff3.append(line)

print(cleaned_gff3[0:20])
```

And now you've got a set of gff3 formatted gene calls! You can use all of your loop processing skills to slice and dice this data into something great!

# Aside: `stdin`, `stderr`, `stdout`

All unix processes have three default file handles that are available to them:

- `stdin`, where data is passed to the program via a pipe. E.g. `generate-data | my-program`, there the program would read the output of `generate-data` from the pipe.
- `stdout`, the default place where things are written. E.g. if you `print()` in python, it goes to `stdout`. People often redirect `stdout` to a file, like `my-program > output.txt` to save the output.
- `stderr`, *generally* if your program produces output on `stdout`, you might still want to log messages (errors, % done, etc.) If you write to `stdout`, it might get mixed in with the user's outputs, so we write to `stderr`, which also gets printed to the screen, and looks identical as any print statement, but it's coming from a separate pipe.

# Pipes

One of the more complicated cases, however, is when you need pipes.

```python
url = "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/001/721/125/GCF_001721125.1_ASM172112v1/GCF_001721125.1_ASM172112v1_cds_from_genomic.fna.gz"
cds = 'E. Coli CDSs.fna.gz'
subprocess.check_call(['wget', url, '-O', cds])
subprocess.check_call(['gzip', '-d', cds])
```

With subprocesses, you can control the stdin, and stdout of the process by using file handles.

> > <code-in-title>Terminal</code-in-title>
> > Here we pipe a file to a process named `build-icm` which takes one argument, the output name. It reads sequences from stdin.
> > ```
> > cat seq.fa | build-icm test.icm
> > # OR
> > build-icm test.icm < seq.fa
> > ```
> {: .code-in}
> > <code-out-title>Python</code-out-title>
> > Here we need to do a bit more.
> > 1. Open a file handle
> > 2. Pass that file handle to `check_call` or `check_output`. This determines where stdin comes from.
> > ```
> > with open('seq.fa', 'r') as handle:
> >     subprocess.check_call(['build-icm', 'test.icm'], stdin=handle)
> > ```
> {: .code-out}
{: .code-2col}

We'll do that now:

```python
with open('E. Coli CDSs.fna', 'r') as handle:
    subprocess.check_call(['build-icm', 'test.icm'], stdin=handle)
```

> <tip-title>What does this commands look like on the CLI?</tip-title>
> ```
> build-icm test.icm < 'E. Coli CDSs.fna'
> ```
{: .tip}

Here we build a model, based on the sequences of *E. Coli* K-12, that Glimmer3 can use.

```python
output = subprocess.check_output([
    'glimmer3', # Our program
    'Escherichia virus T4.fna', # The input genome
    'test.icm', # The model we just built
    't4-genes'  # The base name for output files. It'll produce t4-genes.detail and t4-genes.predict.
]).decode('utf-8') # And of course we decode as utf-8

print(output)
```

> <tip-title>What does this commands look like on the CLI?</tip-title>
> ```
> glimmer3 'Escherichia virus T4.fna' test.icm t4-genes
> ```
{: .tip}

*What happened here?* The output of the program was written to `stderr`, not `stdout`, so Python may print that out to your screen, but `output` will be empty. To solve this common problem we can re-run the program and collect both `stdout` and `stderr`.

```python
output = subprocess.check_output([
    'glimmer3', # Our program
    'Escherichia virus T4.fna', # The input genome
    'test.icm', # The model we just built
    't4-genes'  # The base name for output files. It'll produce t4-genes.detail and t4-genes.predict.
], stderr=subprocess.STDOUT).decode('utf-8') # And of course we decode as utf-8

print(output)
```

Here we've re-directed the `stderr` to `stdout` and mixed both of them together. This isn't always what we want, but here the program produces no output, and we can do that safely, and now we can parse it or do any other computations we need with it! Our Glimmer3 gene calls are in `t4-genes.detail` and `t4-genes.predict` if we want to open and process those as well.


topic: data-science
tutorial: r-basics
title: R basics in Galaxy
difficulty: introductory
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: ["interactive-tools", "R"]
content_hash: f0fc10dbca7947fe1d0d7399bde1d4a1
last_modified: 2026-02-26T21:12:14.908012
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-basics/tutorial.html

content:
This tutorial will introduce R basics, using an RStudio Interactive Tool in Galaxy

{% include topics/data-science/tutorials/r-basics/tutorial_origin.md %}
{% include topics/data-science/tutorials/r-basics/r_introduction.md %}
{% include topics/data-science/tutorials/r-basics/r_history.md %}
{% include topics/data-science/tutorials/r-basics/r_advantages.md %}
{% include topics/data-science/tutorials/r-basics/disclaimer.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Before diving in the tutorial, we need to open {% tool [RStudio](interactive_tool_rstudio) %}. If you do not know how or never interacted with RStudio, please follow the [dedicated tutorial]({% link topics/galaxy-interface/tutorials/rstudio/tutorial.md %}).

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

# Creating objects in R

> <comment-title>Reminder</comment-title>
> At this point you should be coding along in the `r_basics.R` script we created in the last episode. Writing your commands in the script(and commenting it) will make it easier to record what you did and why.
{: .comment}

What might be called a variable in many languages is called an **object** in R.

To create an object you need:

- a name (e.g. `a`)
- a value (e.g. `1`)
- the assignment operator (`<-`)

> <hands-on-title>Create a first object</hands-on-title>
>
> 1. Assign `1` to the object `a` using the R assignment operator `<-` in your script
> 2. Write a comment in the line above
>
>    ```R
>    # this line creates the object 'a' and assigns it the value '1'
>    a <- 1
>    ```
>
> 3. Select the lines
> 4. Execute them
>
>    > <tip-title>Execute from a script</tip-title>
>    > - Click on the **Run the current line or selection**
>    > - Type <kbd>CTRL</kbd>+<kbd>Enter</kbd> (or <kbd>CMD</kbd>+<kbd>Enter</kbd>)
>    {: .tip}
>
> 5. Check the **Console** and **Environment** panels
>
{: .hands_on}

The **Console** displays the lines of code run from the script and any outputs or status/warning/error messages (usually in red).

In the **Environment**, we have now a table:

Values | |
--- | ---
 a | 1

This **Environment** window allows you to keep track of the objects you have created in R.

> <question-title>Exercise: Create some objects in R</question-title>
>
> Create the following objects, with an appropriate name (your best guess at what name to use is fine):
>
> 1. Create an object that has the value of number of pairs of human chromosomes
> 2. Create an object that has a value of your favorite gene name
> 3. Create an object that has this URL as its value (`ftp://ftp.ensemblgenomes.org/pub/bacteria/release-39/fasta/bacteria_5_collection/escherichia_coli_b_str_rel606/`)
> 4. Create an object that has the value of the number of chromosomes in a diploid human cell
>
> > <solution-title></solution-title>
> >
> > Here as some possible answers to the challenge:
> >
> > 1. `human_chr_number <- 23`
> > 2. `gene_name <- 'pten'`
> > 3. `ensemble_url <- 'ftp://ftp.ensemblgenomes.org/pub/bacteria/release-39/fasta/bacteria_5_collection/escherichia_coli_b_str_rel606/'`
> > 4. `human_diploid_chr_num <- 46`
> >
> {: .solution}
{: .question}

#  Naming objects in R

Here are some important details about naming objects in R.

- **Avoid spaces and special characters**

    Object names cannot contain spaces or the minus sign (`-`). You can use `_` to make names more readable. You should avoid using special characters in your object name (e.g. `!`, `@`, `#`, `.` , etc.). Also, object names cannot begin with a number.

- **Use short, easy-to-understand names**

    You should avoid naming your objects using single letters (e.g. `n`, `p`, etc.). This is mostly to encourage you to use names that would make sense to anyone reading your code (a colleague, or even yourself a year from now). Also, avoiding excessively long names will make your code more readable.

- **Avoid commonly used names**

    There are several names that may already have a definition in the R language (e.g. `mean`, `min`, `max`). One clue that a name already has meaning is that if you start typing a name in RStudio and it gets a colored highlight or RStudio gives you a suggested autocompletion you have chosen a name that has a reserved meaning.

- **Use the recommended assignment operator**

    In R, we use `<-` as the preferred assignment operator. `=` works too, but is most commonly used in passing arguments to functions (more on functions later). There is a shortcut for the R assignment operator:
    - Windows execution shortcut: <kbd>Alt</kbd>+<kbd>-</kbd>
    - Mac execution shortcut: <kbd>Option</kbd>+<kbd>-</kbd>

There are a few more suggestions about naming and style you may want to learn more about as you write more R code. There are several "style guides" that have advice, and one to start with is the [tidyverse R style guide](http://style.tidyverse.org/index.html).

> <comment-title>Pay attention to warnings in the script console</comment-title>
>
> If you enter a line of code in your script that contains an error, RStudio may give you an error message and underline this mistake. Sometimes these messages are easy to understand, but often the messages may need some figuring out. Paying attention to these warnings will help you avoid mistakes.
>
> In the example below, the object name has a space, which is not allowed in R. The error message does not say this directly, but R is "not sure" about how to assign the name to `human_ chr_number` when the object name we want is `human_chr_number`.
>
> ![rstudio script warning](../../images/r-basics/rstudio_script_warning.png)
{: .comment}

# Reassigning object names or deleting objects

Once an object has a value, you can change that value by overwriting it. R will not give you a warning or error if you overwriting an object, which may or may not be a good thing depending on how you look at it.

> <hands-on-title>Overwrite an object</hands-on-title>
>
> 1. Overwrite the `gene_name` with the `tp53`
>
>    ```R
>    # gene_name has the value 'pten' or whatever value you used in the challenge.
>    # We will now assign the new value 'tp53'
>    gene_name <- 'tp53'
>    ```
>
> 2. Check the new value in the **Environment** panel
> 3. Type `gene_name` in the **Script** panel
> 4. Check in the **Console** panel
>
>    ```R
>    > gene_name
>    [1] "tp53"
>    ```
>
>    If you run a line of code that has only an object name, R will normally display the contents of that object.
{: .hands_on}

You can also remove an object from R's memory entirely

> <hands-on-title>Remove an object</hands-on-title>
>
> 1. Delete the `gene_name` object
>
>    ```R
>    # delete the object 'gene_name'
>    rm(gene_name)
>    ```
>
> 2. Check that `gene_name` is not displayed in the **Environment** panel
> 3. Type `gene_name` only in **Script** and check the **Console**
>
>    ```R
>    > gene_name
>    Error: object 'gene_name' not found
>    ```
>
>    R tells us that the object no longer exists.
{: .hands_on}

# Understanding object data types or modes

In R, **every object has two properties**:

- **Length**: how many distinct values are held in that object
- **Mode**: what is the classification (type) of that object.

    The **"mode" property** corresponds to the **type of data an object represents**. The most common modes you will encounter in R are:

    - **Numeric (num)**: numbers such floating point/decimals (1.0, 0.5, 3.14)

        > <comment-title>More specific types</comment-title>
        > There are also more specific numeric types (dbl - Double, int - Integer). These differences are not relevant for most beginners and pertain to how these values are stored in memory
        {: .comment}

    - **Character (chr)**: a sequence of letters/numbers in single `''` or double `" "` quotes
    - **Logical**: boolean values, `TRUE` or `FALSE`

    There are a few other modes (i.e. "complex", "raw" etc.) but these are the three we will work with in this lesson.

Data types are familiar in many programming languages, but also in natural language where we refer to them as the parts of speech, e.g. nouns, verbs, adverbs, etc. Once you know if a word - perhaps an unfamiliar one - is a noun, you can probably guess you can count it and make it plural if there is more than one (e.g. 1 [Tuatara](https://en.wikipedia.org/wiki/Tuatara), or 2 Tuataras). If something is a adjective, you can usually change it into an adverb by adding "-ly" (e.g. [jejune](https://www.merriam-webster.com/dictionary/jejune) vs. jejunely). Depending on the context, you may need to decide if a word is in one category or another (e.g "cut" may be a noun when it's on your finger, or a verb when you are preparing vegetables). These concepts have important analogies when working with R objects.

> <hands-on-title>Create an object and check its mode</hands-on-title>
>
> 1. Assign `'chr02'` to a `chromosome_name` object
>
>    ```R
>    chromosome_name <- 'chr02'
>    ```
>
> 2. Check the mode of the object
>
>    ```R
>    mode(chromosome_name)
>    ```
>
> 3. Check the result in the console
>
{: .hands_on}

The created object seems to a character object.

> <question-title>Create objects and check their modes</question-title>
>
> 1. Create the following objects in R
>    1. `od_600_value` with value `0.47`
>    2. `chr_position` with value `'1001701'`
>    3. `spock` with value `TRUE`
>    4. `pilot` with value `Earhart`
> 2. Guess their mode
> 3. Check them using `mode()`
>
> > <solution-title></solution-title>
> >
> > 1. Object creation
> >
> >    ```R
> >    > od_600_value <- 0.47
> >    > chr_position <- '1001701'
> >    > spock <- TRUE
> >    > pilot <- Earhart
> >    [1] Error in eval(expr, envir, enclos): object 'Earhart' not found
> >    ```
> >
> >    We cannot take a string of alphanumeric characters (e.g. `Earhart`) and assign as a value for an object. In this case, R looks for an object named `Earhart` but since there is no object, no assignment can be made.
> >
> > 2. Modes
> >    1. `od_600_value`: numeric
> >    2. `chr_position`: character
> >
> >       If a series of numbers are given as a value R will consider them to be in the "character" mode if they are enclosed as single or double quotes
> >
> >    3. `spock`: logical
> >    4. `pilot`
> >
> >       ```R
> >       > mode(pilot)
> >       Error in mode(pilot) : object 'pilot' not found
> >       ```
> >
> >       If `Earhart` did exist, then the mode of `pilot` would be whatever the mode of `Earhart` was originally.
> >
> {: .solution}
{: .question}

# Mathematical and functional operations on objects

Once an object exists (which by definition also means it has a mode), R can appropriately manipulate that object. For example, objects of the numeric modes can be added, multiplied, divided, etc. R provides several mathematical (arithmetic) operators including:

- `+`: addition
- `-`: subtraction
- `*`: multiplication
- `/`: division
- `^` or `**`: exponentiation
- `a%%b`: modulus (the remainder after division)

> <hands-on-title>Execute mathematical operations</hands-on-title>
>
> 1. Execute `(1 + (5 ** 0.5))/2`
>
>    ```R
>    > (1 + (5 ** 0.5))/2
>    [1] 1.618034
>    ```
>
> 2. Multiply the object `human_chr_number` by 2
>
>    ```R
>    > human_chr_number <- 23
>    # multiply the object 'human_chr_number' by 2
>    > human_chr_number * 2
>    [1] 46
>    ```
>
{: .hands_on}

> <question-title>Exercise: Compute the golden ratio</question-title>
>
> One approximation of the golden ratio ($$\varphi$$) is
>
> $$\frac{1 + \sqrt{5}}{2}$$
>
> Compute the golden ratio to 3 digits of precision using the `sqrt()` and `round()` functions.
>
> Hint: remember the `round()` function can take 2 arguments.
>
> > <solution-title></solution-title>
> >
> > ```R
> > > round((1 + sqrt(5))/2, digits = 3)
> > [1] 1.618
> > ```
> >
> > Notice that you can place one function inside of another.
> >
> {: .solution}
{: .question}

# Vectors

Vectors are probably the most used commonly used object type in R. **A vector is a collection of values that are all of the same type (numbers, characters, etc.)**.

One of the most common ways to create a vector is to use the `c()` function - the "concatenate" or "combine" function. Inside the function you may enter one or more values, separated by a comma.

> <hands-on-title>Create a vector</hands-on-title>
>
> 1. Create a `snp_genes` object containing "OXTR", "ACTN3", "AR", "OPRM1"
>
>    ```R
>    # Create the SNP gene name vector
>    > snp_genes <- c("OXTR", "ACTN3", "AR", "OPRM1")
>    ```
>
> 2. Check how this object is stored in the **Environment** panel
{: .hands_on}

Vectors always have a **mode** and a **length**. In the **Environment** panel, we could already have an insight on these properties: `chr [1:4]` may indicate character and 4 values.

> <hands-on-title>Check vector properties</hands-on-title>
>
> 1. Check the mode of `snp_genes` object
>
>    ```R
>    # Check the mode of 'snp_genes'
>    > mode(snp_genes)
>    [1] "character"
>    ```
>
> 2. Check the mode of `snp_genes` length
>
>    ```R
>    # Check the mode of 'snp_genes'
>    > length(snp_genes)
>    [1] "4"
>    ```
>
> 3. Check both properties using `str` function
>
>    ```R
>    # Check the structure of 'snp_genes'
>    > str(snp_genes)
>    [1] chr [1:4] "OXTR" "ACTN3" "AR" "OPRM1"
>    ```
{: .hands_on}

The `str()` (structure) function is giving the same information as the **Environmnent** panel.

Vectors are quite important in R. Another data type that we will not work in this tutorial but in extra tutorial: data frames, are collections of vectors. What we learn here about vectors will pay off even more when we start working with data frames.

## Creating and subsetting vectors

Once we have vectors, one thing we may want to do is specifically retrieve one or more values from our vector. To do so, we use **bracket notation**. We type the name of the vector followed by square brackets. In those square brackets we place the index (e.g. a number) in that bracket.

> <hands-on-title>Get values from vectors</hands-on-title>
>
> 1. Create several vectors
>    - `snps` object with "rs53576", "rs1815739", "rs6152", "rs1799971"
>    - `snp_chromosomes` object with "3", "11", "X", "6"
>    - `snp_positions` object with 8762685, 66560624, 67545785, 154039662
>
>    ```R
>    # Some interesting human SNPs
>    # while accuracy is important, typos in the data won't hurt you here
>    snps <- c('rs53576', 'rs1815739', 'rs6152', 'rs1799971')
>    snp_chromosomes <- c('3', '11', 'X', '6')
>    snp_positions <- c(8762685, 66560624, 67545785, 154039662)
>    ```
>
> 2. Get the 3rd value in the `snp_genes` vector
>
>    ```R
>    # get the 3rd value in the snp_genes vector
>    > snp_genes[3]
>    [1] "AR"
>    ```
{: .hands_on}

In R, every item your vector is indexed, starting from the first item (1) through to the final number of items in your vector.

You can also retrieve a range of numbers:

> <hands-on-title>Retrieve a range of values from vectors</hands-on-title>
> 1. Get the 1st through 3rd value in the `snp_genes` vector
>
>    ```R
>    # get the 1st through 3rd value in the snp_genes vector
>    > snp_genes[1:3]
>    [1] "OXTR"  "ACTN3" "AR"
>    ```
>
> 2. Get the 1st, 3rd, and 4th value in the `snp_genes` vector
>
>    ```R
>    # get the 1st, 3rd, and 4th value in the snp_genes vector
>    > snp_genes[c(1, 3, 4)]
>    [1] "OXTR"  "AR"    "OPRM1"
>    ```
{: .hands_on}

To retrieve several (but not necessarily sequential) items from a vector, you pass a **vector of indices**, a vector that has the numbered positions you wish to retrieve.

There are additional (and perhaps less commonly used) ways of subsetting a vector (see [these
examples](https://thomasleeper.com/Rcourse/Tutorials/vectorindexing.html)). Also, several of these subsetting expressions can be combined.

> <hands-on-title>Retrieve a complex range of values from vectors</hands-on-title>
> 1. Get the 1st through the 3rd value, and 4th value in the `snp_genes` vector
>
>    ```R
>    # get the 1st through the 3rd value, and 4th value in the snp_genes vector
>    # yes, this is a little silly in a vector of only 4 values.
>    > snp_genes[c(1:3,4)]
>    [1] "OXTR"  "ACTN3" "AR"    "OPRM1"
>    ```
{: .hands_on}

## Adding to, removing, or replacing values in existing vectors

Once you have an existing vector, you may want to add a new item to it. To do so, you can use the `c()` function again to add your new value.

> <hands-on-title>Add values to vectors</hands-on-title>
> 1. Add "CYP1A1", "APOA5" to `snp_genes` vector
>
>    ```R
>    # add the gene 'CYP1A1' and 'APOA5' to our list of snp genes
>    # this overwrites our existing vector
>    snp_genes <- c(snp_genes, "CYP1A1", "APOA5")
>    ```
>
> 2. Check the content of `snp_genes`
>
>    ```R
>    > snp_genes
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1" "APOA5"
>    ```
{: .hands_on}

To remove a value from a vection, we can use a negative index that will return a version a vector with that index's value removed.

> <hands-on-title>Remove values to vectors</hands-on-title>
> 1. Check value corresponding to `-6` in `snp_genes`
>
>    ```R
>    > snp_genes[-6]
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1"
>    ```
>
> 2. Remove the 6th value of `snp_genes`
>
>    ```R
>    > snp_genes <- snp_genes[-6]
>    > snp_genes
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1"
>    ```
{: .hands_on}

We can also explicitly rename or add a value to our index using double bracket notation.

> <hands-on-title>Rename values in vectors</hands-on-title>
> 1. Rename the 7th value to "APOA5"
>
>    ```R
>    > snp_genes[[7]]<- "APOA5"
>    > snp_genes
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1" NA       "APOA5"
>    ```
{: .hands_on}

Notice in the operation above that R inserts an `NA` value to extend our vector so that the gene "APOA5" is an index 7. This may be a good or not-so-good thing depending on how you use this.

> <question-title>Exercise: Examining and subsetting vectors</question-title>
>
> Which of the following are true of vectors in R?
> 1. All vectors have a mode **or** a length
> 2. All vectors have a mode **and** a length
> 3. Vectors may have different lengths
> 4. Items within a vector may be of different modes
> 5. You can use the `c()` to one or more items to an existing vector
> 6. You can use the `c()` to add a vector to an exiting vector
>
> > <solution-title></solution-title>
> >
> > 1. False: vectors have both of these properties
> > 2. True
> > 3. True
> > 4. False: vectors have only one mode (e.g. numeric, character) and all items in a vector must be of this mode.
> > 5. True
> > 6. True
> >
> {: .solution}
{: .question}

## Logical Subsetting

There is one last set of cool subsetting capabilities we want to introduce. It is possible within R to retrieve items in a vector based on a logical evaluation or numerical comparison.

For example, let's say we wanted get

> <hands-on-title>Subset logically vectors</hands-on-title>
> 1. Extract all of the SNPs in our vector of SNP positions (`snp_positions`) that were greater than 100,000,000
>
>    ```R
>    > snp_positions[snp_positions > 100000000]
>    [1] 154039662
>    ```
{: .hands_on}

In the square brackets you place the name of the vector followed by the comparison operator and (in this case) a numeric value. Some of the most common logical operators you will use in R are:

- `<`: less than
- `<=`: less than or equal to
- `>`: greater than
- `>=`: greater than or equal to
- `==`: exactly equal to
- `!=`: not equal to
- `!x`: not x
- `a | b`: a or b
- `a & b`: a and b

The reason why the expression `snp_positions[snp_positions > 100000000]` works can be better understood if you examine what the expression `snp_positions > 100000000` evaluates to:

```R
> snp_positions > 100000000
[1] FALSE FALSE FALSE  TRUE
```

The output above is a logical vector, the 4th element of which is `TRUE`. When you pass a logical vector as an index, R will return the true values:

```R
> snp_positions[c(FALSE, FALSE, FALSE, TRUE)]
[1] 154039662
```

If you have never coded before, this type of situation starts to expose the "magic" of programming. We mentioned before that in the bracket notation you take your named vector followed by brackets which contain an index: `named_vector[index]`. The "magic" is that the index needs to *evaluate to* a number. So, even if it does not appear to be an integer (e.g. 1, 2, 3), as long as R can evaluate it, we will get a result. That our expression `snp_positions[snp_positions > 100000000]` evaluates to a number can be seen in the following situation.

How to know which **index** (1, 2, 3, or 4) in our vector of SNP positions was the one that was greater than 100,000,000?

> <hands-on-title>Getting which indices of any item that evaluates as TRUE</hands-on-title>
> 1. Return the indices in our vector of SNP positions (`snp_positions`) that were greater than 100,000,000
>
>    ```R
>    > which(snp_positions > 100000000)
>    [1] 4
>    ```
{: .hands_on}

**Why this is important?** Often in programming we will not know what inputs and values will be used when our code is executed. Rather than put in a pre-determined value (e.g 100000000) we can use an object that can take on whatever value we need.

> <hands-on-title>Subset logically vectors</hands-on-title>
> 1. Create a `snp_marker_cutoff` containing `100000000`
> 1. Extract all of the SNPs in `snp_positions` that were greater than `snp_marker_cutoff`
>
> ```R
> > snp_marker_cutoff <- 100000000
> > snp_positions[snp_positions > snp_marker_cutoff]
> [1] 154039662
> ```
>
{: .hands_on}

Ultimately, it's putting together flexible, reusable code like this that gets at the "magic" of programming!

## A few final vector tricks

Finally, there are a few other common retrieve or replace operations you may want to know about. First, you can check to see if any of the values of your vector are missing (i.e. are `NA`). Missing data will get a more detailed treatment later, but the `is.NA()` function will return a logical vector, with TRUE for any NA value:

> <hands-on-title>Check for missing values</hands-on-title>
> 1. Check what are the missing values in `snp_genes`
>
>    ```R
>    # current value of 'snp_genes':
>    # chr [1:7] "OXTR" "ACTN3" "AR" "OPRM1" "CYP1A1" NA "APOA5"
>    > is.na(snp_genes)
>    [1] FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
>    ```
>
{: .hands_on}

Sometimes, you may wish to find out if a specific value (or several values) is present a vector. You can do this using the comparison operator `%in%`, which will return TRUE for any value in your collection that is in the vector you are searching.

> <hands-on-title>Check for presence of values</hands-on-title>
> 1. Check if "ACTN3" and "APOA5" are in `snp_genes`
>
>    ```R
>    # current value of 'snp_genes':
>    # chr [1:7] "OXTR" "ACTN3" "AR" "OPRM1" "CYP1A1" NA "APOA5"
>    # test to see if "ACTN3" or "APO5A" is in the snp_genes vector
>    # if you are looking for more than one value, you must pass this as a vector
>    > c("ACTN3","APOA5") %in% snp_genes
>    [1] TRUE TRUE
>    ```
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What data types/modes are the following vectors?
>    1. `snps`
>    2. `snp_chromosomes`
>    3. `snp_positions`
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > mode(snps)
>    > [1] "character"
>    > > mode(snp_chromosomes)
>    > [1] "character"
>    > > mode(snp_positions)
>    > [1] "double"
>    > ```
>    {: .solution}
>
> 2. Add the following values to the specified vectors:
>    1. Add "rs662799" to the `snps` vector
>    2. Add 11 to the `snp_chromosomes` vector
>    3. Add 116792991 to the `snp_positions` vector
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > snps <- c(snps, 'rs662799')
>    > > snps
>    > [1] "rs53576"   "rs1815739" "rs6152"    "rs1799971" "rs662799"
>    > > snp_chromosomes <- c(snp_chromosomes, "11") # did you use quotes?
>    > > snp_chromosomes
>    > [1] "3"  "11" "X"  "6"  "11"
>    > > snp_positions <- c(snp_positions, 116792991)
>    > > snp_positions
>    > [1]   8762685  66560624  67545785 154039662 116792991
>    > ```
>    {: .solution}
>
> 3. Make the following change to the `snp_genes` vector:
>    1. Create a new version of `snp_genes` that does not contain CYP1A1
>    2. Add 2 NA values to the end of `snp_genes`
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > snp_genes <- snp_genes[-5]
>    > > snp_genes <- c(snp_genes, NA, NA)
>    > > snp_genes
>    > [1] "OXTR"  "ACTN3" "AR"    "OPRM1" NA      "APOA5" NA      NA
>    > ```
>    >
>    {: .solution}
>
> 4. Using indexing, create a new vector named `combined` that contains:
>    - 1st value in `snp_genes`
>    - 1st value in `snps`
>    - 1st value in `snp_chromosomes`
>    - 1st value in `snp_positions`
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > combined <- c(snp_genes[1], snps[1], snp_chromosomes[1], snp_positions[1])
>    > > combined
>    > [1] "OXTR"    "rs53576" "3"       "8762685"
>    > ```
>    >
>    {: .solution}
>
> 5. What type of data is `combined`?
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > typeof(combined)
>    > [1] "character"
>    > ```
>    >
>    {: .solution}
{: .question}

## Coercing values

Sometimes, it is possible that R will misinterpret the type of data represented in a vector, or store that data in a mode which prevents you from operating on the data the way you wish. For example, a long list of gene names isn't usually thought of as a categorical variable, the way that your experimental condition (e.g. control, treatment) might be. More importantly, some R packages you use to analyze your data may expect characters as input, not factors. At other times (such as plotting or some statistical analyses) a factor may be more appropriate. Ultimately, you should know how to change the mode of an object.

First, its very important to recognize that coercion happens in R all the time. This can be a good thing when R gets it right, or a bad thing when the result is not what you expect. Consider:

```R
> chromosomes <- c('3', '11', 'X', '6')
> mode(chromosomes)
[1] "character"
```

Although there are several numbers in our vector, they are all in quotes, so we have explicitly told R to consider them as characters. However, even if we removed the quotes from the numbers, R would coerce everything into a character:

```R
> chromosomes_2 <- c(3, 11, 'X', 6)
> typeof(chromosomes_2)
[1] "character"
> chromosomes_2[1]
[1] "3"
```

We can use the `as.` functions to explicitly coerce values from one form into another.

> <hands-on-title>Coerce values</hands-on-title>
> 1. Create a `positions` object with "8762685", "66560624", "67545785", "154039662"
>
>    ```R
>    positions <- c("8762685", "66560624", "67545785", "154039662")
>    ```
>
> 2. Check the type and first element
>
>    ```R
>    > typeof(positions)
>    [1] "character"
>    > positions[1]
>    [1] "8762685"
>    ```
>
> 3. Coerce into a numeric type using `as.numeric()`
>
>    ```R
>    > snp_positions_2 <- as.numeric(positions)
>    > typeof(snp_positions_2)
>    [1] "double"
>    > snp_positions_2[1]
>    [1] 8762685
>    ```
{: .hands_on}

Here the coercion is straight forward, but what would happen if we tried using `as.numeric()` on `snp_chromosomes_2`

> <hands-on-title>Issue with coercing values</hands-on-title>
> 1. Create a `chromosomes_2` object with 3, 11, 'X', 6
>
>    ```R
>    chromosomes_2 <- c(3, 11, 'X', 6)
>    ```
>
> 2. Coerce into a numeric type using `as.numeric()`
>
>    ```R
>    > snp_chromosomes_2 <- as.numeric(chromosomes_2)
>    Warning message:
>    NAs introduced by coercion
>    ```
>
> 3. Check the content of `snp_chromosomes_2`
>
>    ```R
>    > snp_chromosomes_2
>    [1]  3 11 NA  6
>    ```
{: .hands_on}

We see that an `NA` value (R's default value for missing data) has been introduced.

Let's summarize this section on coercion with a few take home messages:

- Be careful to check the result when you explicitly coerce one data type into another (this is known as **explicit coercion**)

    Ideally, you should try to see if its possible to avoid steps in your analysis that force you to coerce.

- Check the structure (`str()`) of your data before working with them!

    R will sometimes coerce without you asking for it. This is called (appropriately) **implicit coercion**. For example when we tried to create a vector with multiple data types, R chose one type through implicit coercion.

# Lists

Lists are quite useful in R. You may come across lists in the way that some bioinformatics programs may store and/or return data to you. One of the key attributes of a list is that, unlike a vector, a list may contain data of more than one mode.

> <comment-title>Learning more about lists</comment-title>
> Learn more about creating and using lists using this [nice tutorial](https://r4ds.had.co.nz/vectors.html#lists). In this one example, we will create a named list and show you how to retrieve items from the list.
{: .comment}

> <hands-on-title>Create and manipulate list objects</hands-on-title>
> 1. Create a named list containing the genes, reference SNPs, chromomosome and position objects we create before
>
>    ```R
>    # Create a named list using the 'list' function and our SNP examples
>    snp_data <- list(genes = snp_genes,
>                     reference_snp = snps,
>                     chromosome = snp_chromosomes,
>                     position = snp_positions)
>    ```
>
>    > <comment-title>Easy reading</comment-title>
>    > For easy reading we have placed each item in the list on a separate line. Nothing special about this, you can do this for any multiline commands. Just make sure the entire command (all 4 lines) are highlighted before running.
>    >
>    > As we are doing all this inside the list() function use of the `=` sign is good style
>    {: .comment}
>
> 2. Examine the structure of the list
>
>    ```R
>    > str(snp_data)
>    List of 4
>     $ genes         : chr [1:8] "OXTR" "ACTN3" "AR" "OPRM1" ...
>     $ refference_snp: chr [1:5] "rs53576" "rs1815739" "rs6152" "rs1799971" ...
>     $ chromosome    : chr [1:5] "3" "11" "X" "6" ...
>     $ position      : num [1:5] 8.76e+06 6.66e+07 6.75e+07 1.54e+08 1.17e+08
>    ```
>
> 3. Get all values for the `position` object in the list
>
>    ```R
>    > snp_data$position
>    [1]   8762685  66560624  67545785 154039662 116792991
>    ```
>
> 4. Get the first value in the `position` object
>
>    ```R
>    > snp_data$position[1]
>    [1] 8762685
>    ```
{: .hands_on}


topic: data-science
tutorial: gnmx-lecture6
title: Data manipulation with Pandas
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 9e3aace33f1c4dbbef9b5a18eff14423
last_modified: 2026-02-26T21:12:14.881989
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture6/tutorial.html

content:
#  SARS-CoV-2 as a research problem

To learn about pandas we will use SARS-CoV-2 data. Before actually jumping to Pandas let's learn about the coronavirus molecular biology.

The following summary is based on these publications:

- [Masters:2006](http://dx.doi.org/10.1016/S0065-3527(06)66005-3)
- [Fehr and Perlman:2015](http://dx.doi.org/10.1007/978-1-4939-2438-7_1)
- [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218)
- [Kirchdoerfer:2016](http://dx.doi.org/10.1038/nature17200)
- [Walls:2020](http://dx.doi.org/10.1016/j.cell.2020.02.058)
- [Jackson:2022](https://www.nature.com/articles/s41580-021-00418-x)

## Genome organization

All coronaviruses contain non-segmented positive-strand RNA genome approx. 30 kb in length. It is invariably 5'-leader-UTR-replicase-S-E-M-N-3'UTR-poly(A). In addition,
it contains a variety of accessory proteins interspersed throughout the genome (see Fig. below; (From [Fehr and Perlman:2015](http://dx.doi.org/10.1007/978-1-4939-2438-7_1))).  

![Genome organozation](https://media.springernature.com/original/springer-static/image/chp%3A10.1007%2F978-1-4939-2438-7_1/MediaObjects/317916_1_En_1_Fig1_HTML.gif "Genomic organization of representative α, β, and γ CoVs. An illustration of the MHV genome is depicted at the top. The expanded regions below show the structural and accessory proteins in the 3′ regions of the HCoV-229E, MHV, SARS-CoV, MERS-CoV and IBV. Size of the genome and individual genes are approximated using the legend at the top of the diagram but are not drawn to scale. HCoV-229E human coronavirus 229E, MHV mouse hepatitis virus, SARS-CoV severe acute respiratory syndrome coronavirus, MERS-CoV Middle East respiratory syndrome coronavirus, IBV infectious bronchitis virus.")

The long replicase gene encodes a number of non-structural proteins (nsps) and occupies 2/3 of the genome. Because of -1 ribosomal frameshifting within ORFab approx. in 25% of the cases
polyprotein pp1ab is produced in place of pp1a. pp1a encodes 11 nsps while pp1ab encodes 15 (Fig. below from [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218)). 

![Discontinuous transcription](https://www.annualreviews.org/docserver/fulltext/virology/2/1/vi20265.f1.gif "Coronavirus genome structure and gene expression. (a) Coronavirus genome structure. The upper scheme represents the TGEV genome. Labels indicate gene names; L corresponds to the leader sequence. Also represented are the nsps derived from the processing of the pp1a and pp1ab polyproteins. PLP1, PLP2, and 3CL protease sites are depicted as inverted triangles with the corresponding color code of each protease. Dark gray rectangles represent transmembrane domains, and light gray rectangles indicate other functional domains. (b) Coronavirus genome strategy of sgmRNA expression. The upper scheme represents the TGEV genome. Short lines represent the nested set of sgmRNAs, each containing a common leader sequence (black) and a specific gene to be translated (dark gray). (c) Key elements in coronavirus transcription. A TRS precedes each gene (TRS-B) and includes the core sequence (CS-B) and variable 5′ and 3′ flanking sequences. The TRS of the leader (TRS-L), containing the core sequence (CS-L), is present at the 5′ end of the genome, in an exposed location (orange box in the TRS-L loop). Discontinuous transcription occurs during the synthesis of the negative-strand RNA (light blue), when the copy of the TRS-B hybridizes with the TRS-L. Dotted lines indicate the complementarity between positive-strand and negative-strand RNA sequences. Abbreviations: EndoU, endonuclease; ExoN, exonuclease; HEL, helicase; MTase, methyltransferase (green, N7-methyltransferase; dark purple, 2′-O-methyltransferase); nsp, nonstructural protein; PLP, papain-like protease; RdRp, RNA-dependent RNA polymerase; sgmRNA, subgenomic RNA; TGEV, transmissible gastroenteritis virus; TRS, transcription-regulating sequence; UTR, untranslated region.")

## Virion structure

Coronavirus is a spherical particle approx. 125 nm in diameter. It is covered with S-protein projections giving it an appearance of solar corona - hence the term coronavirus. Inside the envelope is a nucleocapsid that has helical symmetry far more typical of (-)-strand RNA viruses. There are four main structure proteins: spike (S), membrane (M), envelope (E), and nucleocapsid (N; Fig. below [Masters:2006](http://dx.doi.org/10.1016/S0065-3527(06)66005-3)): 

![Virion structure](https://ars.els-cdn.com/content/image/1-s2.0-S0065352706660053-gr1.jpg "Schematic of the coronavirus virion, with the minimal set of structural proteins")

Mature S protein is a trimer of two subunits: S1 and S2. The two subunits are produced from a single S-precursor by host proteases (see [Kirchdoerfer:2016](http://dx.doi.org/10.1038/nature17200); this however is not the case for all coronaviruses such as SARS-CoV). S1 forms the receptor-binding domain, while S2 forms the stalk. 

M protein is the most abundant structural component of the virion and determines its shape. It possibly exists as a dimer.

E protein is the least abundant protein of the capsid and possesses ion channel activity. It facilitates the assembly and release of the virus. In SARS-CoV it is not required for replication but is essential for pathogenesis. 

N proteins form the nucleocapsid. Its N- and C-terminal domains are capable of RNA binding. Specifically, it binds to transcription regulatory sequences and the genomic packaging signal. It also binds to `nsp3`
and M protein possible tethering viral genome and replicase-transcriptase complex. 

## Entering the cell

Virion attaches to the cells as a result of interaction between S-protein and a cellular receptor. In the case of SARS-CoV-2 (as is in SARS-CoV) angiotensin converting enzyme 2 (ACE2) serves as the receptor binding to C-terminal portion of S1 domain. After receptor binding S protein is cleaved at two sites within the S2 subdomain. The first cleavage separates the receptor-binding domain of S from the membrane fusion domain. The second cleavage (at S2') exposes the fusion peptide. The SARS-CoV-2 is different from SARS-CoV in that it contains a [furin](https://en.wikipedia.org/wiki/Furin) cleavage site that is processed during viral maturation within endoplasmatic reticulum (ER; see [Walls:2020](http://dx.doi.org/10.1016/j.cell.2020.02.058)). The fusion peptide inserts into the cellular membrane and triggers joining on the two heptad regions with S2 forming an antiparallel six-helix bundle resulting in fusion and release of the genome into the cytoplasm (Fig. below from [Jackson:2022](https://www.nature.com/articles/s41580-021-00418-x). 

![Entering the cell](https://i.imgur.com/DD7EO3G.png "Two spike (S) protein cleavage events are typically necessary for the coronavirus entry process: one at the junction of the S1 and S2 subunits and the other at the S2′ site, internal to the S2 subunit. In the case of SARS-CoV-2, the polybasic sequence at the S1–S2 boundary is cleaved during virus maturation in an infected cell, but the S2′ site is cleaved at the target cell following angiotensin-converting enzyme 2 (ACE2) binding. Virus binding to ACE2 (step 1) induces conformational changes in the S1 subunit and exposes the S2′ cleavage site in the S2 subunit. Depending on the entry route taken by SARS-CoV-2, the S2′ site is cleaved by different proteases. Left: If the target cell expresses insufficient transmembrane protease, serine 2 (TMPRSS2) or if a virus–ACE2 complex does not encounter TMPRSS2, the virus–ACE2 complex is internalized via clathrin-mediated endocytosis (step 2) into the endolysosomes, where S2′ cleavage is performed by cathepsins, which require an acidic environment for their activity (steps 3 and 4). Right: In the presence of TMPRSS2, S2′ cleavage occurs at the cell surface (step 2). In both entry pathways, cleavage of the S2′ site exposes the fusion peptide (FP) and dissociation of S1 from S2 induces dramatic conformational changes in the S2 subunit, especially in heptad repeat 1, propelling the fusion peptide forward into the target membrane, initiating membrane fusion (step 5 on the left and step 3 on the right). Fusion between viral and cellular membranes forms a fusion pore through which viral RNA is released into the host cell cytoplasm for uncoating and replication (step 6 on the left and step 4 on the right). Several agents disrupt interaction between the S protein and ACE2: ACE2 mimetics, therapeutic monoclonal antibodies targeting the neutralizing epitopes on the S protein and antibodies elicited by vaccination block virus binding to ACE2 and thus inhibit both entry pathways. By contrast, strategies targeting post-receptor-binding steps differ between the two pathways. Being a serine protease inhibitor, camostat mesylate restricts the TMPRSS2-mediated entry pathway. Hydroxychloroquine and chloroquine block endosomal acidification, which is necessary for cathepsin activity, and thus restrict the cathepsin-mediated entry pathway")

## Replication

The above figure shows that in addition to the full length (+)-strand genome there is a number of (+)-strand subgenomic RNAs corresponding to 3'-end of the complete viral sequence. All of these *subgenomic* RNAs (sgRNAs) share the same leader sequence that is present only once at the extreme 5'-end of the viral genome. These RNAs are produced via discontinuous RNA synthesis when the RNA-dependent RNA-polymerase (RdRp) switches template:

<iframe title="vimeo-player" src="https://player.vimeo.com/video/133088346" width="640" height="480" frameborder="0" allowfullscreen></iframe>

  &#9757; Model for the formation of genome high-order structures regulating N gene transcription. The upper linear scheme represents the coronavirus genome. The red line indicates the leader sequence in the 5′ end of the genome. The hairpin indicates the TRS-L. The gray line with arrowheads represents the nascent negative-sense RNA. The curved blue arrow indicates the template switch to the leader sequence during discontinuous transcription. The orange line represents the copy of the leader added to the nascent RNA after the template switch. The RNA-RNA interactions between the pE (nucleotides 26894 to 26903) and dE (nucleotides 26454 to 26463) and between the B-M in the active domain (nucleotides 26412 to 26421) and the cB-M in the 5′ end of the genome (nucleotides 477 to 486) are represented by solid lines. Dotted lines indicate the complementarity between positive-strand and negative-strand RNA sequences. Abbreviations: AD, active domain secondary structure prediction; B-M, B motif; cB-M, complementary copy of the B-M; cCS-N, complementary copy of the CS-N; CS-L, conserved core sequence of the leader; CS-N, conserved core sequence of the N gene; dE, distal element; pE, proximal element; TRS-L, transcription-regulating sequence of the leader (From [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218)).

Furthermore, [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218) suggests the coronavirus transcription model in which transcription initiation complex forms at the 3'-end of (+)-strand genomic RNA:

![Coronavirus transcription](https://www.annualreviews.org/docserver/fulltext/virology/2/1/vi20265.f3.gif "Three-step model of coronavirus transcription. (1) Complex formation. Proteins binding transcription-regulating sequences are represented by ellipsoids, the leader sequence is indicated with a red bar, and core sequences are indicated with orange boxes. (2) Base-pairing scanning. Negative-strand RNA is shown in light blue; the transcription complex is represented by a hexagon. Vertical lines indicate complementarity between the genomic RNA and the nascent negative strand. (3) Template switch. Due to the complementarity between the newly synthesized negative-strand RNA and the transcription-regulating sequence of the leader, template switch to the leader is made by the transcription complex to complete the copy of the leader sequence")

## How virus is screened

Two approaches: (1) RNAseq and (2) amplicons

RNAseq approach requires a large sample quantity, but is suitable for detecting low-frequency variants. Ampliconic approaches are useful for samples containing a small fraction of viral RNA and are
by far the most widely used (From [NEB](https://primer-monitor.neb.com/lineages)):

![Primer schemes](https://i.imgur.com/9wLeHnZ.png "Nucleotide variants related to lineages of concern are indicated in yellow above a schematic SARS-CoV-2 genome. Primers that do not overlap with variants associated with these lineages are shown in blue. Overlapping primers are indicated in orange.")

# Pandas!

> This is an aggregated tutorial relying on material from the following fantastic sources:
>-  [Justin Bois](http://justinbois.github.io/bootcamp/2020/index.html). It contains modified training datasets and adopts content to Colab environment.
>- [BIOS821 course at Duke](https://people.duke.edu/~ccc14/bios-821-2017/index.html)
>- [Pandas documentation](https://pandas.pydata.org/docs/user_guide/index.html/)
{: .quote}

Pandas (from "Panel Data") is an essential piece of scientific (and not only) data analysis infrastructure. It is, in essence, a highly optimized library for manipulating very large tables (or "Data Frames"). 

Today we will be using a single notebook that explains the basics of this powerful tool:

## Pandas learning resources

- [Getting started](https://pandas.pydata.org/docs/getting_started/index.html#getting-started) - official introduction from Pandas.
- [Data Science Tools](http://people.duke.edu/~ccc14/bios-821-2017/index.html) - Data Science for Biologists from Duke University.
- [Data Carpentry](https://datacarpentry.org/) - a collection of lessons *à la* Software Carpentry.


```python
# Pandas, conventionally imported as pd
import pandas as pd
```

Throughout your research career, you will undoubtedly need to handle data, possibly lots of data. The data comes in lots of formats, and you will spend much of your time **wrangling** the data to get it into a usable form.

Pandas is the primary tool in the Python ecosystem for handling data. Its primary object, the `DataFrame` is extremely useful in wrangling data. We will explore some of that functionality here and will put it to use in the next lesson.

# Basics

## The data set

The dataset we will be using is a subset of metadata describing SARS-CoV-2 datasets from the [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra).  

It is obtained by going to https://www.ncbi.nlm.nih.gov/sra and performing a query with the following search terms: `txid2697049[Organism:noexp]`.

Results are downloaded using `Send to:` menu selecting `File` and then `RunInfo`. Let's get these results into this notebook:


```python
!wget https://zenodo.org/records/10680001/files/sra_ncov.csv.gz
```


```python
!gunzip -c  sra_ncov.csv.gz | head
```

The first line contains the **headers** for each column. The data follow. Given the file I/O skills you recently learned, you could write some functions to parse this file and extract the data you want.
You can imagine that this might be kind of painful. However, if the file format is nice and clean like we more or less have here, we can use pre-built tools. Pandas has a very powerful function, `pd.read_csv()`
that can read in a CSV file and store the contents in a convenient data structure called a **data frame**. In Pandas, the data type for a data frame is `DataFrame`, and we will use "data frame" and "`DataFrame`" interchangeably.

## Reading in data

Let's first look at the doc string of `pd.read_csv()`.


```python
pd.read_csv?
```

Holy cow! There are so many options we can specify for reading in a CSV file. You will likely find reasons to use many of these throughout your research. For now, however, we do not need most of them.
So, let's load in the data set. Note that even though the dataset is compressed with `gzip` we do not need to do anything additional - pandas magically understands and uncompresses the data while loading it into the dataframe.


```python
df = pd.read_csv('sra_ncov.csv.gz')
```

The same result can be achieved directly without downloading the file first:

```python
df = pd.read_csv('https://zenodo.org/records/10680001/files/sra_ncov.csv.gz')
```

We now have the data stored in a data frame. We can look at it in the Jupyter Notebook since Jupyter will display it in a well-organized, pretty way. Note that because our dataframe is big,
we only display the first five rows using `head()` function:


```python
df.head()
```

## Indexing data frames

The data frame is a convenient data structure for many reasons that will become clear as we start exploring. Let's start by looking at how data frames are indexed. Let's try to look at the first row.


```python
df[0]
```

Yikes! Lots of errors. The problem is that we tried to index numerically by row. **We index DataFrames, by columns.** And no column has the name `0` in this data frame, though there could be.
Instead, you might want to look at the column with the percentage of correct face-matching tasks.


```python
df['Run'].head()
```

This gave us the numbers we were after. Notice that when it was printed, the index of the rows came along with it. If we wanted to pull out a single percentage correct, say corresponding to index `4`, we can do that.


```python
df['Run'][4]
```

However, this is **not** the preferred way to do this. It is better to use `.loc`. This gives the location in the data frame we want.

> <details-title><tt>loc</tt> versus <tt>iloc</tt></details-title>
>
>`loc` and `iloc` are both methods in the Pandas library for DataFrame indexing.
>
>`loc`: It is label-based indexing, meaning that you use the actual row and column labels to make selections. This means that you specify rows and columns based on their index labels.
> For example, df.loc[3, 'column_name'] will select the value in the third row and the column labeled 'column_name'.
>
>`iloc`: It is integer-based indexing, where you use the integer position of the rows and columns to make selections. This means that you specify rows and columns based on their integer position.
> For example, df.iloc[3, 2] will select the value in the fourth row and the third column (remembering that indexing starts at 0).
>
>In summary, `loc` uses labels for indexing, while `iloc` uses integer positions.
{: .details}

```python
df.loc[4, 'Run']
```

```python
df.iloc[4:6]
```

```python
df.iloc[4:6, [0,2,4]]
```

```python
df.loc[4:6, ['Run','size_MB','LibraryStrategy']]
```

```python
df.loc[4:6, 'Run':]
```

```python
df.loc[4:6, 'Run':'LibrarySource']
```

It is also important to note that **row indices need not be integers**. And you should not count on them being integers. In practice, you will almost never use row indices, but rather use **Boolean indexing**.

## Filtering: Boolean indexing of data frames

Let's say I wanted to pull out accession numbers of runs produced by Pacific Biosciences machines (in this table such datasets are labeled as `PACBIO_SMRT`. I can use Boolean indexing to specify the row. Specifically, I want the row for which `df['Platform'] == 'PACBIO_SMRT'`. You can essentially plop this syntax directly when using `.loc`.


```python
df.loc[df['Platform'] == 'PACBIO_SMRT', 'Run']
```

If I want to pull the whole record for that participant, I can use `:` for the column index.

```python
df.loc[df['Platform'] == 'PACBIO_SMRT', :].head(10)
```

Now, let's pull out all PacBio records that were obtained from Amplicon sequencing. We can again use Boolean indexing, but we need to use an `&` operator.
We have not covered this bitwise operator before, but the syntax is self-explanatory in the example below. Note that it is important that each Boolean operation you are doing is in parentheses because of the precedence of the operators involved.

```python
df.loc[(df['Platform'] == 'PACBIO_SMRT') & (df['LibraryStrategy'] == 'AMPLICON'), :].head(3)
```

What is going on will be clearer if we set up our Boolean indexing first, as follows.

```python
inds = (df['Platform'] == 'PACBIO_SMRT') & (df['LibraryStrategy'] == 'AMPLICON')
inds[2:6]
```

Notice that `inds` is an array (actually a Pandas `Series`, essentially a `DataFrame` with one column) of `True`s and `False`s. We can apply the `unique` function from [NumPy](https://numpy.org/) to see how
many `True` and `False` rows we have:

```python
import numpy as np
np.unique(inds, return_counts=True)
```

When we index with it using .loc, we get back rows where inds is True:

```python
df.loc[inds, :]
```

```python
df.loc[(df['Platform'] == 'PACBIO_SMRT') & (df['LibraryStrategy'] == 'AMPLICON'), :].head(3)
```

## Calculating with data frames

The SRA data contains (sometimes) size of the run in MB (`size_MB`). Let's suppose we want to consider only those run where `size_MB` is above 100. We might like to add a column to the data frame that specifies whether or not the corresponding run is above 100Mb. We can conveniently compute with columns. This is done element-wise.

```python
df['size_MB'] >= 100
```

This tells us which run is above 100 Mb. We can simply add this back to the data frame.


```python
# Add the column to the DataFrame
df['Over100Mb'] = df['size_MB'] >= 100

# Take a look
df.head()
```


```python
df.assign(over100 = df['size_MB']>=100)
```

## A note of `assign`

The `assign` method in the pandas library is used to add new columns to a pandas DataFrame. The method allows you to perform calculations or operations on existing columns to generate new columns without modifying the original DataFrame.

For example, you can use the `assign` method to add a new column that calculates the sum of two existing columns or to add a column based on a complex calculation that involves multiple columns. The method also supports adding columns based on external data sources, such as NumPy arrays or other pandas DataFrames.

Using `assign` method is useful because it allows you to create new columns and add them to a DataFrame in a way that is both readable and easy to maintain. The method is also chain-able, meaning you can add multiple columns in one call, making your code more concise and efficient.

## A note about vectorization

Notice how applying the `<=` operator to a `Series` resulted in **elementwise** application. This is called `vectorization`. It means that we do not have to write a `for` loop to do operations on the elements of a `Series` or other array-like object. Imagine if we had to do that with a `for` loop.


```python
big_runs = []
for size in df['size_MB']:
    big_runs.append(size >= 100)
```

This is cumbersome. The vectorization allows for much more convenient calculation. Beyond that, the vectorized code is almost always faster when using Pandas and Numpy because the looping is done with compiled code under the hood. This can be done with many operators, including those you've already seen, like `+`, `-`, `*`, `/`, `**`, etc.

## Outputting a new CSV file

Now that we added the `Over100Mb` column, we might like to save our data frame as a new CSV that we can reload later. We use `df.to_csv()` for this with the `index` kwarg to ask Pandas not to explicitly write the indices to the file.


```python
df.to_csv('over100Mb_data.csv', index=False)
```

Let's take a look at what this file looks like.

```python
!head over100Mb_data.csv
```

# Tidy data

[Hadley Wickham](https://en.wikipedia.org/wiki/Hadley_Wickham) wrote a [great article](http://dx.doi.org/10.18637/jss.v059.i10) in favor of "tidy data." Tidy data frames follow the rules:

1. Each variable is a column.
2. Each observation is a row.
3. Each type of observation has its own separate data frame.

This is less pretty to visualize as a table, but we rarely look at data in tables. Indeed, the representation of data that is convenient for visualization is different from that which is convenient for analysis.
A tidy data frame is almost always **much** easier to work with than non-tidy formats.

Also, let's take a look at this [article](https://dtkaplan.github.io/DataComputingEbook/chap-tidy-data.html#chap:tidy-data).

## The data set

The dataset we will be using is a list of all SARS-CoV-2 datasets in [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) as of January 20, 2021.  

It is obtained by going to https://www.ncbi.nlm.nih.gov/sra and performing a query with the following search terms: `txid2697049[Organism:noexp]`.

Results are downloaded using `Send to:` menu selecting `File` and then `RunInfo`. Let's get these results into this notebook:


```python
df = pd.read_csv('https://zenodo.org/records/10680001/files/sra_ncov.csv.gz')
df = df[df['size_MB']> 0].reset_index(drop=True)

# Take a look
df
```

This data set is in tidy format. Each row represents a single SRA dataset. The properties of each run are given in each column. We already saw the power of having the data in this format when we did Boolean indexing in the last lesson.

## Finding unique values and counts

How many unique sequencing platforms do we have?

```python
df['Platform'].unique()
```


```python
df['Platform'].value_counts()
```

## Sorting

(and axes!)

Let's start by sorting on index:


```python
df_subset = df.sample(n=10)
```


```python
df_subset
```


```python
df_subset.sort_index()
```


```python
df_subset.sort_index(axis = 1)
```

Now let's try sorting by values:


```python
df_subset.sort_values(by=['LibraryLayout'])
```


```python
df_subset.sort_values(by=['LibraryLayout','size_MB'])
```


```python
df_subset.sort_values(by=['LibraryLayout','size_MB'],ascending=[True,False])
```

# Split-apply-combine

Let's say we want to compute the total size of SRA runs for each `BioProject`. Ignoring for the second the mechanics of how we would do this with Pandas, let's think about it in English. What do we need to do?

1. **Split** the data set up according to the `'BioProject'` field, i.e., split it up so we have a separate data set for each BioProject ID.
2. **Apply** a median function to the activity in these split data sets.
3. **Combine** the results of these averages on the split data set into a new, summary data set that contains classes for each platform and medians for each.

We see that the strategy we want is a **split-apply-combine** strategy. This idea was put forward by Hadley Wickham in [this paper](http://dx.doi.org/10.18637/jss.v040.i01). It turns out that this is a strategy we want to use *very* often. Split the data in terms of some criterion. Apply some function to the split-up data. Combine the results into a new data frame.

Note that if the data are tidy, this procedure makes a lot of sense. Choose the column you want to use to split by. All rows with like entries in the splitting column are then grouped into a new data set. You can then apply any function you want into these new data sets. You can then combine the results into a new data frame.

Pandas's split-apply-combine operations are achieved using the `groupby()` method. You can think of `groupby()` as the splitting part. You can then apply functions to the resulting `DataFrameGroupBy` object. The [Pandas documentation on split-apply-combine](http://pandas.pydata.org/pandas-docs/stable/groupby.html) is excellent and worth reading through. It is extensive though, so don't let yourself get intimidated by it.

## Aggregation

Let's go ahead and do our first split-apply-combine on this tidy data set. First, we will split the data set up by `BioProject`.


```python
grouped = df.groupby(['BioProject'])

# Take a look
grouped
```

There is not much to see in the `DataFrameGroupBy` object that resulted. But there is a lot we can do with this object. Typing `grouped.` and hitting tab will show you the many possibilities. For most of these possibilities, the apply and combine steps happen together and a new data frame is returned. The `grouped.sum()` method is exactly what we want.


```python
df_sum = grouped.sum()

# Take a look
df_sum
```


```python
df_sum = pd.DataFrame(grouped['size_MB'].sum())
df_sum
```

The outputted data frame has the sums of numerical columns only, which we have only one: `size_MS`. Note that this data frame has `Platform` as the name of the row index.
If we want to instead keep `Platform` (which, remember, is what we used to split up the data set before we computed the summary statistics) as a column, we can use the `reset_index()` method.


```python
df_sum.reset_index()
```

Note, though, that this was not done in-place. If you want to update your data frame, you have to explicitly do so.


```python
df_sum = df_sum.reset_index()
```

We can also use multiple columns in our `groupby()` operation. To do this, we simply pass in a list of columns into `df.groupby()`. We will **chain the methods**, performing a groupby,
applying a median, and then resetting the index of the result, all in one line.


```python
df.groupby(['BioProject', 'Platform']).sum().reset_index()
```

This type of operation is called an **aggregation**. That is, we split the data set up into groups, and then computed a summary statistic for each group, in this case the median.

You now have tremendous power in your hands. When your data are tidy, you can rapidly accelerate the ubiquitous split-apply-combine methods. Importantly, you now have a logical
framework to think about how you slice and dice your data. As a final, simple example, I will show you how to go start to finish after loading the data set into a data frame,
splitting by `BioProject` and `Platform`, and then getting the quartiles and extrema, in addition to the mean and standard deviation.


```python
df.groupby(['BioProject', 'Platform'])['size_MB'].describe()
```


```python
df.groupby(['BioProject', 'Platform'])['size_MB'].describe().reset_index()
```


```python
import numpy as np
df.groupby(['BioProject', 'Platform']).agg({'size_MB':np.mean, 'Run':'nunique'})
```

Yes, that's right. One single, clean, easy-to-read line of code. In the coming tutorials, we will see how to use tidy data to quickly generate plots.

> <tip-title>Quotes versus no quotes?</tip-title>
>
>[Why `np.mean` is without quotes and `nunique` is with quotes?](https://stackoverflow.com/questions/66443260/why-are-some-pandas-aggregation-functions-in-quotes-and-others-not)
{: .tip}

## Tidying a data set

You should always organize your data sets in a tidy format. However, this is sometimes just not possible, since your data sets can come from instruments that do not output the
data in tidy format (though most do, at least in my experience), and you often have collaborators that send data in untidy formats.

The most useful function for tidying data is `pd.melt()`. To demonstrate this we will use a dataset describing read coverage across SARS-CoV-2 genomes for several samples.


```python
df = pd.read_csv('https://zenodo.org/records/10680470/files/coverage.tsv.gz',sep='\t')

df.head()
```

Clearly, these data are not tidy. When we melt the data frame, the data within it, called **values**, become a single column. The headers, called **variables**, also become new columns.
So, to melt it, we need to specify what we want to call the values and what we want to call the variable. [`pd.melt()`](https://pandas.pydata.org/docs/reference/api/pandas.melt.html#pandas.melt) does the rest!

![Dataframe melt](https://pandas.pydata.org/docs/_images/07_melt.svg)

> Image from [Pandas Docs](https://pandas.pydata.org/docs/getting_started/intro_tutorials/07_reshape_table_layout.html#wide-to-long-format).


```python
melted = pd.melt(df, value_name='coverage', var_name=['sample'],value_vars=df.columns[3:],id_vars=['start','end'])

melted.head()
```

...and we are good to do with a tidy DataFrame! Let's take a look at the summary. This would allow us to easily plot coverage:


```python
import seaborn as sns
sns.relplot(data=melted, x='start',y='coverage',kind='line')
```


```python
sns.relplot(data=melted, x='start',y='coverage',kind='line',hue='sample')
```


```python
melted.groupby(['sample']).describe()
```


```python
melted.groupby(['sample'])['coverage'].describe()
```

To get back from melted (narrow) format to wide format we can use [`pivot()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html#pandas.DataFrame.pivot) function.

![Dataframe pivot](https://pandas.pydata.org/docs/_images/07_pivot.svg)

> Image from [Pandas Docs](https://pandas.pydata.org/docs/getting_started/intro_tutorials/07_reshape_table_layout.html#long-to-wide-table-format).


```python
melted.pivot(index=['start','end'],columns='sample',values='coverage')
```

# Working with multiple tables

Working with multiple tables often involves joining them on a common key:

![Left join](https://pandas.pydata.org/docs/_images/08_merge_left.svg)

In fact, this can be done in several different ways described below. But first, let's define two simple dataframes:


```python
!pip install --upgrade pandasql
```

```python
from pandasql import sqldf
pysqldf = lambda q: sqldf(q, globals())
```

```python
df1 = pd.DataFrame({"key": ["A", "B", "C", "D"], "value": np.random.randn(4)})
df2 = pd.DataFrame({"key": ["B", "D", "D", "E"], "value": np.random.randn(4)})
```

```python
df1
```

```python
df2
```

## Inner join

![Inner join](https://upload.wikimedia.org/wikipedia/commons/thumb/1/18/SQL_Join_-_07_A_Inner_Join_B.svg/234px-SQL_Join_-_07_A_Inner_Join_B.svg.png?20170204165143)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key")
```

Using `pysqldf`:


```python
pysqldf('select * from df1 join df2 on df1.key=df2.key')
```

## Left join

![Left join again](https://upload.wikimedia.org/wikipedia/commons/thumb/d/dc/SQL_Join_-_01b_A_Left_Join_B.svg/234px-SQL_Join_-_01b_A_Left_Join_B.svg.png?20170204144906)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key", how="left").fillna('.')
```

Using `pysqldf`:

```python
pysqldf('select * from df1 left join df2 on df1.key=df2.key').fillna('.')
```

```python
pysqldf('select df1.key, df1.value as value_x, df2.value as value_y from df1 left join df2 on df1.key=df2.key').fillna('.')
```

## Right join

![Right join](https://upload.wikimedia.org/wikipedia/commons/thumb/5/5f/SQL_Join_-_03_A_Right_Join_B.svg/234px-SQL_Join_-_03_A_Right_Join_B.svg.png?20170130230641)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key", how="right").fillna('.')
```

## Full join

![Full join](https://upload.wikimedia.org/wikipedia/commons/thumb/6/61/SQL_Join_-_05_A_Full_Join_B.svg/234px-SQL_Join_-_05_A_Full_Join_B.svg.png?20170130230643)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key", how="outer").fillna('.')
```

# Putting it all together: Pandas + Altair

## Understanding [Altair](https://altair-viz.github.io/)

Vega-Altair is a declarative statistical visualization library for Python, based on Vega and Vega-Lite. It offers a powerful and concise grammar that enables you to quickly build a wide range of statistical visualizations. [This site](https://uwdata.github.io/visualization-curriculum/intro.html) contains a complete set of "how-to" tutorials explaining all aspects of this remarkable package.  

## Importing data

First, import all the packages we need:

```py
import pandas as pd
import altair as alt
from datetime import date
today = date.today()
```

Next, read a gigantic dataset from DropBox:

```py
sra = pd.read_csv(
    "https://zenodo.org/records/10680776/files/ena.tsv.gz",
    compression='gzip',
    sep="\t",
    low_memory=False
)
```

This dataset contains *a lot* of rows:

```py
len(sra)
```
Which would output:

```
1000000
```

Let's look at the five random lines from this table (scroll sideways):

```py
sra.sample(5)
```

|         | study_accession   | run_accession   | collection_date     | instrument_platform   | library_strategy   | library_construction_protocol   |
|--------:|:------------------|:----------------|:--------------------|:----------------------|:-------------------|:--------------------------------|
| 2470282 | PRJEB37886        | ERR8618165      | 2022-02-03 00:00:00 | ILLUMINA              | AMPLICON           | nan                             |
| 5747352 | PRJNA686984       | SRR21294506     | 2022-07-10 00:00:00 | OXFORD_NANOPORE       | AMPLICON           | nan                             |
| 2588443 | PRJEB37886        | ERR8997118      | 2022-02-15 00:00:00 | ILLUMINA              | AMPLICON           | nan                             |
| 3662323 | PRJNA716984       | SRR16033134     | 2021-09-08 00:00:00 | PACBIO_SMRT           | AMPLICON           | Freed primers                   |
| 5298351 | PRJNA716984       | SRR20455932     | 2022-04-10 00:00:00 | PACBIO_SMRT           | AMPLICON           | Freed primers                   |

This dataset also has *a lot* of columns:

```py
for _ in sra.columns: print(_)
```

We do not need all the columns, so let's restrict the dataset only to columns we would need. This would also make it much smaller:

```py
sra = sra[
    [ 
        'study_accession',
        'run_accession',
        'collection_date',
        'instrument_platform',
        'library_strategy',
        'library_construction_protocol' 
    ]
]
```

## Cleaning the data

The `collection_date` field will be useful for us to be able to filter out nonsense as you will see below. But to use it properly, we need tell Pandas that it is not just a text, but actually dates:

```py
sra = sra.assign(collection_date = pd.to_datetime(sra["collection_date"]))
```

Let's see what are the earliest and latest entries:

```py
print('Earliest entry:', sra['collection_date'].min())
print('Latest entry:', sra['collection_date'].max())
```

> <warning-title>Metadata is &#128169;</warning-title>
>&#128561; | Don't get surprised here - the metadata is only as good as the person who entered it. So, **when you enter metadata for you sequencing data -- pay attention!!!**
{: .warning}

> <question-title></question-title>
>
>Can you write a statement that would show how many rows contain these erroneous dates?
>
> > <solution-title></solution-title>
> >
> >```py
> > sra[sra['collection_date'] == sra['collection_date'].min()]['run_accession'].nunique()
> > sra[sra['collection_date'] == sra['collection_date'].max()]['run_accession'].nunique()
> > ```
> {: .solution}
{: .question}

The data will likely need a bit of cleaning:

```py
sra = sra[ 
    ( sra['collection_date'] >= '2020-01-01' ) 
    & 
    ( sra['collection_date'] <= '2023-02-16' ) 
]
```

Finally, in order to build the heatmap, we need to aggregate the data:

```py
heatmap_2d = sra.groupby(
    ['instrument_platform','library_strategy']
).agg(
    {'run_accession':'nunique'}
).reset_index()
```

This will look something like this:

|    | instrument_platform   | library_strategy    |   run_accession |
|---:|:----------------------|:--------------------|----------------:|
|  0 | BGISEQ                | AMPLICON            |              21 |
|  1 | BGISEQ                | OTHER               |            1067 |
|  2 | BGISEQ                | RNA-Seq             |              64 |
|  3 | BGISEQ                | Targeted-Capture    |              38 |
|  4 | BGISEQ                | WGA                 |               1 |
|  5 | CAPILLARY             | AMPLICON            |               3 |
|  6 | DNBSEQ                | AMPLICON            |             325 |
|  7 | DNBSEQ                | OTHER               |               5 |
|  8 | DNBSEQ                | RNA-Seq             |              64 |
|  9 | ILLUMINA              | AMPLICON            |         4833110 |
| 10 | ILLUMINA              | OTHER               |             204 |
| 11 | ILLUMINA              | RNA-Seq             |           34826 |
| 12 | ILLUMINA              | Targeted-Capture    |           14843 |
| 13 | ILLUMINA              | WCS                 |              74 |
| 14 | ILLUMINA              | WGA                 |           89499 |
| 15 | ILLUMINA              | WGS                 |           59618 |
| 16 | ILLUMINA              | WXS                 |               2 |
| 17 | ILLUMINA              | miRNA-Seq           |              28 |
| 18 | ION_TORRENT           | AMPLICON            |          108617 |
| 19 | ION_TORRENT           | RNA-Seq             |              75 |
| 20 | ION_TORRENT           | WGA                 |             724 |
| 21 | ION_TORRENT           | WGS                 |            1385 |
| 22 | OXFORD_NANOPORE       | AMPLICON            |          445491 |
| 23 | OXFORD_NANOPORE       | OTHER               |              31 |
| 24 | OXFORD_NANOPORE       | RNA-Seq             |           18617 |
| 25 | OXFORD_NANOPORE       | WGA                 |            7842 |
| 26 | OXFORD_NANOPORE       | WGS                 |           13670 |
| 27 | PACBIO_SMRT           | AMPLICON            |          533112 |
| 28 | PACBIO_SMRT           | FL-cDNA             |              11 |
| 29 | PACBIO_SMRT           | RNA-Seq             |            1013 |
| 30 | PACBIO_SMRT           | Synthetic-Long-Read |              29 |
| 31 | PACBIO_SMRT           | Targeted-Capture    |             465 |
| 32 | PACBIO_SMRT           | WGS                 |              48 |


## Plotting the data

Now let's create a graph. This graph will be layered: the "back" will be the heatmap squares and the "front" will be the numbers (see heatmap at the beginning of this page):

```py
back = alt.Chart(heatmap_2d).mark_rect(opacity=1).encode(
    x=alt.X(
        "instrument_platform:N",
        title="Instrument"
    ),
    y=alt.Y(
        "library_strategy:N",
        title="Strategy",
        axis=alt.Axis(orient='right')
    ),
    color=alt.Color(
        "run_accession:Q",
        title="# Samples",
        scale=alt.Scale(
            scheme="goldred",
            type="log"
        ),
    ),
    tooltip=[
        alt.Tooltip(
            "instrument_platform:N",
            title="Machine"
        ),
        alt.Tooltip(
            "run_accession:Q",
            title="Number of runs"
        ),
        alt.Tooltip(
            "library_strategy:N",
            title="Protocol"
        )
    ]
).properties(
    width=500,
    height=150,
    title={
      "text": 
        ["Breakdown of datasets (unique accessions) from ENA",
         "by Platform and Library Strategy"],
      "subtitle":"(Updated {})".format(today.strftime("%B %d, %Y"))
    }
)

back
```

This would give us a grid:

![Grid](https://i.imgur.com/74itE5M.png)

Now, it would be nice to fill the rectangles with actual numbers:

```py
front = back.mark_text(
    align="center",
    baseline="middle",
    fontSize=12,
    fontWeight="bold",
).encode(
    text=alt.Text("run_accession:Q",format=",.0f"),
    color=alt.condition(
        alt.datum.run_accession > 200,
        alt.value("white"),
        alt.value("black")
    )
)

front
```

This would give us the text:

![Text for grid](https://i.imgur.com/pKalDaC.png)

To superimpose these on top of each other we should simply do this:

```py
back + front
```

![Full graph](https://i.imgur.com/SzhM1QW.png)


topic: data-science
tutorial: gnmx-lecture1
title: A (very) brief history of genomics
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: a8c5379a685be8aaf6d0a68fcefa0c94
last_modified: 2026-02-26T21:12:14.873662
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture1/tutorial.html

content:
![Fluctuation test](./images/luria_small.png)

# What do you know?

This is the first lecture of the class. We will begin with a short quiz that is designed tio let me know what you do and do not know. This quiz will have no effect on your grade in this course. 

# Why history?

Knowing history is essential for understanding how we arrived to the current state of affairs in our field. It is also full of accidental discoveries and dramatic relationships making it quite interesting to read about. I strongly advise you to take a look at the manuscripts below:

## Classical publications

* 1965 : [A history of genetics](http://www.amazon.com/A-History-Genetics-A-H-Sturtevant/dp/0879696079)
* 1943 : [Delbruck & Luria](http://www.bx.psu.edu/~anton/bioinf1-2014/delbruck-luria-1943.pdf)
* 1944 : [Avery, MacLeod, & McCarty](http://www.bx.psu.edu/~anton/bioinf1-2014/avery-1944.pdf)
* 1952 : [Herhey & Chase](http://www.bx.psu.edu/~anton/bioinf1-2014/hershey-chase-1952.pdf)
* 1953 : [Watson & Crick](http://www.bx.psu.edu/~anton/bioinf1-2014/watsoncrick.pdf)
* 1958 : [Meselson & Stahl](http://www.bx.psu.edu/~anton/bioinf1-2014/Proc%20Natl%20Acad%20Sci%20USA%201958%20Meselson.pdf)
* 1960 : [Jacob and Monod](http://www.bx.psu.edu/~anton/bioinf1-2014/jacob-monod-1961.pdf)

## Popular (yet very informative) literature

![Books](./images/books.png)

------

# Let's get to it

Today's lecture will be organized as a, well, a lecture - we will be going over a set of slides linked at the beginning of this page. 

-----


topic: data-science
tutorial: online-resources-protein
title: One protein along the UniProt page
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["online-resources-gene"]
tags_json: []
content_hash: c2c62509dfe2eeeef13e1824c7cfe950
last_modified: 2026-02-26T21:12:14.898827
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/online-resources-protein/tutorial.html

content:
When doing a biological data analysis, we might end up with some interesting proteins, that we need explore these genes. But how can we do that? What are the resources available for that? And how to navigate through them?

The aim of this tutorial is to familiarize ourselves with that, using Human opsins as an example.

> <comment-title></comment-title>
> This tutorial is a bit atypical: we will not work in Galaxy but mostly outside of it, in the [UniProt](https://uniprot.org) database pages.
{: .comment}

> <comment-title></comment-title>
> This tutorial  designed to be the continuation of the tutorial ["One gene across file formats"]({% link topics/data-science/tutorials/online-resources-gene/tutorial.md %}), but it can also be consulted as a stand-alone module. 
{: .comment}

Opsins are found in the cells of your retina. They catch light and begin the sequence of signals that result in vision, and that is the reason why, when compromised, they are associated to color-blindness and other visual impairments. 

> <comment-title>Sources of information from this tutorial</comment-title>
>
> The tutorial you are consulting was mainly developed consulting UniProtKB resources, in particular the [Explore UniProtKB entry](https://www.uniprot.org/help/explore_uniprotkb_entry) tutorial. 
> Some sentences are reported from there with no modifications.
> 
> In addition, the topic was chosen based Gale Rhodes' [Bioinformatics Tutorial](https://spdbv.unil.ch/TheMolecularLevel/Matics/index.html). Although the tutorial cannot be anymore be followed step by step due to how the mentioned resources changed in time, it could provide additional insights on opsins and in particular how one could build structural models of proteins based on evolutionary information. 
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
{: .agenda}

# The UniProtKB Entry page

The portal to visit to obtain all information about a protein is [UniProtKB](https://www.uniprot.org/). We can search it using a text search, or the gene or protein name. Let's try first a set of generic keywords, like `Human opsin`. 

> <hands-on-title>Search for Human opsin on UniProtKB</hands-on-title>
>
> 1. Open the [UniProtKB](https://www.uniprot.org/)
> 2. Type `Human opsin` in the search bar
> 3. Launch the search
>
{: .hands-on}

> <question-title></question-title>
>
> How many results did we get?
>
> > <solution-title></solution-title>
> >
> > 410 results (at the time of preparation of this tutorial)
> >
> {: .solution}
{: .question}

These 410 results give us the feeling that we need to be more specific (although - spoiler - our actual target is among the first hits).

To be specific enough, we suggest to use an unique identifier. From the [previous tutorial]({% link topics/data-science/tutorials/online-resources-gene/tutorial.md %}) we know the gene name of the protein that we are looking for, `OPN1LW`.

> <hands-on-title>Search for OPN1LW on UniProtKB</hands-on-title>
>
> 1. Type `OPN1LW` in the top search bar
> 2. Launch the search
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many results did we get?
> 2. What should we do to reduce this number?
>
> > <solution-title></solution-title>
> >
> > 1. 200+ results (at the time of preparation of this tutorial)
> > 2. We need to clarify what we are looking for: Human OPN1LW
> {: .solution}
{: .question}

We need to add `Human` to clarify what we are looking for.

> <hands-on-title>Search for Human OPN1LW on UniProtKB</hands-on-title>
>
> 1. Type `Human OPN1LW` in the top search bar
> 2. Launch the search
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many results did we get?
> 2. Do we have one result including OPN1LW as gene name?
>
> > <solution-title></solution-title>
> >
> > 1. 7 results (at the time of preparation of this tutorial)
> > 2. The first result is labeled with `Gene: OPN1LW (RCP)`
> {: .solution}
{: .question}

The first result, labeled with `Gene: OPN1LW (RCP)`, is our target, `P04000 · OPSR_HUMAN`. Before opening the page, two things to notice: 

1. The name of the protein `OPSR_HUMAN` is different than the gene name, as well as their IDs are. 
2. This entry has a golden star, which means that was manually annotated and curated. 

# Inspecting a UniProt entry

> <hands-on-title>Open a result on UniProt</hands-on-title>
>
> 1. Click on `P04000 · OPSR_HUMAN`
{: .hands-on}

![Screenshot of the UniProt entry page header](./images/UniProt.png "UniProt page")

To navigate this long page, the menu (navigation bar) on the left will be extremely useful. Just from it, we understand that this database contains information about the entry on: 
- the known functions, 
- the taxonomy, 
- the location,
- variants and associated diseases, 
- Post Translational Modification (PTMs), 
- the expression, 
- the interactions, 
- the structure, 
- the domains and their classification, 
- the sequences
- similar proteins. 

The navigation bar stays in the same place on the screen as you move up and down in an entry, so that you can quickly navigate to sections of interest. We will consult all the mentioned sections separately, but let's first focus on the headers on the left. 

At the top of the page, you can see the UniProt entry accession and name, the name of the protein and gene, the organism, whether the protein entry has been manually reviewed by a UniProt curator, its annotation score, and the evidence level for its existence.

Below the main header, you find a series of tabs (*Entry*, *Variant viewer*, *Feature viewer*, *Publications*, *External links*, *History*). The tabs allow you to switch between the entry, a graphical view of sequence features (Feature viewer), publications, and external links, but ignore them for the moment and do not move from the *Entry* tab. 

## Entry

The next menu is already part of the *Entry* tab. It allows us to run a BLAST sequence similarity search on the entry, align it with all its isoforms, download the entry in various formats, or add it to the basket to save for later.

> <question-title></question-title>
>
> 1. What are the available formats in the *Download* drop-down menu? 
> 2. What type of information would we download through these file formats?
>
> > <solution-title></solution-title>
> > 
> > 1. The formats are: `Text`, `FASTA (canonical)`, `FASTA (canonical & isoform`, `JSON`, `XML`, `RDF/XML`, `GFF`
> > 2. The`FASTA` formats should sound familiar (after the preliminary tutorial), and include the protein sequence, eventually with its isoforms (in which case it will be a multi-FASTA). Besides these, all the other formats are not protein- or even biology-specific. These are general file formats extensively used by websites to include the information included in the page. Hence, by downloading `text` (or even better the `json`) file, we would download the same annotation that we access in this page, but in a format that is easier to parse programmatically.  
> >
> {: .solution}
{: .question}

Let's now scroll the entry page, section by section. 

### Function

This section summarises the functions of this protein as follows: 

Visual pigments are the light-absorbing molecules that mediate vision. They consist of an apoprotein, opsin, covalently linked to cis-retinal.

Regardless of the level of details that you understand (depending on your background), this is impressively short and specific considering the enourmous amount of literature and studies that exist beyond the determination of a protein function. Anyway, someone did the work for us, and this protein is already fully classified in the Gene Ontology (GO), which describes any classified protein's molecular function, biological process and cellular component. 

GO is a perfect example of database / resource that builds on a very complex universe of knowledge and translates it into a simpler graph, at the risk of losing  details. This has the great advantage of organising the information, making it countable and analysable and programmatically accessible, ultimately allowing us to have this long summary pages and KnowledgeBases. 

> <question-title></question-title>
>
> 1. To which molecular functions is this protein annotated?
> 2. To which cellular components is this protein annotated?
> 3. To which biological processes this protein annotated?
>
> > <solution-title></solution-title>
> > 
> > 1. Photoreceptor protein, G-protein coupled receptor
> > 2. Photoreceptor disc membrane
> > 2. Sensory transduction, Vision 
> >
> {: .solution}
{: .question}

### Names and Taxonomy

Other examples of structured information are available in the next section, e.g. in the taxonomy. This section also reports other unique identifiers that refer to the same biological entity or to linked entities (e.g. associated diseases in the `MIM` menu).

> <question-title></question-title>
>
> 1. What is the taxonomic identifier associated with this protein?
> 2. What is the proteome identifier associated with this protein?
>
> > <solution-title></solution-title>
> > 
> > 1. 9606, i.e. Homo sapiens
> > 2. UP000005640, componenent of Chromosome Xs
> {: .solution}
{: .question}

## Subcellular location

We already know where our protein is in the human body (in the retina, as specified by the function summary), but where is it in the cell? 

> <question-title></question-title>
>
> 1. Where is our protein in the cell?
> 2. Is it consistent with the GO annotation observed before?
>
> > <solution-title></solution-title>
> > 
> > 1. The section explains that it is a "Multi-pass membrane protein", which means that it is a protein inserted in the cell membrane and goin through it multiple times.
> > 2. The GO Annotation on the top mentions that we are referring to the photoreceptor (cell) membrane in particular. 
> {: .solution}
{: .question}

The Subcellular location section includes a *Features* area detailing which sections, along the protein sequence, are inserted in the membrane (Transmembrane) and which not (Topological domain). 

> <question-title></question-title>
>
> How many Transmembrane domains and Topological domains are there?
>
> > <solution-title></solution-title>
> > 
> > 8 Transmembrane and 7 Topological domains
> {: .solution}
{: .question}

### Disease & Variants

As we know from the previous tutorial, this gene/protein is associated to multiple diseases. This section details this association also listing the specific variants that have been detected as disease-related. 

> <question-title></question-title>
>
> What types of scientific studies allow to assess the association of a genetic variant to diseases?
>
> > <solution-title></solution-title>
> > 
> > Three commonly used methods for assessing the association of a genetic variant with a disease are:
> >
> > - Genome-Wide Association Studies (GWAS)
> >
> >   GWAS are widely used to identify common genetic variants associated with diseases. They involve scanning the entire genome of a large number of individuals to identify variations linked to a particular disease or trait.
> >
> > - Case-Control Studies
> >
> >   Case-control studies are frequently employed to compare individuals with a disease to those without it, focusing on the presence or frequency of specific genetic variants in both groups.
> >
> > - Family Studies
> >
> >   Family-based studies involve analyzing genetic variants within families where multiple members are affected by a disease. By studying the inheritance patterns of genetic variants and their association with the disease within families, researchers can identify potential disease-associated genes.
> >
> >
> > These type of studies would imply extensive usage of the file types to manage genomic data, such as: SAM (Sequence Alignment Map), BAM (Binary Alignment Map), VCF (Variant Calling Format) etc.  
> >
> {: .solution}
{: .question}

This section also includes a *Features* area, where the natural variants are mapped along the sequence. Below, it also highlights that a more detailed view of features along the sequence is provided in the *Disease & Variants* tab, but let's not open it for now. 

### PTM/Processing

A post-translational modification (PTM) is a covalent processing event resulting from a proteolytic cleavage or from the addition of a modifying group to one amino acid. 

> <question-title></question-title>
>
> What are the post-translational modifications for our protein?
>
> > <solution-title></solution-title>
> > 
> > Chain, glycosylation, disulfide bond, modified residue
> {: .solution}
{: .question}

### Expression

We already know where the protein is in the cell, but for human proteins we often have information on where it is in the human body, i.e. in which tissues. This information can come from the Human [ExpressionAtlas](https://www.ebi.ac.uk/gxa/home) or other similar resources. 

> <question-title></question-title>
>
> In which tissue is the protein found?
>
> > <solution-title></solution-title>
> > 
> > The three color pigments are found in the cone photoreceptor cells.
> {: .solution}
{: .question}

### Interaction

Proteins perform their function through their interaction with the surroundings, in particular with other proteins. This section reports the interactors of our protein of interest, in a table that we can also filter by subcellular location, diseases, and type of interaction. 

The source of this information are databases like STRING, and the entry page for our protein is directly linked from this section. 

> <hands-on-title>Search for Human OPN1LW on UniProtKB</hands-on-title>
>
> 1. Click on the [STRING link](https://string-db.org/network/9606.ENSP00000358967) in a different tab 
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many different file formats can you download from there?
> 2. What kind of information will be conveyed in each file?
>
> > <solution-title></solution-title>
> > 
> >  STRING provides data in downloadable file formats to support further analyses. 
> >  The primary file format used by STRING is the 'TSV' (Tab-Separated Values) format, which presents protein interaction data in a structured, tabular layout. This format is well-suited for easy integration into various data analysis tools and software. 
> >  Additionally, STRING offers data in PSI-MI (Proteomics Standards Initiative Molecular Interactions) XML format, a standard for representing protein interaction data that allows for compatibility with other interaction databases and analysis platforms. These file formats enable researchers to harness the wealth of protein interaction information in STRING for their own studies and analyses. Researchers can also download visual representations of protein networks in image formats such as PNG and SVG, which are suitable for presentations and publications. For advanced analysis, STRING offers 'flat files' containing detailed interaction information, and 'MFA' (Multiple Alignment Format) files, which are useful for comparing multiple protein sequences. These diverse downloadable file formats enable researchers to harness the wealth of protein interaction information in STRING for their own studies and analyses.
> >
> {: .solution}
{: .question}

### Structure

Are you curious about the intricate three-dimensional structures of proteins? The *Structure* section on the UniProtKB entry page is your gateway to exploring the fascinating world of protein architecture. 

In this section, you'll find information about experimentally determined protein structures. These structures provide crucial insights into how proteins function and interact with other molecules. You will discover interactive views of the protein's structure that you can explore directly within the UniProtKB entry. This feature provides an engaging way to navigate through the protein's domains, binding sites, and other functional regions. By delving into the *Structure* section, you'll gain a deeper understanding of the physical basis of protein function and discover the wealth of information that structural data can unlock.

> <question-title></question-title>
>
> 1. What is the variant associated to color blindness?
> 2. Can you find that specific amino acid in the structure? 
> 3. Can you formulate a guess of why this mutation is disruptive?
>
> > <solution-title></solution-title>
> >
> > 1. In the *Disease & Variants* section, we discover that the change from Glycine (G) to Glutamic acid (E) at position 338 along the protein sequence is associated to Colorblindness. 
> > 2. In the structure viewer, we can move the molecule around and hover the mouse on the structure to find the AA in position 338. It might take some time to follow through the multiple helical arrangements of these structures. The Glycine at 338 is not in a helix, but in what looks like a loop just before a low confidence area in the structure. 
> > 3. Based on the information we collected so far, we could make a hypothesis of why this is distruptive. It is not in a helix (usually, in transmembrane proteins, helices are inserted in the membrane), hence, it is in one of the bigger domains that protrude from the membrane, in or out of the cell. This mutation probably doesn't distrupt the structure in its intra-membrane segments, but rather one of the functional domain. If you want to dig deeper, you can check if this is the extra- or intra-cellular segment in the **Feature viewer**.
> >
> {: .solution}
{: .question}

Where does the information in the structure viewer come from?

> <hands-on-title>Search for Human OPN1LW on UniProtKB</hands-on-title>
>
> 1. Click on the download icon below the structure
> 2. Check the file that has been downloaded
>
{: .hands-on}

This is a PDB (Protein Data Bank) file, allowing you to visualize and analyze the protein's arrangement of atoms and amino acids. 

However, there is no reference to the PDB database in the links among the *3D structure databases*. Instead, the first link refers to the AlphaFoldDB. The AlphaFold Database is a comprehensive resource that provides predicted 3D structures for a wide range of proteins. Using deep learning techniques and evolutionary information, AlphaFold accurately predicts the spatial arrangement of atoms within a protein, contributing to our understanding of protein function and interactions. 

Hence, this is a *prediction* of the structure, not a structure experimentally validated. This is the reason why it is colored by confidence: the sections in blue are those with a high confidence value, so the ones for which the prediction is very reliable, while the ones in orange are less reilable or have a disordered (more flexible and mobile) structure. Nevertheless, this information is represented through a PDB file, because it is still structural. 

### Family and Domains

The *Family and Domains* section on the UniProtKB entry page provides a comprehensive view of the evolutionary relationships and functional domains within a protein. This section offers insights into the protein's membership in protein families, superfamilies, and domains, shedding light on its structural and functional characteristics. 

The *Features* area confirms indeed that at least one of the two domains protruding from the membrane (the N-terminal one) is disordered. This area usually includes information about conserved regions, motifs, and important sequence features that contribute to the protein's role in various biological processes. The section confirms once again that we are looking at a transmembrane protein, and links to several resources of phylogenetic, protein families or domains data - guiding us on understading how proteins share common ancestry, evolve, and acquire specialized functions.

### Sequence

All this information about the protein's evolution, function, structure, is ultimately encoded in its sequence. Once again, in this section we have the opportunity to download the FASTA file transcribing it, as well as to access the source of this data: the genomic sequencing experiments that assessed it. This section also reports when isoforms have been detected. 

> <question-title></question-title>
>
> How many potential isoforms are mapped to this entry?
>
> > <solution-title></solution-title>
> >
> > 1: H0Y622
> >
> {: .solution}
{: .question}

### Similar proteins

The last section of the UniProt Entry page reports similar proteins (this is basically the result of a clustering, with 100%, 90% and 50% identity thresholds).

> <question-title></question-title>
>
> 1. How many similar proteins at 100% identity?
> 2. How many similar proteins at 90% identity?
> 3. How many similar proteins at 50% identity?
>
> > <solution-title></solution-title>
> >
> > 1. 0
> > 2. 83
> > 3. 397
> >
> {: .solution}
{: .question}

As you might have guessed while looking at this page, a lot of the processing of biological data about a protein actually consists of mapping different types of information along the sequence, and understand how they influence each other. A visual mapping (and a table with the same information) is provided by the two alternative tabs to view this entry, that is the *Variant viewer* and the *Feature viewer*. 

## Variant viewer

> <hands-on-title>Variant viewer</hands-on-title>
>
> 1. Click on *Variant viewer* tab
>
{: .hands-on}

The *Variant viewer* maps all the known alternative versions of this sequence. For some of them the effect (pathogenic or benign) is known, for others not. 

> <question-title></question-title>
>
> How many variant are likely pathogenic?
>
> > <solution-title></solution-title>
> >
> > By zooming out in the variant view, we see that we have 5 red points, so 5 variants that are likely pathogenic.
> >
> {: .solution}
{: .question}

The high number of variants that you find in this section suggests that "protein sequences" (as well as gene sequences, protein structures etc.) are actually less fixed entities than what we would think.

## Feature viewer

> <hands-on-title>Feature viewer</hands-on-title>
>
> 1. Click on *Feature viewer* tab
>
{: .hands-on}

The *Feature viewer* is basically a merged version of all the *Features* areas that we found in the *Entry* page, including *Domains & sites*, *Molecule processing*, *PTMs*, *Topology*, *Proteomics*, *Variants*. If in the viewer you click on any feature, the correspondent region in the structure will be focused, as the variant of interest

> <hands-on-title>Variant viewer</hands-on-title>
>
> 1. Expand the *Variants* part
> 2. Zoom out
> 3. Click on our variant of interest (the red point at position 338)
>
{: .hands-on}

> <question-title></question-title>
>
> What is the topology at this location?
>
> > <solution-title></solution-title>
> >
> > A topological cytoplasmic domain
> >
> {: .solution}
{: .question}

Finally, let's have a quick look at the other tabs. 

## Publications

> <hands-on-title>Publication</hands-on-title>
>
> 1. Click on *Publication* tab
>
{: .hands-on}

The *Publications* one lists scientific publications related to the protein. These are collected by merging a fully curated list in UniProtKB/Swiss-Prot and automatically imported ones. In this tab, you can filter the publications list by source and categories that are based on the type of data a publication contains about the protein (such as function, interaction, sequence, etc.), or by the number of proteins in the corresponding study it describes ("small scale" vs "large scale").

> <question-title></question-title>
>
> 1. How many publications are associated with this protein?
> 2. How many publications contains information about its function?
>
> > <solution-title></solution-title>
> >
> > 1. 57
> > 2. 23
> >
> {: .solution}
{: .question}

## External links

> <hands-on-title>External links</hands-on-title>
>
> 1. Click on *External links* tab
>
{: .hands-on}

The *External links* tab puts together all the references to external databases and resources of information that we found in each Entry page section. The links text often reports the unique identifiers that represent the same biological entity in other databases. To get a feeling of this compexity, check the following image (that is already partially outdated).

![A graph representing how all the different databases are connected by unique IDs, where major nodes are DBs, and arrows connect them to the IDs (minor nodes) that they report. The map is very crowded, especially around the UniProt Entry Name, Gene ID and Ensembl Gene ID](./images/complexDB.jpeg "Best illustration of the complex ID crossref: bioDBnet Network Diagram - [source](https://biodbnet-abcc.ncifcrf.gov/dbInfo/netGraph.php)")

## History

Finally, the *History* tab is also an interesting one. It reports and makes available for download all the previous versions of this entry annotations, that is: all the "evolution" of its annotation, in this case dating back to 1988. 

> <question-title></question-title>
>
> Was this entry ever not manually annotated?
>
> > <solution-title></solution-title>
> >
> > To answer this question you can scroll back in time through the table and check the `Database` column. Was this ever in TrEMBL instead that in SwissProt? No, so this entry was manually annotated since its beginning.
> >
> {: .solution}
{: .question}


topic: data-science
tutorial: python-conda
title: Conda Environments For Software Development
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: ["conda"]
content_hash: eff554d57b37ccfa6b5688d8a90e7421
last_modified: 2026-02-26T21:12:14.901010
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-conda/tutorial.html

content:
Conda environments, like Python Virtual Environments allow you to easily manage your installed packages and prevent conflicts between different project's dependencies. This tutorial follows an identical structure to the virtualenv tutorial, but with conda.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}


If you have a python project you are using, you will often see something like
following two lines somewhere at the top.

```python
from matplotlib import pyplot as plt
import numpy as np
```

This means that our code requires two *external libraries* (also called third-party packages or dependencies) -
`numpy` and `matplotlib`.

Python applications often use external libraries that don’t come as part of the standard Python distribution. This means
that you will have to use a *package manager* tool to install them on your system.

Applications will also sometimes need a
specific version of an external library (e.g. because they require that a particular
bug has been fixed in a newer version of the library), or a specific version of Python interpreter.
This means that each Python application you work with may require a different setup and a set of dependencies so it
is important to be able to keep these configurations separate to avoid confusion between projects.
The solution for this problem is to create a self-contained 
*environment* per project, which contains a particular version of Python installation plus a number of
additional external libraries.

If you see something like 

```python
import pysam
```

You know you'll need additional packages installed on your system, as it relies on [htslib, a C library for working with HTS data](https://github.com/samtools/htslib). This usually means installing additional packages and things that are not always available from within Python's packaging ecosystem.

Conda environments go beyond virtual environments, and make it easier to develop, run, test and share code with others. In this tutorial, we learn how
to set up an environment to develop our code and manage our external dependencies.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Conda Environments

So what exactly are conda environments, and why use them?

A conda environment is an **isolated working copy** of specific versions of
one of more packages and all of their dependencies.

This is in fact simply a *directory with a particular structure* which includes
links to and enables multiple side-by-side installations of different packages
or different versions of the same external library to coexist on your machine
and only one to be selected for each of our projects. This allows you to work on
a particular project without worrying about affecting other projects on your
machine.

As more external libraries are added to your project over time, you can add them to
its specific environment and avoid a great deal of confusion by having separate (smaller) environments
for each project rather than one huge global environment with potential package version clashes. Another big motivator
for using environments is that they make sharing your code with others much easier (as we will see shortly).
Here are some typical scenarios where the usage of environments is highly recommended (almost unavoidable):

- You have two dependencies with conflicting dependencies! You cannot install
  the specific version of software X alongside software Y, as they both depend
  on different versions of a dependency, that cannot co-exist. This is solved by
  having different environments for both.
- You have an older project that only works under Python 2. You do not have the time to migrate the project to Python 3
  or it may not even be possible as some of the third party dependencies are not available under Python 3. You have to
  start another project under Python 3. The best way to do this on a single machine is to set up two separate Python
  environments.
- One of your Python 3 projects is locked to use a particular older version of a third party dependency. You cannot use the
  latest version of the
  dependency as it breaks things in your project. In a separate branch of your project, you want to try and fix problems
  introduced by the new version of the dependency without affecting the working version of your project. You need to set up
  a separate conda environments for your branch to 'isolate' your code while testing the new feature.

You do not have to worry too much about specific versions of external libraries that your project depends on most of the time.
Conda environments enable you to always use the latest available version without specifying it explicitly.
They also enable you to use a specific older version of a package for your project, should you need to.

> <tip-title>A Specific Package Version is Only Ever Installed Once</tip-title>
> Note that you will not have a separate package installations for each of your projects - they will only
> ever be installed once on your system (in `$CONDA/pkgs`) but will be referenced from different environments.
{: .tip}

### Managing Conda Environments

There are several commonly used command line tools for managing environments:
- `homebrew`, historically used on OSX to manage packages.
- `nix`, which has a steep learning curve but allows you to declare the state of your entire system
- `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community)
- `docker` and `singularity` are somewhat similar to other environment managers, as they can have isolated images with software and dependencies.
- Other, language specific managers

While there are pros and cons for using each of the above, all will do the job of managing
environments for you and it may be a matter of personal preference which one you go for. The Galaxy project is heavily invested in the Conda ecosystem and recommends it as an entry point as it is the most generally useful, and convenient. [The BioConda ecosystem](http://bioconda.github.io/) provides an unbelievably large number of packages for bioinformatics specific purposes, which makes it a good choice in general.

### Managing Packages

Part of managing your (virtual) working environment involves installing, updating and removing external packages
on your system. The Conda command (`conda`) is most commonly used for this - it interacts
 and obtains the packages from one or more Conda repositories (e.g. Conda Forge, BioConda, etc.)

> <tip-title>A Note on Anaconda and `conda`</tip-title>
> Anaconda is an open source Python
> distribution commonly used for scientific programming - it conveniently installs Python, package and environment management `conda`, and a 
> number of commonly used scientific computing packages so you do not have to obtain them separately. 
> `conda` is an independent command line tool (available separately from the Anaconda distribution too) with dual functionality: (1) it is a package manager that helps you find Python packages from
> remote package repositories and install them on your system, and (2) it is also a virtual environment manager. So, you can use `conda` for both tasks instead of using `venv` and `pip`.
{: .tip}

`venv` and `pip` are considered the *de facto* standards for environment and package management for Python 3.
However, the advantages of using Anaconda and `conda` are that you get (most of the) packages needed for
scientific code development included with the distribution. If you are only collaborating with others who are also using
Anaconda, you may find that `conda` satisfies all your needs. 

It is good, however, to be aware of all these tools (`pip`, `venv`, `pyenv`, etc.),
and use them accordingly. As you become more familiar with them you will realise that equivalent tools work in a similar
way even though the command syntax may be different (and that there are equivalent tools for other programming languages
too to which your knowledge can be ported).

![Python environment hell XKCD comic  showing boxes like pip, easy_install, homebrew 2.7, anaconda, homebrew 3.6, /usr/local/Cellar, ~/python/, and a chaotic mess of arrows moving between them all. At the bottom is the text: My python environment has become so degraded that my laptop has been declared a superfund site. (A superfund site is generally an environmental disaster area.)](../../images/xkcd/python_environment.png "Python Environment Hell from XKCD 1987 (CC-BY-NC 2.5)")

Let us have a look at how we can create and manage environments and their packages from the command line using `conda`.

### Instaling Miniconda

We will use Miniconda, a minimal conda installer that is commonly used, in place of the larger and slower to download full anaconda distribution.

> <hands-on-title>Installing Conda via Miniconda</hands-on-title>
> 1. Go to the [Miniconda installation page](https://docs.conda.io/en/latest/miniconda.html) and find the appropriate installer for your system.
> 2. Download and run the script.
> 3. You will probably need to close, and restart your terminal.
> 4. Check that you can run the `conda` command, otherwise something may have gone wrong.
{: .hands_on}

*If* you're running on Linux *and* following this tutorial via a Jupyter/CoCalc notebook, and you agree to the [Anaconda terms of service](https://legal.anaconda.com/policies/en/?name=terms-of-service), you can simply run the following cell:

```bash
wget -c https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b 
```

Now you can add it to your `~/.bashrc` which will cause Conda to be automatically loaded whenever you open a terminal:

```
~/miniconda3/bin/conda init bash
```

> <tip-title>Reopen Shell / Restart Jupyter Kernel</tip-title>
> Here you will need to restart your kernel, or if you're in a desktop environment, restart your terminal.
{: .tip}

### Installing our First Package

Let's install our first package, the new `libmamba` solver for Conda, as an example of how to install a package. A side benefit is that it will significant speed up your package installations!

```bash
conda install -y -q conda-libmamba-solver=22.8
```

> <tip-title>Does it get stuck?</tip-title>
> This step we have sometimes seen get "stuck", it will finish executing the transaction and hang, despite successfully installing the software. You can restart the kernel if this happens.
{: .tip}

Here we see a few things:

- `-y` - installs without asking questions like "do you want to do this". Generally people don't use this, but in a Notebook environment it's a bit nicer.
- `-q` - quiet installation, by default it prints a *lot* of progress update messages.
- `conda-libmamba-solver=22.8`, the package and version of that package that we wish to install.


We'll now configure conda to use `mamba` by default:

```bash
conda config --set experimental_solver libmamba
```

While we're at it, let's configure Conda to use the same default repositories as Galaxy:

```bash
conda config --add channels bioconda
conda config --add channels conda-forge
```

This will give us access to the vast repositories of BioConda (bioinformatics software) and Conda Forge (languages and libraries).

### Creating a new Environment
Creating a new environment is done by executing the following command:

```bash
conda create -y -n my-env
```

where `my-env` is any arbitrary name for this Conda environment. Environment names are global, so pick something meaningful when you create one!

For our project, let's create an environment called `hts`

```bash
conda create -y -n hts
```

You can list all of the created environments with

```bash
conda env list 
```

You'll notice that there is a `base` environment created by default, where you can install packages and play around with Conda. We do not recommend installing things into the `base` environment, if at all possible. Create a new environment for each tool you need to install

> <tip-title>Why separate environments?</tip-title>
> Conda's package resolution takes into account every other package installed in an environment. Especially if you use R packages, this can result in environments taking an inreasing amount to time to install new packages and resolve all of the dependencies.
>
> Thus by using isolated environments, you can be sure package resolution is quite fast.
{: .tip}

Once you’ve created an environment, you will need to activate it:

```bash
conda activate hts
```

Activating the environment will change your command line’s prompt to show what environment
you are currently using (indicated by its name in round brackets at the start of the prompt),
and modify the environment so that any packages you install will be available on the CLI.

When you’re done working on your project, you can exit the environment with:

```bash
conda deactivate
```

If you've just done the `deactivate`, ensure you reactivate the environment ready for the next part:

```bash
conda activate hts
```

### Installing External Libraries in an Environment

We noticed earlier that our code depends on two *external libraries* - `numpy` and `matplotlib` as well as `pysam` which depends on `htslib`. In order for the code to run on your machine, you need to
install these dependencies into your environment.

To install the latest version of a package with `conda` you use conda's `install` command and specify the package’s name, e.g.:

```bash
conda install -y -q python=3 numpy matplotlib pysam
```

Note that we needed to pick a version of python that we'd use, here we specify  `python=3` meaning "any Python version that starts with 3", so it won't use Python 2.7 or a future Python 4.

If you run the `conda install` command on a package that is already installed, `conda` will notice this and do nothing.

To install a specific version of a package give the package name followed by `=` and the version number, e.g.
`conda install numpy=1.21.1`.

To specify a minimum version of a Python package, you can
do `pip3 install 'numpy>=1.20'`.

To upgrade a package to the latest version, e.g. `conda update numpy`. (If it's at the latest version it will attempt to downgrade the package)

To display information about the current environment:

```bash
conda info
```

To display information about a particular package installed in your current environment:

```bash
conda list python
```

To list all packages installed with `pip` (in your current environment):

```bash
conda list
```

To uninstall a package installed in the environment do: `conda remove package-name`.
You can also supply a list of packages to uninstall at the same time.

### Exporting/Importing an Environment with `conda`

You are collaborating on a project with a team so, naturally, you will want to share your environment with your
collaborators so they can easily 'clone' your software project with all of its dependencies and everyone
can replicate equivalent environments on their machines. `conda` has a handy way of exporting,
saving and sharing environments.

To export your active environment - use `conda env export` command to
produce a list of packages installed in the environment.
A common convention is to put this list in a `environment.yml` file:

```bash
conda env export > environment.yml
cat environment.yml
```

The first of the above commands will create a `environment.yml` file in your current directory.
The `environment.yml` file can then be committed to a version control system and
get shipped as part of your software and shared with collaborators and/or users. They can then replicate your environment and
install all the necessary packages from the project root as follows:

```bash
conda env create -y -f environment.yml
```

The name is bundled directly into the environment, someone else re-creating
this environment from the yaml file will also be able to `conda activate hts`
afterwards. If you want it under a different name, you can use the `-n` flag to
supply your own name.

As your project grows - you may need to update your environment for a variety of reasons. For example, one of your project's dependencies has
just released a new version (dependency version number update), you need an additional package for data analysis
(adding a new dependency) or you have found a better package and no longer need the older package (adding a new and
removing an old dependency). What you need to do in this case (apart from installing the new and removing the
packages that are no longer needed from your environment) is update the contents of the `environment.yml` file
accordingly by re-issuing `conda env export` command and propagate the updated `environment.yml` file to your collaborators
via your code sharing platform (e.g. GitHub).

> <tip-title>Official Documentation</tip-title>
> For a full list of options and commands, consult the [official `conda` documentation](https://docs.conda.io/projects/conda/en/latest/commands.html)
{: .tip}


topic: data-science
tutorial: python-advanced-np-pd
title: Advanced Python
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 8e6832bd46c20fd5b265af65d333c366
last_modified: 2026-02-26T21:12:14.899505
zenodo_link: https://zenodo.org/record/3477564
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-advanced-np-pd/tutorial.html

content:
In this lesson, we will be using Python 3 with some of its most popular scientific libraries. This tutorial assumes that the reader is familiar with the fundamentals of the Python programming language, as well as, how to run Python programs using Galaxy. Otherwise, it is advised to follow the "Introduction to Python" tutorial available in the same platform. We will be using JupyterNotebook, a Python interpreter that comes with everything we need for the lesson. Please note:  JupyterNotebook is only currently available on the [usegalaxy.eu](https://usegalaxy.eu/) and [usegalaxy.org](https://usegalaxy.org/) sites.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which is licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Analyze data using numpy
NumPy is a python library and it stands for Numerical Python. In general, you should use this library when you want to perform operations and manipulate numerical data, especially if you have matrices or arrays. To tell Python that we’d like to start using NumPy, we need to import it:

```python
import numpy as np
```
A Numpy array contains one or more elements of the same type. To examine the basic functions of the library, we will create an array of random data. These data will correspond to arthritis patients’ inflammation. The rows are the individual patients, and the columns are their daily inflammation measurements. We will use the `random.randint()` function. It has 4 arguments as inputs `randint(low, high=None, size=None, dtype=int)`. `low` nad `high` specify the limits of the random number generator. `size` determines the shape of the array and it can be an integer or a tuple.

```python
np.random.seed(2021)  #create reproducible work
random_data = np.random.randint(1, 25, size=(50,70))
```
If we want to check the data have been loaded, we can print the variable’s value:

```python
print(random_data)
```

Now that the data are in memory, we can manipulate them. First, let’s ask what type of thing data refers to:

```python
print(type(random_data))
```


The output tells us that data currently refers to an N-dimensional array, the functionality for which is provided by the NumPy library. These data correspond to arthritis patients’ inflammation. The rows are the individual patients, and the columns are their daily inflammation measurements.

The `type` function will only tell you that a variable is a NumPy array but won’t tell you the type of thing inside the array. We can find out the type of the data contained in the NumPy array.

```python
print(random_data.dtype)
```

This tells us that the NumPy array’s elements are integer numbers.

With the following command, we can see the array’s shape:

```python
print(random_data.shape)
```

The output tells us that the data array variable contains 50 rows and 70 columns. When we created the variable `random_data` to store our arthritis data, we did not only create the array; we also created information about the array, called members or attributes. This extra information describes `random_data` in the same way an adjective describes a noun. `random_data.shape` is an attribute of `random_data` which describes the dimensions of `random_data`.

If we want to get a single number from the array, we must provide an index in square brackets after the variable name, just as we do in math when referring to an element of a matrix. Our data has two dimensions, so we will need to use two indices to refer to one specific value:

```python
print('first value in data:', random_data[0, 0])
```

```python
print('middle value in data:', random_data[25, 35])
```


The expression random_data[25, 35] accesses the element at row 25, column 35. While this expression may not surprise you, random_data[0, 0] might. Programming languages like Fortran, MATLAB and R start counting at 1 because that’s what human beings have done for thousands of years. Languages in the C family (including C++, Java, Perl, and Python) count from 0 because it represents an offset from the first value in the array (the second value is offset by one index from the first value). As a result, if we have an M×N array in Python, its indices go from 0 to M-1 on the first axis and 0 to N-1 on the second.

Slicing data
An index like [25, 35] selects a single element of an array, but we can select whole sections as well, using slicing the same way as previously with the strings. For example, we can select the first ten days (columns) of values for the first four patients (rows) like this:

```python
print(random_data[0:4, 0:10])
```


We don’t have to include the upper and lower bound on the slice. If we don’t include the lower bound, Python uses 0 by default; if we don’t include the upper, the slice runs to the end of the axis, and if we don’t include either (i.e., if we use ‘:’ on its own), the slice includes everything:

```python
small = random_data[:3, 36:]
print('small is:')
print(small)
```

The above example selects rows 0 through 2 and columns 36 through to the end of the array.

## Process the data
NumPy has several useful functions that take an array as input to perform operations on its values. If we want to find the average inflammation for all patients on all days, for example, we can ask NumPy to compute random_data’s mean value:

```python
print(np.mean(random_data))
```


Let’s use three other NumPy functions to get some descriptive values about the dataset. We’ll also use multiple assignment, a convenient Python feature that will enable us to do this all in one line.

```python
maxval, minval, stdval = np.max(random_data), np.min(random_data), np.std(random_data)

print('maximum inflammation:', maxval)
print('minimum inflammation:', minval)
print('standard deviation:', stdval)
```


How did we know what functions NumPy has and how to use them? If you are working in IPython or in a Jupyter Notebook, there is an easy way to find out. If you type the name of something followed by a dot, then you can use tab completion (e.g. type `np.` and then press Tab) to see a list of all functions and attributes that you can use. After selecting one, you can also add a question mark (e.g. `np.cumprod?`), and IPython will return an explanation of the method! This is the same as doing `help(np.cumprod)`.

When analyzing data, though, we often want to look at variations in statistical values, such as the maximum inflammation per patient or the average inflammation per day. One way to do this is to create a new temporary array of the data we want, then ask it to do the calculation:

```python
patient_0 = random_data[0, :] # 0 on the first axis (rows), everything on the second (columns)
print('maximum inflammation for patient 0:', np.max(patient_0))
```


What if we need the maximum inflammation for each patient over all days (as in the next diagram on the left) or the average for each day (as in the diagram on the right)? As the diagram below shows, we want to perform the operation across an axis:

![A cartoon shows a data structure on the left with Patient 0 to N as rows in a table, and then Day 0 to N as columns of the same table. Below is written "numpy.max(data, axis=1)" which does an aggregation and finds the max per patient (so maximum daily value for each patient.) On the right a similar table is shown but written below is numpy.max(data, axis=0) and now the maximum is calculated per day, which patient had the maximum value.](../../images/python-advanced-np-pd/Figure8_Operations_across_axis.png)

To support this functionality, most array functions allow us to specify the axis we want to work on. If we ask for the average across axis 0 (rows in our 2D example), we get:

```python
print(np.mean(random_data, axis=0))
```


As a quick check, we can ask this array what its shape is:

```python
print(np.mean(random_data, axis=0).shape)
```


The expression (70,) tells us we have an N×1 vector, so this is the average inflammation per day for all patients. If we average across axis 1 (columns in our 2D example), we get the average inflammation per patient across all days.:

```python
print(np.mean(random_data, axis=1))
```


### Stacking arrays
Arrays can be concatenated and stacked on top of one another, using NumPy’s `vstack` and `hstack` functions for vertical and horizontal stacking, respectively.

```python
import numpy as np

A = np.array([[1,2,3], [4,5,6], [7, 8, 9]])
print('A = ')
print(A)

B = np.hstack([A, A])
print('B = ')
print(B)

C = np.vstack([A, A])
print('C = ')
print(C)
```


### Remove NaN values

Sometimes there are missing values in an array, that could make it difficult to perform operations on it. To remove the `NaN` you must first find their indexes and then replace them. The following example replaces them with `0`.

```python
a = array([[1, 2, 3], [0, 3, NaN]])
print(a)
a[np.isnan(a)] = 0
print(a)
```


> <question-title>Selecting and stacking arrays</question-title>
>
> Write some additional code that slices the first and last columns of A, and stacks them into a 3x2 array. Make sure to print the results to verify your solution.
>
> > <solution-title></solution-title>
> >
> > A ‘gotcha’ with array indexing is that singleton dimensions are dropped by default. That means `A[:, 0]` is a one dimensional array, which won’t stack as desired. To preserve singleton dimensions, the index itself can be a slice or array. For example, `A[:, :1]` returns a two dimensional array with one singleton dimension (i.e. a column vector).
> > ```
> > D = np.hstack((A[:, :1], A[:, -1:]))
> > print('D = ')
> > print(D)
> > ```
> >
> {: .solution}
{: .question}

> <question-title>Selecting with conditionals</question-title>
>
> Given the followind array `A`, keep only the elements that are lower that `0.05`.
>```
>A = np.array([0.81, 0.025, 0.15, 0.67, 0.01])
>```
> > <solution-title></solution-title>
> >
> > ```
> > A = A[A<0.05]
> > ```
> >
> {: .solution}
{: .question}



# Use pandas to work with dataframes

Pandas ({% cite reback2020pandas %}, {% cite mckinney-proc-scipy-2010 %}) is a widely-used Python library for statistics, particularly on tabular data. If you are familiar with R dataframes, then this is the library that integrates this functionality. A dataframe is a 2-dimensional table with indexes and column names. The indexes indicate the difference in rows, while the column names indicate the difference in columns. You will see later that these two features are useful when you’re manipulating your data. Each column can contain different data types.

Load it with import pandas as `pd`. The alias `pd` is commonly used for pandas.

```python
import pandas as pd
```

There are many ways to create a pandas dataframe. For example you can use a numpy array as input.

```python
data = np.array([['','Col1','Col2'],
                ['Row1',1,2],
                ['Row2',3,4]])

print(pd.DataFrame(data=data[1:,1:],
                  index=data[1:,0],
                  columns=data[0,1:]))
```



For the purposes of this tutorial, we will use a file with the annotated differentially expressed genes that was produced in the [Reference-based RNA-Seq data analysis]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial

We can read a tabular file with `pd.read_csv`. The first argument is the filepath of the file to be read. The `sep` argument refers to the symbol used to separate the data into different columns. You can check the rest of the arguments using the `help()` function.

```python
data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t")
print(data)
```

The columns in a dataframe are the observed variables, and the rows are the observations. Pandas uses backslash `\` to show wrapped lines when output is too wide to fit the screen.

## Explore the data

You can use `index_col` to specify that a column’s values should be used as row headings.

By default row indexes are numbers, but we could use a column of the data. To pass the name of the column to `read_csv`, you can use its `index_col` parameter. Be careful though, because the row indexes must be unique for each row.

```python
data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
print(data)
```

You can use the `DataFrame.info()` method to find out more about a dataframe.

```python
data.info()
```


We learn that this is a DataFrame. It consists of 130 rows and 12 columns. None of the columns contains any missing values. 6 columns contain 64-bit floating point `float64` values, 2 contain 64-bit integer `int64` values and 4 contain character `object` values. It uses 13.2KB of memory.

The `DataFrame.columns` variable stores information about the dataframe’s columns.

Note that this is an attribute, not a method. (It doesn’t have parentheses.) Called a member variable, or just member.

```python
print(data.columns)
```


You could use `DataFrame.T` to transpose a dataframe. The `Transpose` (written `.T`) doesn’t copy the data, just changes the program’s view of it. Like columns, it is a member variable.

```python
print(data.T)
```

You can use `DataFrame.describe()` to get summary statistics about the data. `DataFrame.describe()` returns the summary statistics of only the columns that have numerical data.  All other columns are ignored, unless you use the argument `include='all'`. Depending on the data type of each column, the statistics that can't be calculated are replaced with  the value `NaN`.

```python
print(data.describe(include='all'))
```


> <question-title>Using pd.head and pd.tail</question-title>
>
> After reading the data, use `help(data.head)` and `help(data.tail)` to find out what `DataFrame.head` and `DataFrame.tail` do.
> 	a. What method call will display the first three rows of the data?
> 	b. What method call will display the last three columns of this data? (Hint: you may need to change your view of the data.)
>
> > <solution-title></solution-title>
> >
> > a. We can check out the first five rows of the data by executing `data.head()` (allowing us to view the head of the DataFrame). We can specify the number of rows we wish to see by specifying the parameter `n` in our call to `data.head()`. To view the first three rows, execute:
> > ```
> > data.head(n=3)
> > ```
> >
> > |  | Base mean | log2(FC) | StdErr | Wald-Stats | P-value | P-adj | Chromosome | Start | End | Strand | Feature | Gene name | GeneID |
> > | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
> > | FBgn0039155 | 1086.974295 | -4.148450 | 0.134949 | -30.740913 | 1.617357e-207 | 1.387207e-203 | chr3R | 24141394 | 24147490 | + | protein_coding | Kal1 |
> > | FBgn0003360 | 6409.577128 | -2.999777 | 0.104345 | -28.748637 | 9.419922e-182 | 4.039734e-178 | chrX | 10780892 | 10786958 | - | protein_coding | sesB |
> > | FBgn0026562 | 65114.840564 | -2.380164 | 0.084327 | -28.225437 | 2.850430e-175 | 8.149380e-172 | chr3R | 26869237 | 26871995 | - | protein_coding | BM-40-SPARC |
> >
> >
> >    b. To check out the last three rows, we would use the command, `data.tail(n=3)`, analogous to `head()` used above. However, here we want to look 	at the last three columns so we need to change our view and then use `tail()`. To do so, we create a new DataFrame in which rows and columns are 	switched:
> >```
> >data_flipped = data.T
> >```
> > We can then view the last three columns of the data by viewing the last three rows of data_flipped:
> > ```
> > data_flipped.tail(n=3)
> > ```
> > | GeneID | FBgn0039155 | FBgn0003360 | FBgn0026562 | FBgn0025111 | FBgn0029167 | FBgn0039827 | FBgn0035085 | FBgn0034736 | FBgn0264475 | FBgn0000071 | ... | FBgn0264343 | FBgn0038237 | FBgn0020376 | FBgn0028939 | FBgn0036560 | FBgn0035710 | FBgn0035523 | FBgn0038261 | FBgn0039178 | FBgn0034636 |
> > | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
> > | Strand | + | - | - | - | + | + | + | + | + | + | ... | + | - | + | + | + | - | + | + | + | - |
> > | Feature | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | lincRNA | protein_coding | ... | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding |
> > | Gene name | Kal1 | sesB | BM-40-SPARC | Ant2 | Hml | CG1544 | CG3770 | CG6018 | CR43883 | Ama | ... | CG43799 | Pde6 | Sr-CIII | NimC2 | CG5895 | SP1173 | CG1311 | CG14856 | CG6356 | CG10440 |
> >
> {: .solution}
{: .question}

 > <question-title>Saving in a csv file</question-title>
>
> As well as the `read_csv` function for reading data from a file, Pandas provides a `to_csv` function to write dataframes to files. Applying what you’ve learned about reading from files, write one of your dataframes to a file called `processed.csv`. You can use `help` to get information on how to use `to_csv`.
>
> > <solution-title></solution-title>
> >
> > ```
> > data_flipped.to_csv('processed.csv')
> > ```
> >
> {: .solution}
{: .question}


* Note about Pandas DataFrames/Series

A [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) is a collection of [Series](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html); The DataFrame is the way Pandas represents a table, and Series is the data-structure Pandas use to represent a column.

Pandas is built on top of the Numpy library, which in practice means that most of the methods defined for Numpy Arrays apply to Pandas Series/DataFrames.

What makes Pandas so attractive is the powerful interface to access individual records of the table, proper handling of missing values, and relational-databases operations between DataFrames.

## Select data
To access a value at the position `[i,j]` of a DataFrame, we have two options, depending on what is the meaning of i in use. Remember that a DataFrame provides an index as a way to identify the rows of the table; a row, then, has a position inside the table as well as a label, which uniquely identifies its entry in the DataFrame.

You can use `DataFrame.iloc[..., ...]` to select values by their (entry) position and basically specify location by numerical index analogously to 2D version of character selection in strings.

```python
print(data.iloc[0, 0])
```
You can also use `DataFrame.loc[..., ...]` to select values by their (entry) label and basically specify location by row name analogously to 2D version of dictionary keys.

```python
print(data.loc["FBgn0039155", "Base mean"])
```

You can use Python's usual slicing notation, to select all or a subset of rows and/or columns. For example, the following code selects all the columns of the row `"FBgn0039155"`.

```python
print(data.loc["FBgn0039155", :])
```

Which would get the same result as printing `data.loc["FBgn0039155"]` (without a second index).

You can select multiple columns or rows using `DataFrame.loc` and a named slice or `Dataframe.iloc` and the numbers corresponding to the rows and columns.

```python
print(data.loc['FBgn0003360':'FBgn0029167', 'Base mean':'Wald-Stats'])
print(data.iloc[1:4 , 0:3])
```

* Note the difference between the 2 outputs.

When choosing or transitioning between `loc` and `iloc`, you should keep in mind that the two methods use slightly different indexing schemes.

`iloc` uses the Python stdlib indexing scheme, where the first element of the range is included and the last one excluded. So `0:10` will select entries `0,...,9`. `loc`, meanwhile, indexes inclusively. So `0:10` will select entries `0,...,10`.

This is particularly confusing when the DataFrame index is a simple numerical list, e.g. `0,...,1000`. In this case `df.iloc[0:1000]` will return 1000 entries, while `df.loc[0:1000]` return 1001 of them! To get 1000 elements using `loc`, you will need to go one lower and ask for `df.loc[0:999]`.

The result of slicing is a new dataframe and can be used in further operations. All the statistical operators that work on entire dataframes work the same way on slices. E.g., calculate max of a slice.

```python
print(data.loc['FBgn0003360':'FBgn0029167', 'Base mean'].max())
```


## Use conditionals to select data

You can use conditionals to select data. A comparison is applied element by element and returns a similarly-shaped dataframe of `True` and `False`. The last one can be used as a mask to subset the original dataframe. The following example creates a new dataframe consisting only of the columns 'P-adj' and 'Gene name', then keeps the rows that comply with the expression `'P-adj' < 0.000005`

```python
subset = data.loc[:, ['P-adj', 'Gene name']]
print(subset)
```

```python
mask = subset.loc[:, 'P-adj'] < 0.000005
new_data = subset[mask]
print(new_data)
```

If we have not had specified the column, that the expression should be applied to, then it would have been applied to the entire dataframe. But the dataframe contains different type of data. In that case, an error would occur.

Consider the following example of a dataframe consisting only of numerical data. The expression and the mask would be normally applied to the data and the mask would return `NaN` for the data that don't comply with the expression.

```python
subset = data.loc[:, ['StdErr',	'Wald-Stats', 'P-value', 'P-adj']]
mask = subset < 0.05
new_data = subset[mask]
print(new_data)
```

This is very useful because NaNs are ignored by operations like max, min, average, etc.

```python
print(new_data.describe())
```

> <question-title>Manipulating dataframes</question-title>
>
> Explain what each line in the following short program does: what is in first, second, etc.?
>
>```
>first = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
>second = first[first['log2(FC)'] > 0 ]
>third = second.drop('FBgn0025111')
>fourth = third.drop('StdErr', axis = 1)
>fourth.to_csv('result.csv')
>```
> > <solution-title></solution-title>
> >
> > Let’s go through this piece of code line by line.
> > ```
> > first = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
> > ```
> > This line loads the data into a dataframe called first. The `index_col='GeneID'` parameter selects which column to use as the row labels in the dataframe.
> >
> > ```
> > second = first[first['log2(FC)'] > 0 ]
> > ```
> >
> > This line makes a selection: only those rows of first for which the ‘log2(FC)’ column contains a positive value are extracted. Notice how the Boolean expression inside the brackets is used to select only those rows where the expression is true.
> >
> > ```
> > third = second.drop('FBgn0025111')
> > ```
> >
> > As the syntax suggests, this line drops the row from second where the label is ‘FBgn0025111’. The resulting dataframe third has one row less than the original dataframe second.
> >
> > ```
> > fourth = third.drop('StdErr', axis = 1)
> > ```
> >
> > Again we apply the drop function, but in this case we are dropping not a row but a whole column. To accomplish this, we need to specify also the axis parameter.
> >
> > ```
> > fourth.to_csv('result.csv')
> > ```
> >
> >The final step is to write the data that we have been working on to a csv file. Pandas makes this easy with the `to_csv()` function. The only required argument to the function is the filename. Note that the file will be written in the directory from which you started the Jupyter or Python session.
> >
> {: .solution}
{: .question}


> <question-title>Finding min-max indexes</question-title>
>
> Explain in simple terms what `idxmin` and `idxmax` do in the short program below. When would you use these methods?
> ```
> data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
>
> print(data['Base mean'].idxmin())
> print(data['Base mean'].idxmax())
> ```
>
> > <solution-title></solution-title>
> >
> > `idxmin` will return the index value corresponding to the minimum; idxmax will do the same for the maximum value.
> >
> > You can use these functions whenever you want to get the row index of the minimum/maximum value and not the actual minimum/maximum value.
> >
> > Output:
> > FBgn0063667
> > FBgn0026562
> >
> {: .solution}
{: .question}

> <question-title>Selecting with conditionals</question-title>
>
> Assume Pandas has been imported and the previous dataset has been loaded. Write an expression to select each of the following:
> a. P-value of each gene
> b. all the information of gene `FBgn0039178`
> c. the information of all genes that belong to chromosome `chr3R`
>
> > <solution-title></solution-title>
> >
> > a. ```data['P-value']```
> > b.```data.loc['FBgn0039178', :]```
> > c. ```data[data['Chromosome'] == 'chr3R']```
> >
> {: .solution}
{: .question}



## Group-by and analyze the data

Many data analysis tasks can be approached using the “split-apply-combine” paradigm: split the data into groups, apply some analysis to each group, and then combine the results.

Pandas makes this very easy through the use of the `groupby()` method, which splits the data into groups. When the data is grouped in this way, the aggregate method `agg()` can be used to apply an aggregating or summary function to each group.

```python
summarised_data = data.groupby('Chromosome').agg({'Base mean':'first',
                             'log2(FC)': 'max'})
print(summarised_data)
```

There are a couple of things that should be noted. The `agg()` method accepts a dictionary as input that specifies the function to be applied to each column. The output is a new dataframe, that each row corresponds to one group. The output dataframe uses the grouping column as index. We could change the last one by simply using the `reset_index()` method.

```python
summarised_data = data.groupby('Chromosome').agg({'Base mean':'first',
                                                  'log2(FC)': 'max'}).reset_index()
print(summarised_data)
```

> <question-title>Finding the max of each group</question-title>
>
> Using the same dataset, try to find the longest genes in each chromosome.
>
>> <solution-title></solution-title>
> >
> > ```
> > data['Gene Length'] = data['End'] - data['Start']
> > data.groupby('Chromosome').agg(max_length = ('Gene Length', 'max'))
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Grouping with multiple variables</question-title>
>
> Using the same dataset, try to find how many genes are found on each strand of each chromosome.
>
> > <solution-title></solution-title>
> >
> > You can group the data according to more than one column.
> > ```
> > data.groupby(['Chromosome', 'Strand']).size()
> > ```
> {: .solution}
{: .question}

# Conclusion

This tutorial aims to serve as an introduction to data analysis using the Python programming language. We hope you feel more confident in Python!


topic: data-science
tutorial: sql-game
title: SQL Educational Game - Murder Mystery
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["game", "SQL"]
content_hash: 7ea4d26c9e37ece7cc088606fc89c416
last_modified: 2026-02-26T21:12:14.911922
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-game/tutorial.html

content:
This is not a tutorial like most GTN content but a fun exercise for you to play around and learn a bit about SQL in a more 'practical', and hopefully re-inforce the skills you covered in Basic and Advanced SQL skills. It makes use of the [NUKnightLab/sql-mysteries](https://github.com/NUKnightLab/sql-mysteries) SQL murder mystery project and released under open licenses:

> <comment-title>Source</comment-title>
> Original code for NUKnightLab/sql-mysteries is released under the MIT License.
> Original text and other content is released under Creative Commons CC BY-SA 4.0.
{: .comment}

Download the database and connector:

```python
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c https://github.com/NUKnightLab/sql-mysteries/raw/master/sql-murder-mystery.db
```

Setup the database connection:

```python
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///sql-murder-mystery.db")
%load_ext sql
%sql sqlite:///sql-murder-mystery.db
%config SqlMagic.displaycon=False
```

## Tables

Which tables are available to you? What columns do they contain? Here's a handy reference for you:

```python
import pandas as pd
from sqlalchemy import MetaData
m = MetaData()
m.reflect(engine)
results = []
for table in m.tables.values():
    results.append([table.name, ', '.join([c.name for c in table.c])])
pd.set_option('display.max_colwidth', None)
pd.DataFrame(results, columns=["Table", "Columns"])
```


## The search!

Begin your search for the truth

> A crime has taken place and the police are both useless *and* corrupt and it
> is up to you and your community to solve the mystery. They failed to secure
> their database, and now their crime scene reports are public, it is time to
> figure out who the murderer was.
>
> You know that the crime was a **murder** that occurred sometime on **Jan.15,
> 2018** and that it took place in **SQL City**.
>
> All the clues to this mystery are buried in a huge database, and you need to
> use SQL to navigate through this vast network of information. Your first step
> to solving the mystery is to retrieve the corresponding crime scene report
> from the police department’s database. From there, you can use your SQL
> skills to find the murderer.
{: .quote}

```python
%%sql
select * from crime_scene_report limit 8;
```

Try using the 'Insert → Cell Below' functionality to keep track of important query results as you go!

## Solution

Write the following queries in your SQL environment to check whether you've found the right murderer:

```python
%%sql
INSERT INTO solution VALUES (1, "Insert the name of the person you found here");
SELECT value FROM solution;
```


topic: data-science
tutorial: python-iterables
title: Python - Lists & Strings & Dictionaries
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 30df0f2dedfa5a5eeeeea1fbece9d31d
last_modified: 2026-02-26T21:12:14.902666
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-iterables/tutorial.html

content:
Doing calculations with hundreds of variables called `pressure_001`, `pressure_002`, etc. would be at least as slow as doing them by hand. Using a *list* to store many values together solves that problems. Lists are surrounded by square brackets: `[`, `]`, with values separated by commas:

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


```python
pressures = [0.273, 0.275, 0.277, 0.275, 0.276]
print(f'pressures: {pressures}')
print(f'length: {len(pressures)}')
```

## Indexing

You can use an item's index to fetch it from a list.

```python
print(f'zeroth item of pressures: {pressures[0]}')
print(f'fourth item of pressures: {pressures[4]}')
```

## Replacement

Lists' values can be changed or replaced by assigning a new value to the position in the list.

```python
pressures[0] = 0.265
print(f'pressures is now: {pressures}')
```

Note how the first item has changed from `0.273`

## Appending

Appending items to a list lengthens it. You can do `list_name.append()` to add items to the end of a list.

```python
primes = [2, 3, 5]
print(f'primes is initially: {primes}')
primes.append(7)
print(f'primes has become: {primes}')
```


`.append()` is a *method* of lists. It's like a function, but tied to a particular object. You use `object_name.method_name` to call methods, which deliberately resembles the way we refer to things in a library.

We will meet other methods of lists as we go along. Use `help(list)` for a preview. `extend` is similar to `append`, but it allows you to combine two lists.  For example:

```python
teen_primes = [11, 13, 17, 19]
middle_aged_primes = [37, 41, 43, 47]
print(f'primes is currently: {primes}')
primes.extend(teen_primes)
print(f'primes has now become: {primes}')
primes.append(middle_aged_primes)
print(f'primes has finally become: {primes}')
```

Note that while `extend` maintains the "flat" structure of the list, appending a list to a list makes the result two-dimensional - the last element in `primes` is a list, not an integer.

This starts to become a more complicated data structure, and we'll use more of these later. A list containing both integers and a list can be called a "hetereogenous" list, since it has multiple different data types. This is relatively uncommon, most of the lists you'll encounter will have a single data type inside of them. Sometimes you'll see a list of lists, which can be used to store positions, like a chessboard.

## List Indices

In computer science and programming we number the positions within a list starting from `0`, rather than from `1`.

```python
# Position  0         1          2            3           4
weekdays = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday']
print(weekdays[0])
print(weekdays[4])
print(weekdays[3])
```

But if you try an access a position that is outside of the list, you'll get an error

```python
print(weekdays[9])
```

returns a `IndexError: list index out of range`.

> <tip-title>Reading Error Messages</tip-title>
> So how do you read this?
> ```
> 1 | ---------------------------------------------------------------------------
> 2 | IndexError                                Traceback (most recent call last)
> 3 | /tmp/ipykernel_648319/137030145.py in <module>
> 4 | ----> 1 print(weekdays[9])
> 5 |
> 6 | IndexError: list index out of range
> ```
>
> 1. This is just a line of `-`s as a separator
> 2. `IndexError`, here Jupyter/CoCalc/etc are trying to be helpful and highlight the error for us. This is the important bit of information!
> 3. This is the path to where the code is, Jupyter/CoCalc/etc create temporary files to execute your code.
> 4. Here an arrow points to the line number where something has broken. 1 shows that it's the first line within the cell, and it points to the print statement. Really it's pointing at the `weekdays[9]` within the print statement.
> 5. Blank
> 6. This is where we normally look for the **most important part of the Traceback**. The error message. An `IndexError`, namely that the list index (9) is out of the range of possible values (the length of the list.)
{: .tip}

However, sometimes you want to access the very end of a list! You can either start at the beginning and count along to find the last item or second to last item, or you can use Negative Indices

```python
# Position  0         1          2            3           4
weekdays = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday']
# Position  -5        -4         -3           -2          -1

print(weekdays[-1])
print(weekdays[-2])
print(weekdays[-4])
```

If you wanted to find the last value in a list, you could also use `len(elements)` and then subtract back to find the index you want

```python
elements[len(elements)-1]
```

This is essentially how negative indexes work, except you don't have to use `len(elements)`, that's done for you automatically.


## Removing Items.

You can use `del` to remove items from a list entirely. We use `del list_name[index]` to remove an element from a list (in the example, 9 is not a prime number) and thus shorten it. `del` is not a function or a method, but a statement in the language.

```python
primes = [2, 3, 5, 7, 9]
print(f'primes before removing last item: {primes}')
del primes[4]
print(f'primes after removing last item: {primes}')
```

## Empty Lists

The empty list contains no values. When you want to make a new list, use `[]` on its own to represent a list that doesn't contain any values. This is helpful as a starting point for collecting values, which we'll see soon.

## Heterogeneous Lists

Lists may contain values of different types. A single list may contain numbers, strings, and anything else.

```python
goals = [1, 'Create lists.', 2, 'Extract items from lists.', 3, 'Modify lists.']
```

## Strings are like Lists

Text is often called a "string" in the programming world. Strings of text like `name = "Helena"` or `patient_id = "19237zud830"` are very similar conceptually to lists. Except instead of being a list of numbers, they're a lists of characters.

In a number of older programming languages, strings are indeed arrays of numbers internally. However python hides a lot of that complexity from us, so we can just work with text.

Still, many of the operations you use on lists, can also be used on strings as well! Strings can be indexed like lists so you can get single elements from lists.

```python
element = 'carbon'
print(f'zeroth character: {element[0]}')
print(f'third character: {element[3]}')
```

Strings, however, cannot be modified, you can't change a single letter in a string. Things that cannot be modified after creation are called *immutable* or sometimes *frozen*, compared to things which can be modified which are called *mutable*.
Python considers the string to be a single value with parts, not a collection of values.

```python
element[0] = 'C'
```

## Bounds

You cannot access values beyond the end of the list, this will result in an error. Python reports an `IndexError` if we attempt to access a value that doesn't exist. This is a kind of **runtime error**, as it cannot be detected as the code is parsed. Imagine if you had a script which let you read in a file, depending on how many lines were in the file, whether index 90 was valid or invalid, would depend on how big your file was.

```python
print(f'99th element of element is: {element[99]}')
```

## Exercises

> <question-title>Checking suffixes</question-title>
>
> 1. How could you check that the extension of a filename is `.csv`
> 2. Can you find another way? Maybe check the help page for `str`
>
> > <solution-title></solution-title>
> >
> > 1. `a[-4:] == "csv"` (Here we use `==` for comparing two values)
> > 2. `a.endswith('.csv')`
> >
> {: .solution}
{: .question}

```python
# Test code here!
a = "1234.csv"
b = "1273.tsv"
c = "9382.csv"
d = "1239.csv"
```

> <question-title>Say it loud!</question-title>
>
> 1. Can you find a method in the `str`'s help that converts the string to upper case
> 2. or lower case?
> 3. Can you use it to fix mixed case DNA sequence?
>
> > <solution-title></solution-title>
> > 1. `"shout it out".upper()`
> > 2. `"WHISPER THIS".lower()`
> > 3. `terrible_sequence.upper()`
> {: .solution}
{: .question}

```python
# Test answers here!
print("shout it out")
print("WHISPER THIS")
# Fix this mess to be all capital
terrible_sequence = "AcTGAGccGGTt"
print(terrible_sequence)
```


> <question-title>Splitting</question-title>
> 1. We use `.split()` to split a string by some character. Here we have a comma separated list of values, try splitting that up by a comma, but we actually wanted it separated by `|` characters. Can you split it up, and then re-join it with that new character?
> 2. Does `help(str)` give you another option for replacing a character like that.
> 3. What happens if you split by another value like `3`?
>
> > <solution-title></solution-title>
> > 1. `data.split(",")`
> > 2. `data.replace(",", "|")`
> > 3. Those characters will disappear! If you want to reconstruct the same string
> {: .solution}
{: .question}

```python
# Split me
data = "0,0,1,3,1,2,4,7,8,3,3,3,10,5,7,4,7,7,12,18,6,13,11,11,7,7,4,6,8,8,4,4,5,7,3,4,2,3,0,0"
```

# Slicing & Dicing

All of the data types we've talked about today can be sliced, and this will be a key part of using lists.

```python
elements = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F']
# Instead of accessing a single element
print(elements[0])
# We'll access a range
print(elements[0:4])
```

Accessing only a portion of a list is commonly used, say if you have a list of FastQ files from paired end sequencing, perhaps you want two of them at a time. You could access those with `[0:2]`.

```python
# You don't need to start at 0
print(elements[6:8])
```

```python
# But your end should be bigger than your start.
# What do you think this will return?
# Make a guess before you run it
print(elements[6:5])
```

If you don't supply an end value, Python will default to going to the end of the list. Likewise, if you don't provide a start value, Python will use `0` as the start by default, until whatever end value you provide.


> <question-title>Valid and Invalid Slices</question-title>
> Which of these do you think will be valid? Which are invalid? Predict what they will return:
> ```
> # 1
> elements = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F']
> # 2
> elements[0:3]
> # 3
> elements[:3]
> # 4
> elements[-3:3]
> # 5
> elements[-8:-3]
> # 6
> elements[:]
> # 7
> elements[0:20]
> # 8
> elements['H':'Li']
> # 9
> elements[1.5:]
> ```
>
> > <solution-title></solution-title>
> > All of these are valid except the last two.
> > 1. If you dont' fill in a position, Python will use the default. 0 for the left hand side of the `:`, and `len(elements)` for the right hand side.
> > 2. You can request a slice longer than your list (e.g. up to 20), but Python may not give you that many items back.
> > 3. List slicing can only be done with integers, not floats.
> {: .solution}
{: .question}

```python
# Check your answers here!
```

## Stride

However, list slicing can be more complicated. You can additionally use a 'stride' parameter, which is how Python should strep through the list. To take every other element from a list:

```python
values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
print(values[0:12:2]) # every other value
print(values[1:12:2]) # every other value from the second value
print(values[::2]) # the start and end are optional
print(values[::3]) # every third value in the list.
```

So list slicing together is either `list[low:high]` or `list[low:high:stride]`, where low and high are optional if you just want to go to the end of the list.

## Sorting

Lists occasionally need to be sorted. For example, you have a list of students you might want to alphabetise, and here you can use the function `sorted` to help you.

```python
students = [
    'Koos Christabella',
    'Zackary Habiba',
    'Jumana Rostam',
    'Sorina Gaia',
    'Kalyani Bessarion',
    'Enéas Nirmala',
    '王奕辰',
    '刘依诺',
]
students = sorted(students)
print(students)
```

> <tip-title>Sorting names is hard!</tip-title>
> Some people have 1 name, some have 4 or more! Some cultures have surnames first, some not. Sorting names is a complex situation, so be sure you consider your data before sorting and assuming it's correct. Test with multiple values to make sure your code works!
{: .tip}

> <tip-title>Results can be dependent on analysis order!</tip-title>
> Some analyses (especially simultaions) can be dependent on data input order or data sorting. This was recently seen in {% cite Bhandari_Neupane_2019 %} where the data files used were sorted one way on Windows, and another on Linux, resulting in different results for the same code and the same datasets! Yikes!
>
> If you know your analyses are dependent on file ordering, then you can use `sorted()` to make sure the data is provided in a uniform way every time.
>
> If you're not sure if your results will be dependent, you can try sorting anyway. Or better yet, randomising the list of inputs to make sure your code behaves properly in any scenario.
{: .tip}

## Type Conversion

Just list with converting `"1.5"` to an float with the `float()` function, or `3.1` to a string with `str()`, we can do the same with lists using the `list()` function, and sets with `set()`:

```python
# Convert text to a list
print(list("sometext"))
```

Converting a list back into text is likewise possible, but you need to use the special function `join`. Join is a function of a `str`, which accepts a list

```python
word = ['c', 'a', 'f', 'e']
print("-".join(word))
```

It takes the string you called it on, and uses that as a separator. Then for the list that you provide, it joins that together with the separator.

# Exercise Time

> <question-title>Fill in the Blanks</question-title>
>
> Fill in the blanks so that the program below produces the output shown.
>
> ```
> values = ____
> values.____(1)
> values.____(3)
> values.____(5)
> print(f'first time: {values}')
> values = values[____]
> print(f'second time: {values}')
> ```
>
> ```
> first time: [1, 3, 5]
> second time: [3, 5]
> ```
>
> > <solution-title></solution-title>
> > ```
> > values = []
> > values.append(1)
> > values.append(3)
> > values.append(5)
> > print(f'first time: {values}')
> > values = values[1:]
> > print(f'second time: {values}')
> > ```
> {: .solution}
{: .question}

```python
# Fill in the blanks here!

values = ____
values.____(1)
values.____(3)
values.____(5)
print(f'first time: {values}') # Should print [1, 3, 5]
values = values[____]
print(f'second time: {values}') # should print [3, 5]
```

> ## How Large is a Slice?
>
> If `start` and `stop` are both non-negative integers,
> how long is the list `values[start:stop]`?
>
> > ## Solution
> > The list `values[start:stop]` has up to `stop - start` elements.  For example,
> > `values[1:4]` has the 3 elements `values[1]`, `values[2]`, and `values[3]`.
> > Why 'up to'?
> > If `stop` is greater than the total length of the list `values`,
> > we will still get a list back but it will be shorter than expected.
> {: .solution}
{: .question}

> ## From Strings to Lists and Back
>
> Given this:
>
> ```
> print(f'string to list: {list('tin')}')
> print(f'list to string: {''.join(['g', 'o', 'l', 'd'])}')
> ```
>
> 1.  What does `list('some string')` do?
> 2.  What does `'-'.join(['x', 'y', 'z'])` generate?
>
> > ## Solution
> > 1. [`list('some string')`](https://docs.python.org/3/library/stdtypes.html#list) converts a string into a list containing all of its characters.
> > 2. [`join`](https://docs.python.org/3/library/stdtypes.html#str.join) returns a string that is the _concatenation_
> >    of each string element in the list and adds the separator between each element in the list. This results in
> >    `x-y-z`. The separator between the elements is the string that provides this method.
> {: .solution}
{: .question}

```python
# Test code here
```

> ## Working With the End
>
> What does the following program print?
>
> ```
> element = 'helium'
> print(element[-1])
> ```
>
> 1.  How does Python interpret a negative index?
> 2.  If a list or string has N elements,
>     what is the most negative index that can safely be used with it,
>     and what location does that index represent?
> 3.  If `values` is a list, what does `del values[-1]` do?
> 4.  How can you display all elements but the last one without changing `values`?
>     (Hint: you will need to combine slicing and negative indexing.)
>
> > ## Solution
> > The program prints `m`.
> > 1. Python interprets a negative index as starting from the end (as opposed to
> >    starting from the beginning).  The last element is `-1`.
> > 2. The last index that can safely be used with a list of N elements is element
> >    `-N`, which represents the first element.
> > 3. `del values[-1]` removes the last element from the list.
> > 4. `values[:-1]`
> {: .solution}
{: .question}

```python
# Test code here
```

> ## Stepping Through a List
>
> What does the following program print?
>
> ```
> element = 'fluorine'
> print(element[::2])
> print(element[::-1])
> ```
>
> 1.  If we write a slice as `low:high:stride`, what does `stride` do?
> 2.  What expression would select all of the even-numbered items from a collection?
>
> > ## Solution
> > The program prints
> > ```
> > furn
> > eniroulf
> > ```
> > 1. `stride` is the step size of the slice.
> > 2. The slice `1::2` selects all even-numbered items from a collection: it starts
> >    with element `1` (which is the second element, since indexing starts at `0`),
> >    goes on until the end (since no `end` is given), and uses a step size of `2`
> >    (i.e., selects every second element).
> {: .solution}
{: .question}

```python
# Test code here
```

> ## Slice Bounds
>
> What does the following program print?
>
> ```
> element = 'lithium'
> print(element[0:20])
> print(element[-1:3])
> ```
>
> > ## Solution
> > ```
> > lithium
> >
> > ```
> >
> > The first statement prints the whole string, since the slice goes beyond the total length of the string.
> > The second statement returns an empty string, because the slice goes "out of bounds" of the string.
> {: .solution}
{: .question}

```python
# Test code here
```

# Dictionaries

When you think of a Dictionary, you should think of a real life Dictionary, they map some key to a value. Like a term to it's definition

Key | Value
--- | ---
`Eichhörnchen` | Squirrel
`火锅` | Hot Pot

Or a Country to it's population

Key               | Value
---               | ---
South Sudan       | 492,970
Australia         | 411,667
Guinea            | 1,660,973
Morocco           | 573,895
Maldives          | 221,678
Wallis and Futuna | 1,126
Eswatini          | 94,874
Namibia           | 325,858
Turkmenistan      | 1,031,992

In Python we create a dictionary with `{}` and use `:` to separate keys and values. Turning the above list into a Python dictionary, it would look like:

```python
populations = {
  "South Sudan": 492970,
  "Australia": 411667,
  "Guinea": 1660973,
  "Morocco": 573895,
  "Maldives": 221678,
  "Wallis and Futuna": 1126,
  "Eswatini": 94874,
  "Namibia": 325858,
  "Turkmenistan": 1031992,
}
```

You can see a string (the country name) being used for the key, and then the number (an integer) as the value. (Would a float make sense? Why or why not?)

> <tip-title>Other Names</tip-title>
> They're also sometimes called associative arrays (because they're an array or list of values that associate a key to a value) or maps (because they map a key to a value), depending on what you're reading.
{: .tip}

## Methods

You can access both the keys, and the values

```python
print(populations.keys())
print(populations.values())
```

These will print out two list-like objects. They will become more useful in the future when we talk about looping over dictionaries and processing all of the values within.

## Accessing Values

Just like lists where you access by the position in the list

```python
print(populations["Namibia"])
```

And just like lists, if you try an access a key that isn't there or an index outside of the range of the list:

```python
print(populations["Mars"])
```


> <tip-title>Dictionaries are faster than lists for looking up values</tip-title>
> Just like in real life, searching a dictionary for a specific term is quite fast. Often a lot faster than searching a list for a specific value.
>
> For those of you old enough to remember the paper version of a dictionary, you knew that As would be at the start and Zs at the end, and probably Ms around the middle. And if you were looking for a word like "Squirrel", you'd open the dictionary in the middle, maybe decide it was in the second half of the book, randomly choose a page in the second half, and you could keep deciding if it was "before" or "after" the current page, never even bothering to search the first half.
>
> Conceptually, compared with a list, you can't make this guess of if the item is in the first or second half. You need to search item by item, it would be like reading page by page until you get to Squirrel in the dictionary.
{: .tip}

## Modifying Dictionaries

Adding new values to a dictionary is easy, it's very similar to replacing a value in a list.

```python
# For lists we did
x = ['x', 'y', 'z']
x[0] = 'a'
print(x)
```

For dictionaries, it's essentially the same, we access the 'place' in the dictionary just like we did with a list, and set it to a value

```python
populations["Mars"] = 6 # robots
print(populations)
```

And similarly, removing items is the same as it was for lists:

```python
print(x)
del x[0] # Removes the first item
print(x)
```

And with dictionaries you delete by specifying which position/key you want to remove

```python
del populations['Australia']
print(populations)
```

## Exercises

> <question-title>DNA Complement</question-title>
>
> DNA is usually in the form of dsDNA, a paired strand, where A maps to T and C maps to G and vice versa.
> But when we're working with DNA sequences in bioinformatics, we often only store one strand, because we can calculate the *complement* on the fly, when we need.
>
> Write a dictionary that lets you look up the letters A, C, T, and G and find their complements.
>
> > <solution-title></solution-title>
> > You need to have the complements of every base. If you just defined 'A' and 'C', how would you look up the complement when you want to translate a 'T' or a 'G'? It's not easily possible to look up a key by a value, only to search a key and find a value.
> > ```
> > translation = {
> > 'A': 'T',
> > 'T': 'A',
> > 'C': 'G',
> > 'G': 'C',
> > }
> > ```
> {: .solution}
{: .question}

```python
# Test code here!
translation = {

}
print(translation)
```

> <question-title>Modifying an array</question-title>
>
> Fill in the blanks to make the execution correct:
>
> ```
> variants = {
>   'B.1.1.7': 26267,
>   'B.1.351': 439,
> }
> variants[_____] =  _____
> print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384}
> __________
> print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384, 'B.1.617.2': 43486}
> # Maybe we've exterminated B.1.1.7 and B.1.351, remove their numbers.
> del _______
> del _______
> print(variants[______]) # Should print 384
> print(variants[______]) # Should print 43486
> ```
>
> > <solution-title></solution-title>
> >
> > variants = {
> >   'B.1.1.7': 26267,
> >   'B.1.351': 439,
> > }
> > variants['P.1'] = 384
> > print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384}
> > variants['B.1.617.2'] = 43486
> > print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384, 'B.1.617.2': 43486}
> > # Maybe we've exterminated B.1.1.7 and B.1.351, remove their numbers.
> > del variants['B.1.1.7']
> > del variants['B.1.351']
> > print(variants['P.1']) # Should print 384
> > print(variants['B.1.617.2']) # Should print 43486
> > ```
> >
> {: .solution}
{: .question}

```python
# Test code here!
variants = {
  'B.1.1.7': 26267,
  'B.1.351': 439,
}
variants[_____] =  _____
print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384}
__________
print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384, 'B.1.617.2': 43486}
# Maybe we've exterminated B.1.1.7 and B.1.351, remove their numbers.
del _______
del _______
print(variants[______]) # Should print 384
print(variants[______]) # Should print 43486
```


# Choosing the Right Data Type

Choosing the correct data type can sometimes require some thought, and even discussion with colleagues. And don't be afraid to search the internet for how other people have done it!


Data type       | Examples                  | When to use it | When **not** to use it
---             | ---                       | ---- | ----
Boolean (`bool`) | `True`, `False` | If there are only two possible states, true or false | If your data is not binary
Integer (`int`) | 1, 0, -1023, 42           | Countable, singular items. How many patients are there, how many events did you record, how many variants are there in the sequence | If doubling or halving the value would not make sense: do not use for e.g. patient IDs, or phone numbers. If these are integers you might accidentally do math on the value.
Float (`float`) | 123.49, 3.14159, -3.33334 | If you need more precision or partial values. Recording distance between places, height, mass, etc.
Strings (`str`) | 'patient_12312', 'Jane Doe', '火锅' | To store free text, identifiers, sequence IDs, etc. | If it's truly a numeric value you can do calculations with, like adding or subtracting or doing statistics.
List / Array (`list`) | `['A', 1, 3.4, ['Nested']]` | If you need to store a list of items, like sequences from a file. Especially if you're reading in a table of data from a file. | If you want to retrieve individual values, and there are clear identifiers it might be better as a dict.
Dictionary / Associative Array / map (`dict`) | `{"weight": 3.4, "age": 12, "name": "Fluffy"}` | When you have identifiers for your data, and want to look them up by that value. E.g. looking up sequences by an identifier, or data about students based on their name. Counting values. | If you just have a list of items without identifiers, it makes more sense to just use a list.


## Exercises

> <question-title>Which Datatype</question-title>
>
> 1. Chromosome Length
> 1. Name
> 2. Weight
> 3. Sex
> 4. Hair Colour
> 5. Money/Currency
>
> > <solution-title></solution-title>
> >
> > 1. Here you need to use an integer, a fractional or float value would not make sense. You cannot have half an A/C/T/G.
> > 1. Here a string would be a good choice. (And probably just a single `name` string, rather than a `first` and `last` name, as not all humans have two names! And some have more than two.)
> > 2. An integer is good type for storing weight, if you are using a small unit (e.g. grams). Otherwise you might consider a float, but you'd need to be careful to format it properly (e.g. `{value:0.2f}`) when printing it out. It depends on the application.
> > 3. This is a case where you should consider carefully the application, but `bool` is usally the *wrong answer*. Are you recording patient data? Is their expressed gender the correct variable or did you need sex? {% cite Miyagi_2021 %} goes into detail on this multifaceted issue in a medical research context. For example chromosomal sex is also more complicated and cannot be stored with a true/false value, as people with [Kleinfelters](https://en.wikipedia.org/wiki/Klinefelter_syndrome) exist. A string can be an ok choice here.
> > 4. There is a limited vocabulary humans use to describe hair colour, so a string can be used, or a data type we haven't discussed! An `enum` is an `enumeration`, and when you have a limited set of values that are possible, you can use a `enum` to double check that whatever value is being used (or read from a file, or entered by a user) matches one of the "approved" values.
> > 5. A float is a good guess, but with floats come weird rounding issues. Often times people choose to use an integer storing the value in cents (or fractional cents, to whatever the desired precision is).
> >
> {: .solution}
{: .question}


topic: data-science
tutorial: r-dplyr
title: dplyr & tidyverse for data processing
difficulty: advanced
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["R"]
content_hash: 682245c2388e34995f7b98c649a7ddff
last_modified: 2026-02-26T21:12:14.908852
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-dplyr/tutorial.html

content:
dplyr ({% cite r-dplyr %}) is a powerful R-package to transform and summarize tabular data with rows and columns. It is part of a group of packages (including `ggplot2`) called the `tidyverse` ({% cite r-tidyverse %}), a collection of packages for data processing and visualisation. For further exploration please see the dplyr package vignette: [Introduction to dplyr](https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html)

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [GenomicsClass/labs](https://github.com/genomicsclass/labs/blob/master/intro/dplyr_tutorial.Rmd).
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Why Is It Useful?

The package contains a set of functions (or "verbs") that perform common data manipulation operations such as filtering for rows, selecting specific columns, re-ordering rows, adding new columns and summarizing data.

In addition, dplyr contains a useful function to perform another common task which is the "split-apply-combine" concept.  We will discuss that in a little bit.

## How Does It Compare To Using Base Functions R?

If you are familiar with R, you are probably familiar with base R functions such as split(), subset(), apply(), sapply(), lapply(), tapply() and aggregate(). Compared to base functions in R, the functions in dplyr are easier to work with, are more consistent in the syntax and are targeted for data analysis around tibbles, instead of just vectors.

## How Do I Get dplyr?

To load the required packages:

```r
library(tidyverse)
```

> <tip-title>Package not found?</tip-title>
> Remember that you can install new packages by running
> ```
> install.packages("tidyverse")
> ```
> Or by using the Install button on the RStudio Packages interface
{: .tip}

Here we've imported the entire suite of tidyverse packages. We'll specifically be using:

Package    | Use
---        | ---
`readr`    | This provides the `read_csv` function which is identical to `read.csv` except it returns a tibble
`dplyr`    | All of the useful functions we'll be covering are part of dplyr
`magrittr` | A dependency of `dplyr` that provides the `%>%` operator
`ggplot2`  | The famous plotting library which we'll use at the very end to plot our aggregated data.

# Data: Mammals Sleep

The msleep (mammals sleep) data set contains the sleep times and weights for a set of mammals. This data set contains 83 rows and 11 variables.

```r
url <- "https://raw.githubusercontent.com/genomicsclass/dagdata/master/inst/extdata/msleep_ggplot2.csv"
msleep <- read_csv(url)
head(msleep)
```

The columns (in order) correspond to the following:

column name  | Description
---          | ---
name         | common name
genus        | taxonomic rank
vore         | carnivore, omnivore or herbivore?
order        | taxonomic rank
conservation | the conservation status of the mammal
sleep\_total | total amount of sleep, in hours
sleep\_rem   | rem sleep, in hours
sleep\_cycle | length of sleep cycle, in hours
awake        | amount of time spent awake, in hours
brainwt      | brain weight in kilograms
bodywt       | body weight in kilograms

Compare the above output with the more traditional `read.csv` that is built into R

```r
dfmsleep <- read.csv(url)
head(dfmsleep)
```

This is a "data frame" and was the basis of data processing for years in R, and is still quite commonly used! But notice how `dplyr` has a much prettier and more consice output. This is what is called a `tibble` (like a table). We can immediately see metadata about the table, the separator that was guessed for us, what datatypes each column was (dbl or chr), how many rows and columns we have, etc. The `tibble` works basically exactly like a data frame except it has a lot of features to integrate nicely with the `dplyr` package.


*That said*, all of the functions below you will learn about work equally well with data frames and tibbles, but tibbles will save you from filling your screen with hundreds of rows by automatically truncating large tables unless you specifically request otherwise.

# Important dplyr Verbs To Remember

dplyr verbs | Description | SQL Equivalent Operation
--- | --- | ---
`select()` | select columns  | `SELECT`
`filter()` | filter rows | `WHERE`
`arrange()` | re-order or arrange rows | `ORDER BY`
`mutate()` | create new columns | `SELECT x, x*2 ...`
`summarise()` | summarise values | n/a
`group_by()` | allows for group operations in the "split-apply-combine" concept | `GROUP BY`


# dplyr Verbs In Action

The two most basic functions are `select()` and `filter()`, which selects columns and filters rows respectively.

## Pipe Operator: %>%

Before we go any further, let's introduce the pipe operator: %>%. dplyr imports this operator from another package (magrittr).This operator allows you to pipe the output from one function to the input of another function. Instead of nesting functions (reading from the inside to the outside), the idea of piping is to read the functions from left to right. This is a lot more like how you would write a `bash` data processing pipeline and can be a lot more readable and intuitive than the nested version.

Here's is the more old fashioned way of writing the equivalent code:

```r
head(select(msleep, name, sleep_total))
```

Now in this case, we will pipe the msleep tibble to the function that will select two columns (name and sleep\_total) and then pipe the new tibble to the function `head()`, which will return the head of the new tibble.

```r
msleep %>% select(name, sleep_total) %>% head(2)
```

> <question-title></question-title>
> How would you rewrite the following code to use the pipe operator?
> ```
> prcomp(tail(read.csv("file.csv"), 10))
> ```
> > <solution-title></solution-title>
> > Just read from inside to outside, starting with the innermost `()` and use `%>%` between each step.
> > ```
> > read.csv("file.csv") %>% tail(10) %>% prcomp()
> > ```
> {: .solution}
{: .question}

## Selecting Columns Using `select()`

Select a set of columns: the name and the sleep\_total columns.

```r
msleep %>% select(name, sleep_total)
```

To select all the columns *except* a specific column, use the "-" (subtraction) operator (also known as negative indexing):

```r
msleep %>% select(-name)
```

To select a range of columns by name, use the ":" (colon) operator:

```r
msleep %>% select(name:order)
```

To select all columns that start with the character string "sl", use the function `starts_with()`:

```r
msleep %>% select(starts_with("sl"))
```

Some additional options to select columns based on a specific criteria include:

Function      | Usage
--------      | -----
`ends_with()` | Select columns that end with a character string
`contains()`  | Select columns that contain a character string
`matches()`   | Select columns that match a regular expression
`one_of()`    | Select column names that are from a group of names


## Selecting Rows Using `filter()`

Filter the rows for mammals that sleep a total of more than 16 hours.

```r
msleep %>% filter(sleep_total >= 16)
```

Filter the rows for mammals that sleep a total of more than 16 hours *and* have a body weight of greater than 1 kilogram.

```r
msleep %>% filter(sleep_total >= 16, bodywt >= 1)
```

Filter the rows for mammals in the Perissodactyla and Primates taxonomic order

```r
msleep %>% filter(order %in% c("Perissodactyla", "Primates"))
```

You can use the boolean operators (e.g. >, <, >=, <=, !=, %in%) to create the logical tests.

## Arrange Or Re-order Rows Using `arrange()`

To arrange (or re-order) rows by a particular column, such as the taxonomic order, list the name of the column you want to arrange the rows by:

```r
msleep %>% arrange(order) %>% select(order, genus, name)
```

Now we will select three columns from msleep, arrange the rows by the taxonomic order and then arrange the rows by sleep\_total. Finally, show the final tibble:

```r
msleep %>%
    select(name, order, sleep_total) %>%
    arrange(order, sleep_total)
```

Same as above, except here we filter the rows for mammals that sleep for 16 or more hours, instead of showing the whole tibble:

```r
msleep %>%
    select(name, order, sleep_total) %>%
    arrange(order, sleep_total) %>%
    filter(sleep_total >= 16)
```

Something slightly more complicated: same as above, except arrange the rows in the sleep\_total column in a descending order. For this, use the function `desc()`

```r
msleep %>%
    select(name, order, sleep_total) %>%
    arrange(order, desc(sleep_total)) %>%
    filter(sleep_total >= 16)
```


## Create New Columns Using `mutate()`

The `mutate()` function will add new columns to the tibble. Create a new column called rem_proportion, which is the ratio of rem sleep to total amount of sleep.


```r
msleep %>%
  mutate(rem_proportion = sleep_rem / sleep_total) %>%
  select(starts_with("sl"), rem_proportion)
```

You can many new columns using mutate (separated by commas). Here we add a second column called bodywt_grams which is the bodywt column in grams.

```r
msleep %>%
    mutate(rem_proportion = sleep_rem / sleep_total,
           bodywt_grams = bodywt * 1000) %>%
    select(sleep_total, sleep_rem, rem_proportion, bodywt, bodywt_grams)
```

## Create summaries of the tibble using `summarise()`

The `summarise()` function will create summary statistics for a given column in the tibble such as finding the mean. For example, to compute the average number of hours of sleep, apply the `mean()` function to the column sleep\_total and call the summary value avg\_sleep.

```r
msleep %>%
    summarise(avg_sleep = mean(sleep_total))
```

There are many other summary statistics you could consider such `sd()`, `min()`, `max()`, `median()`, `sum()`, `n()` (returns the length of vector), `first()` (returns first value in vector), `last()` (returns last value in vector) and `n_distinct()` (number of distinct values in vector).

```r
msleep %>%
    summarise(avg_sleep = mean(sleep_total),
              min_sleep = min(sleep_total),
              max_sleep = max(sleep_total),
              total = n())
```


## Group operations using `group_by()`

The `group_by()` verb is an important function in dplyr. As we mentioned before it's related to concept of "split-apply-combine". We literally want to split the tibble by some variable (e.g. taxonomic order), apply a function to the individual tibbles and then combine the output.

Let's do that: split the msleep tibble by the taxonomic order, then ask for the same summary statistics as above. We expect a set of summary statistics for each taxonomic order.

```r
msleep %>%
    group_by(order) %>%
    summarise(avg_sleep = mean(sleep_total),
              min_sleep = min(sleep_total),
              max_sleep = max(sleep_total),
              total = n())
```

## ggplot2

Most people want to slice and dice their data before plotting, so let's demonstrate that quickly by plotting our last dataset.

```r
library(ggplot2)
msleep %>%
    group_by(order) %>%
    summarise(avg_sleep = mean(sleep_total),
              min_sleep = min(sleep_total),
              max_sleep = max(sleep_total),
              total = n()) %>%
    ggplot() + geom_point(aes(x=min_sleep, y=max_sleep, colour=order))
```

Notice how we can just keep piping our data together, this makes it incredibly easier to experiment and play around with our data and test out what filtering or summarisation we want and how that will plot in the end. If we wanted, or if the data processing is an especially computationally expensive step, we could save it to an intermediate variable before playing around with plotting options, but in the case of this small dataset that's probably not necessary.


topic: data-science
tutorial: python-argparse
title: Python - Argparse
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 6555e5fb9d113dceed297aa1d9ea4fd5
last_modified: 2026-02-26T21:12:14.899976
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-argparse/tutorial.html

content:
[`argparse`](https://docs.python.org/3/library/argparse.html) is an argument parsing library for Python that's part of the stdlib. It lets you make command line tools significantly nicer to work with.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Unlike previous modules, this lesson won't use a Jupyter/CoCalc notebook, and that's because we'll be parsing command lines! You'll need to open a code editor on your platform of choice (`nano`, `vim`, `emacs`, VSCode are all options) and use the following blocks of code to construct your command line tool.

## `sys.argv`

In the coding world, whenever you run a Python script on the command line, it has a special variable available to it named `argv`. This is a list of all of the arguments used when you run a command line program.

> <hands-on-title>Print out argv</hands-on-title>
>
> 1. Create / open the file `run.py` in your text editor of choice
> 2. There we'll create a simple Python script that:
>   1. imports `sys`, the system module needed to access argv.
>   2. Prints out `sys.argv`
>
> > <solution-title></solution-title>
> > ```python
> > import sys
> >
> > print(sys.argv)
> > ```
> {: .solution}
>
> 3. Run this with different command line arguments:
>
>    ```
>    python run.py
>    python run.py 1 2 3 4
>    python run.py --help
>    ```
>
{: .hands_on}

> <question-title></question-title>
> What did you notice about the output? There are two main points.
> > <solution-title></solution-title>
> > 1. The name of the script (`run.py`) is included as the first value every time.
> > 2. All of the arguments are passed as strings, no numbers.
> {: .solution}
{: .question}

## Simple tasks

Let's sum up all of the numbers passed on the command line. We'll do this by hand, and then we'll replace it with `argparse` to see how much effort that saves us.

> <hands-on-title></hands-on-title>
> Update your script to sum up every number passed to it on the command line.
>
> It should handle:
> - 1 or more numbers
> - nothing (and maybe print out a message?)
> - invalid values (print out an error message that the value couldn't be processed.)
>
> Hints:
> - Skip the program name
> - Use `try` and `except` to try converting the string to a number.
>
> > <question-title></question-title>
> >
> > How does your updated script look?
> >
> > > <solution-title></solution-title>
> > >
> > > ```python
> > > import sys
> > >
> > > result = 0
> > >
> > > if len(sys.argv) == 1:
> > >     print("no arguments were supplied")
> > > else:
> > >     for arg in sys.argv[1:]:
> > >         try:
> > >             result += float(arg)
> > >         except:
> > >             print(f"Could not parse {arg}")
> > >
> > >     print(result)
> > > ```
> > {: .solution}
> {: .question}
{: .hands_on}

## Argparse

Argparse saves us a lot of work, because it can handle a number of things for us!

- Ensures that the correct number of arguments are provided (and provide a nice error message otherwise)
- Ensure that the correct types of arguments are provided (no strings for a number field)
- Provide a help message describing your program

Argparse is used as follows. First we need to import it

```python
import argparse
```

And then we can define a 'parser' which will parse our command line. Additionally we can provide a description field which tells people what our tool does:

```python
parser = argparse.ArgumentParser(description='Process some integers.')
```

And finally we can define some arguments that are available. Just like we have arguments to functions, we have arguments to command lines. These come in two flavours:

- required (without a `--`)
- optional "flags" (prefixed with `--`)

Here we have an argument named 'integers', which validates that all input values are of the type `int`. `nargs` is the number of arguments, `+` means '1 or more'. And we have some help text as well:

```python
parser.add_argument('integer', type=int, help='an integer parameter')
parser.add_argument('many_integers', type=int, nargs='+', help='an integer parameter')
```

We can also define an optional flag, here it's called `--sum`. We use `store_true` which will set it as true if the flag is used , otherwise false.

```python
parser.add_argument('--sum', action='store_true', help='Should we sum up the integers?')
```

Finally we parse the arguments, which reads `sys.argv` and processes it according to the above rules. The output is stored in `args`.

```python
args = parser.parse_args()
```

We have two main variables we can use now:

```
args.integer # A single integer
args.many_integers # A list of ints
args.sum # A boolean, True or False.
```

## Using argparse

Let's go back to our script, and replace `sys` with argparse.

> <hands-on-title>Replacing argv.</hands-on-title>
>
> 1. Given the following script, replace the use of `argv` with argparse.
>
>    ```python
>    import sys
>
>    result = 0
>
>    if len(sys.argv) == 1:
>        print("no arguments were supplied")
>    else:
>        for arg in sys.argv[1:]:
>            try:
>                result += float(arg)
>            except:
>                print(f"Could not parse {arg}")
>
>        print(result)
>    ```
>    {: .hands_on}
>
>    You should have one argument: numbers (type=float)
>
>    And print out the sum of those numbers.
>
>    > <question-title></question-title>
>    > How does your final script look?
>    > > <solution-title></solution-title>
>    > > ```python
>    > > import argparse
>    > >
>    > > parser = argparse.ArgumentParser(description='Sum some numbers')
>    > > parser.add_argument('integers', type=float, nargs='+',
>    > >                     help='a number to sum up.')
>    > > args = parser.parse_args()
>    > >
>    > > print(sum(args.integers))
>    > > ```
>    > {: .solution}
>    {: .question}
>
> 2. Try running the script with various values
>
>    ```bash
>    python run.py
>    python run.py 1 3 5
>    python run.py 2 4 O
>    python run.py --help
>    ```
{: .hands_on}

Wow that's a lot simpler! We have to learn how `argparse` is invoked but it handles a lot of cases for us:

- No arguments provided
- Responding to `--help`
- Raising an error for invalid values

`--help` is even written for us, without us writing any special code to handle that case! This is why you need to use `argparse`:
- It handles a lot of cases and input validation for you
- It produces a nice `--help` text that can help you if you've forgotten what your tool does
- It's nice for users of your scripts! They don't have to read the code to know how it behaves if you document it well.

There is a lot of documentation in the [`argparse`](https://docs.python.org/3/library/argparse.html) module for all sorts of use cases!


## Why Argparse?

Using argparse can be a big change to your tool but there are some benefits to using it!

1. Standardised interface to your tool that's familiar to everyone who uses command line tools
2. Automatic Help page
3. Automatic Galaxy Tools?

### Generating Automatic Galaxy Tools (Optional)

With the `argparse2tool` project, and eventually `pyGalGen` which will be merged into `planemo`, you can generate Galaxy tools automatically from `argparse` based Python scripts.

> <hands-on-title>Generate a Galaxy tool wrapper from your script</hands-on-title>
> 1. Write out the python script to a file named `main.py`
>
>    ```python
>    import argparse
>
>    parser = argparse.ArgumentParser(description='Sum some numbers')
>    parser.add_argument('integers', type=float, nargs='+',
>                        help='a number to sum up.')
>    args = parser.parse_args()
>
>    print(sum(args.integers))
>    ```
>
> 2. Create a virtual environment, just in case: ``
>
>    ```console
>    python -m venv .venv
>    . .venv/bin/activate
>    ```
>
> 3. Install `argparse2tool` via pip:
>
>    ```console
>    pip install argparse2tool
>    ```
>
> 4. Generate the tool interface:
>
>    > <code-in-title>Command</code-in-title>
>    > ```
>    > PYTHONPATH=$(argparse2tool) python main.py --generate_galaxy_xml
>    > ```
>    {: .code-in }
>
>    > <code-out-title>Galaxy XML</code-out-title>
>    > ```xml
>    > <tool name="main.py" id="main.py" version="1.0">
>    >   <description>Sum some numbers</description>
>    >   <stdio>
>    >     <exit_code range="1:" level="fatal"/>
>    >   </stdio>
>    >   <version_command><![CDATA[python main.py --version]]></version_command>
>    >   <command><![CDATA[python main.py
>    > #set repeat_var_1 = '" "'.join([ str($var.integers) for $var in $repeat_1 ])
>    > "$repeat_var_1"
>    >
>    > > $default]]></command>
>    >   <inputs>
>    >     <repeat title="repeat_title" min="1" name="repeat_1">
>    >       <param label="a number to sum up." value="0" type="float" name="integers"/>
>    >     </repeat>
>    >   </inputs>
>    >   <outputs>
>    >     <data name="default" format="txt" hidden="false"/>
>    >   </outputs>
>    >   <help><![CDATA[TODO: Write help]]></help>
>    > </tool>
>    >
>    > ```
>    {: .code-out }
>
{: .hands_on}


topic: data-science
tutorial: python-linting
title: Python - Coding Style
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 6adc9c07919dd619c0cb343b78be14f7
last_modified: 2026-02-26T21:12:14.902946
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-linting/tutorial.html

content:
But before you dive into writing some more code and
sharing it with others, ask yourself what kind of code should you be writing and publishing? It may be
worth spending some time learning a bit about Python
coding style conventions to make sure that your code is consistently formatted and readable by yourself and others.

> *"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."* - [Martin Fowler](https://en.wikiquote.org/wiki/Martin_Fowler), British software engineer, author and international speaker on software development
{: .quote}

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Python Coding Style Guide

One of the most important things we can do to make sure our code is readable by others
(and ourselves a
few months down the line) is to make sure that it is descriptive, cleanly and consistently formatted and uses sensible,
descriptive names for variable, function and module names. In order to help us format our code, we generally follow
guidelines known as a **style guide**. A style guide is a set of conventions that we agree upon with our colleagues or
community, to ensure that everyone contributing to the same project is producing code which looks similar in style.
While a group of developers may choose to write and agree upon a new style guide unique to each project,
in practice many programming languages have a single style guide which is
adopted almost universally by the communities around the world. In Python, although we do have a choice of style guides
available, the [PEP8](https://www.python.org/dev/peps/pep-0008/) style guide is most commonly used.
A {PEP} is a design document for the Python community, typically
specifications or conventions for how to do something in Python, a description of a new feature in Python, etc.

> <tip-title>Style consistency</tip-title>
> One of the
> [key insights from Guido van Rossum](https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds),
> one of the PEP8 authors, is that code is read much more often than it is
> written. Style guidelines are intended to improve the readability of code and make it consistent across the
> wide spectrum of Python code. Consistency with the style guide is important. Consistency within a project is more
> important. Consistency within one module or function is the most important. However, know when to be inconsistent --
> sometimes style guide recommendations are just not applicable. When in doubt, use your best judgment.
> Look at other examples and decide what looks best. And don't hesitate to ask!
>
{: .tip}

A full list of style guidelines for this style 
is available from the [PEP8 website](https://www.python.org/dev/peps/pep-0008/); here we highlight a few.

### Indentation

Python is a kind of language that uses indentation as a way of grouping statements that belong to a particular
block of code. Spaces are the recommended indentation method in Python code. The guideline is to use 4 spaces per indentation level -
so 4 spaces on level one, 8 spaces on level two and so on.

Many people prefer the use of tabs to spaces to indent the code for many reasons (e.g. additional typing, easy to
introduce an error by missing a single space character, etc.) and do not follow this guideline. Whether you decide to
follow this guideline or not, be consistent and follow the style already used in the project.

> <tip-title>Accessibility</tip-title>
> For many users of Braille Refreshable Displays, encoding the indentation with a tab character is easier to read and more efficient. Rather than 8 space characters, they would only see 2 tab characters which take up less space and are easier to comprehend the indentation level.
{: .tip}

> <tip-title>Indentation in Python 2 vs Python 3</tip-title>
> Python 2 allowed code
> indented with a mixture of tabs and spaces. Python 3 disallows mixing the use of tabs and spaces for indentation.
> Whichever you choose, be consistent throughout the project.
{: .tip}

> <tip-title>Editor support</tip-title>
> Many IDEs and editors have built in support for automatically managing tabs and spaces and auto-converting them back and forth.
{: .tip}

There are more complex rules on indenting single units of code that continue over several lines, e.g. function,
list or dictionary definitions can all take more than one line. The preferred way of wrapping such long lines is by
using Python's implied line continuation inside delimiters such as parentheses (`()`), brackets (`[]`) and braces
(`{}`), or a hanging indent.

```python
# Add an extra level of indentation (extra 4 spaces) to distinguish arguments from the rest of the code that follows
def long_function_name(
        var_one, var_two, var_three,
        var_four):
    print(var_one)


# Aligned with opening delimiter
foo = long_function_name(var_one, var_two,
                         var_three, var_four)

# Use hanging indents to add an indentation level like paragraphs of text where all the lines in a paragraph are
# indented except the first one
foo = long_function_name(
    var_one, var_two,
    var_three, var_four)

# Using hanging indent again, but closing bracket aligned with the first non-blank character of the previous line
a_long_list = [
    [[1, 2, 3], [4, 5, 6], [7, 8, 9]], [[0.33, 0.66, 1], [0.66, 0.83, 1], [0.77, 0.88, 1]]
    ]

# Using hanging indent again, but closing bracket aligned with the start of the multiline contruct
a_long_list2 = [
    1,
    2,
    3,
    # ...
    79
]
```

More details on good and bad practices for continuation lines can be found in
[PEP8 guideline on indentation](https://www.python.org/dev/peps/pep-0008/#indentation).

### Maximum Line Length

All lines should be up to 80 characters long; for lines containing comments or docstrings (to be covered later) the
line length limit should be 73 - see [this discussion](https://www.google.com/url?q=https://stackoverflow.com/questions/15438326/python-pep-8-docstring-line-length&sa=D&source=editors&ust=1619088968027000&usg=AOvVaw3jn26Qt-kwog_tJnaMR48x) for reasoning behind these numbers. Some teams strongly prefer a longer line length, and seemed to have settled on the
length of 100. Long lines of code can be broken over multiple lines by wrapping expressions in delimiters, as
mentioned above (preferred method), or using a backslash (`\`) at the end of the line to indicate
line continuation (slightly less preferred method).

```python
# Using delimiters ( ) to wrap a multi-line expression
if (a == True and
    b == False):

# Using a backslash (\) for line continuation
if a == True and \
    b == False:
```

### Should a Line Break Before or After a Binary Operator?

Lines should break before binary operators so that the operators do not get scattered across different columns
on the screen. In the example below, the eye does not have to do the extra work to tell which items are added
and which are subtracted:

```python
# PEP 8 compliant - easy to match operators with operands
income = (gross_wages
          + taxable_interest
          + (dividends - qualified_dividends)
          - ira_deduction
          - student_loan_interest)
```

### Blank Lines

Top-level function and class definitions should be surrounded with two blank lines. Method definitions inside a class
should be surrounded by a single blank line. You can use blank lines in functions, sparingly, to indicate logical sections.

### Whitespace in Expressions and Statements

Avoid extraneous whitespace in the following situations:

- Immediately inside parentheses, brackets or braces

  ```python
  # PEP 8 compliant:
  my_function(colour[1], {id: 2})

  # Not PEP 8 compliant:
  my_function( colour[ 1 ], { id: 2 } )
  ```

- Immediately before a comma, semicolon, or colon (unless doing slicing where the colon acts like a binary operator
in which case it should should have equal amounts of whitespace on either side)

  ```python
  # PEP 8 compliant:
  if x == 4: print(x, y); x, y = y, x

  # Not PEP 8 compliant:
  if x == 4 : print(x , y); x , y = y, x
  ```

- Immediately before the open parenthesis that starts the argument list of a function call

  ```python
  # PEP 8 compliant:
  my_function(1)

  # Not PEP 8 compliant:
  my_function (1)
  ```

- Immediately before the open parenthesis that starts an indexing or slicing

  ```python
  # PEP 8 compliant:
  my_dct['key'] = my_lst[id]
  first_char = my_str[:, 1]

  # Not PEP 8 compliant:
  my_dct ['key'] = my_lst [id]
  first_char = my_str [:, 1]
  ```

- More than one space around an assignment (or other) operator to align it with another

  ```python
  # PEP 8 compliant:
  x = 1
  y = 2
  student_loan_interest = 3

  # Not PEP 8 compliant:
  x                     = 1
  y                     = 2
  student_loan_interest = 3
  ```

- Avoid trailing whitespace anywhere - it is not necessary and can cause errors. For example, if you use
backslash (`\`) for continuation lines and have a space after it, the continuation line will not be
interpreted correctly.
- Surround these binary operators with a single space on either side: assignment (=),
augmented assignment (+=, -= etc.), comparisons (==, <, >, !=, <>, <=, >=, in, not in, is, is not),
booleans (and, or, not).
- Don't use spaces around the = sign when used to indicate a keyword argument assignment or to indicate a
default value for an unannotated function parameter

  ```python
  # PEP 8 compliant use of spaces around = for variable assignment
  axis = 'x'
  angle = 90
  size = 450
  name = 'my_graph'

  # PEP 8 compliant use of no spaces around = for keyword argument assignment in a function call
  my_function(
      1,
      2,
      axis=axis,
      angle=angle,
      size=size,
      name=name)
  ```

### String Quotes

In Python, single-quoted strings and double-quoted strings are the same. PEP8 does not make a recommendation for this
apart from picking one rule and consistently sticking to it. When a string contains single or double quote characters,
use the other one to avoid backslashes in the string as it improves readability.

### Naming Conventions

There are a lot of different naming styles in use, including:
- lowercase
- lower_case_with_underscores
- UPPERCASE
- UPPER_CASE_WITH_UNDERSCORES
- CapitalisedWords (or PascalCase) (note: when using acronyms in CapitalisedWords, capitalise all the letters of the acronym,
e.g HTTPServerError)
- camelCase (differs from CapitalisedWords/PascalCase by the initial lowercase character)
- Capitalised_Words_With_Underscores

As with other style guide recommendations - consistency is key. Pick one and stick to it, or follow the one already
established if joining a project mid-way. Some things to be wary of when naming things in the code:

- Avoid using the characters 'l' (lowercase letter L), 'O' (uppercase letter o), or 'I' (uppercase letter i)
  as single character variable names. In some fonts, these characters are indistinguishable from the numerals
  one and zero. When tempted to use 'l', use 'L' instead.
- If your audience is international and English is the common language, try to use English words for identifiers and 
  comments whenever possible but try to avoid abbreviations/local slang as they may not be understood by everyone. Also consider
  sticking with either ‘American’ or 'British' English spellings and try not to mix the two.

Stronlgy consider **not** using single character identifiers wherever possible. Use descriptive variable names when possible, it is no slower to execute, and a lot faster to read and comprehend for a reader.

> <tip-title>Function, Variable, Class, Module, Package Naming</tip-title>
>
> - Function and variable names should be lowercase, with words separated by underscores as necessary to improve readability.
> - Class names should normally use the CapitalisedWords convention.
> - Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability.
> - Packages should also have short, all-lowercase names, although the use of underscores is discouraged.
>
> A more detailed guide on
> [naming functions, modules, classes and variables](https://www.python.org/dev/peps/pep-0008/#package-and-module-names)
> is available from PEP8.
>
{: .tip}

### Comments

Comments allow us to provide the reader with additional information on what the code does - reading and understanding
source code is slow, laborious and can lead to misinterpretation, plus it is always a good idea to keep others in mind
when writing code. A good rule of thumb is to assume that someone will *always* read your code at a later date,
and this includes a future version of yourself. It can be easy to forget why you did something a particular way in six
months' time. Write comments as complete sentences and in English unless you are 100% sure the code will never be read
by people who don't speak your language.

> <tip-title>The Good, the Bad, and the Ugly Comments</tip-title>
> As a side reading, check out the ['Putting comments in code: the good, the bad, and the ugly' blogpost](https://www.freecodecamp.org/news/code-comments-the-good-the-bad-and-the-ugly-be9cc65fbf83/).
> Remember - a comment should answer the 'why' question. Occasionally the "what" question.
> The "how" question should be answered by the code itself.
{: .tip}

Block comments generally apply to some (or all) code that follows them, and are indented to the same level as that
code. Each line of a block comment starts with a `#` and a single space (unless it is indented text inside the comment).

```python
def fahr_to_cels(fahr):
    # Block comment example: convert temperature in Fahrenheit to Celsius
    cels = (fahr + 32) * (5 / 9)
    return cels
```

An inline comment is a comment on the same line as a statement. Inline comments should be separated by at least two
spaces from the statement. They should start with a `#` and a single space and should be used sparingly.

```python
def fahr_to_cels(fahr):
    cels = (fahr + 32) * (5 / 9)  # Inline comment example: convert temperature in Fahrenheit to Celsius
    return cels
```

Python doesn't have any multi-line comments, like you may have seen in other languages like C++ or Java. However, there
 are ways to do it using *docstrings* as we'll see in a moment.

The reader should be able to understand a single function or method from its code and its comments, and should not have to look elsewhere in the code for clarification. The kind of things that need to be commented are:

- Why certain design or implementation decisions were adopted, especially in cases where the decision may seem counter-intuitive
- The names of any algorithms or design patterns that have been implemented
- The expected format of input files or database schemas

However, there are some restrictions. Comments that simply restate what the code does are redundant, and comments must be
 accurate and updated with the code, because an incorrect comment causes more confusion than no comment at all.


## Exercise Improving Indentation

Now given all of the above rules we'll go ahread and re-format some code to match the style guide. Please review them carefully and prepare yourself for an hour of editing text… no, of course not. 

There is no reason to do this manually, we're learning about computers and reformatting text is something computers are great at! There is a project called [Black](https://pypi.org/project/black/) which automates this process, and removes the potential arguing over coding style, which is truly it's greatest success. 

> <hands-on-title>Reformatting Code with Black</hands-on-title>
> 1. Install black
>    
>    ```bash
>    pip3 install black
>    ```
>    
> 2. Run Black
>  
>    ```bash
>    black .
>    ```
>  
> 3. Done. That's it.
{: .hands_on}

> <hands-on-title>Optional Exercise: Improve Code Style of Your Other Python Projects</hands-on-title>
> If you have another Python project, check to which extent it conforms to PEP8 coding style.
{: .hands_on}

## Documentation Strings aka Docstrings

If the first thing in a function is a string that is not assigned to a variable, that string is attached to the
function as its documentation. Consider the following code implementing function for calculating the nth
Fibonacci number:

```python
def fibonacci(n):
    """Calculate the nth Fibonacci number.

    A recursive implementation of Fibonacci array elements.

    :param n: integer
    :raises ValueError: raised if n is less than zero
    :returns: Fibonacci number
    """
    if n < 0:
        raise ValueError('Fibonacci is not defined for N < 0')
    if n == 0:
        return 0
    if n == 1:
        return 1

    return fibonacci(n - 1) + fibonacci(n - 2)
```

Note here we are explicitly documenting our input variables, what is returned by the function, and also when the
`ValueError` exception is raised. Along with a helpful description of what the function does, this information can
act as a *contract* for readers to understand what to expect in terms of behaviour when using the function,
as well as how to use it.

A special comment string like this is called a **docstring**. We do not need to use triple quotes when writing one, but
if we do, we can break the text across multiple lines. Docstrings can also be used at the start of a Python module (a file
containing a number of Python functions) or at the start of a Python class (containing a number of methods) to list
their contents as a reference. You should not confuse docstrings with comments though - docstrings are context-dependent and should only
be used in specific locations (e.g. at the top of a module and immediately after `class` and `def` keywords as mentioned).
Using triple quoted strings in locations where they will not be interpreted as docstrings or
using triple quotes as a way to 'quickly' comment out an entire block of code is considered bad practice.

In our example case, we used
the [Sphynx/ReadTheDocs docstring style](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html) formatting
for the `param`, `raises` and `returns` - other docstring formats exist as well.

> <tip-title>Python PEP 257 - Recommendations for Docstrings</tip-title>
> PEP 257 is another one of Python Enhancement Proposals and this one deals with docstring conventions to
>standardise how they are used. For example, on the subject of module-level docstrings, PEP 257 says:
>
> > The docstring for a module should generally list the classes, exceptions and functions (and any other objects) that
> > are exported by the module, with a one-line summary of each. (These summaries generally give less detail than the
> > summary line in the object's docstring.) The docstring for a package
> > (i.e., the docstring of the package's `__init__.py` module) should also list the modules and subpackages exported by
> > the package.
> {: .quote}
>
> Note that `__init__.py` file used to be a required part of a package (pre Python 3.3) where a package was typically
>implemented as a directory containing an `__init__.py` file which got implicitly executed when a package was imported.
{: .tip}

So, at the beginning of a module file we can just add a docstring explaining the nature of a module. For example, if
`fibonacci()` was included in a module with other functions, our module could have at the start of it:

```python
"""A module for generating numerical sequences of numbers that occur in nature.

Functions:
  fibonacci - returns the Fibonacci number for a given integer
  golden_ratio - returns the golden ratio number to a given Fibonacci iteration
  ...
"""
...
```

The docstring for a function or a module is returned when
calling the `help` function and passing its name - for example from the interactive Python console/terminal available
from the command line or when rendering code documentation online 
(e.g. see [Python documentation](https://docs.python.org/3.8/library/index.html)).
PyCharm also displays the docstring for a function/module in a little help popup window when using tab-completion.

```python
help(fibonacci)
```


topic: data-science
tutorial: python-flow
title: Python - Flow Control
difficulty: introductory
time_estimation: 40M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 5827d030ca4903c65dc2861dd1c00f23
last_modified: 2026-02-26T21:12:14.901628
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-flow/tutorial.html

content:
"Flow Control" is how we describe when we change the flow of code's execution, based on some conditions. Here we'll learn how to take different actions depending on what data out program sees, or how to run code only if some condition is true.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which are licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Comparators

In Python we have the following comparators to do compare two values

- `>`: greater than
- `<`: less than
- `==`: equal to
- `!=`: does not equal
- `>=`: greater than or equal to
- `<=`: less than or equal to

They're all "binary" comparators, we can only compare two values at a time.

```python
print(37 < 38)
print(38 < 38)
print(39 < 38)
```

These print out `True` or `False`, these are the two possible values of the [*boolean*](https://en.wikipedia.org/wiki/Boolean_algebra) datatype in Python.

We can use `<=` to check if it's less than or equal to:

```python
print(19 <= 20)
print(20 <= 20)
print(21 <= 20)
```

And we can use `==` for comparing numbers in Python

```
print(11 == 11)
print(11 != 11)
print(22 != 33)
```

And now that we can compare numbers, we can start doing useful things with them!

## Conditionals

We can ask Python to take different actions, depending on a condition, with an `if` statement:

```python
num = 37
if num > 100:
    print('greater')
else:
    print('not greater')
print('done')
```

The second line of this code uses the keyword `if` to tell Python that we want to make a choice.
If the test that follows the `if` statement is true,
the body of the `if`
(i.e., the set of lines indented underneath it) is executed, and "greater" is printed.
If the test is false,
the body of the `else` is executed instead, and "not greater" is printed.
Only one or the other is ever executed before continuing on with program execution to print "done":

![A flowchart diagram of the if-else construct that tests if variable num is greater than 100](../../images/python-flowchart-conditional.png)

Conditional statements don't have to include an `else`. If there isn't one,
Python simply does nothing if the test is false:

```python
num = 53
print('before conditional...')
if num > 100:
    print(f'{num} is greater than 100')
print('...after conditional')
```

> <question-title>If behaviour</question-title>
>
> Try changing the `num` value and see what happens for different values.
>
> What happens if `num` is a:
> 1. 202
> 2. 3.145
> 3. "test"
> 4. 100.000001
>
> > <solution-title></solution-title>
> > 1. Condition is activated!
> > 2. Nothing, but not because it is a float! Because it's less than 100
> > 3. Traceback, a `TypeError`, you cannot compare strings with integers
> > 4. Condition is activated!
> {: .solution}
{: .question}

## Multiple Branches

But what if you want more branches? What if you need to handle more cases? `elif` to the rescue!

We can chain several tests together using `elif`, which is short for "else if".

```
if todays_temperature > 30:
    print("Wear shorts! Remember your sunscreen")
elif todays_temperature > 20:
    print("It's nice weather finally! Gasp!")
elif todays_temperature < 10:
    print("Time to bundle up!")
else:
    print("Dress normally")
```

> <tip-title>If/Elif/Elif/Elif/Else:</tip-title>
> if/elif/else cases follow these rules:
>
> - must start with an `if`
> - can have 0 or more `elif` conditions
> - can have 0 or 1 `else` condition (if no else condition is supplied, it's equivalent to `else: <nothing>`)
{: .tip}

Each of these three sections is a **branch**, the code pauses, and chooses to go down one of the branches based on the conditions.

The following Python code uses `elif` to print the sign of a number.

```python
num = -3

if num > 0:
    print(f'{num} is positive')
elif num == 0:
    print(f'{num} is zero')
else:
    print(f'{num} is negative')
```

**NB**: To test for equality we use a double equals sign `==`
rather than a single equals sign `=` which is used to assign values.

## Combining Tests

We can also combine tests using `and` and `or`.
`and` is only true if both parts are true:

```python
a = 1
b = -1
if (a > 0) and (b <= 0):
    print('both parts are true')
else:
    print('at least one part is false')
```

> <question-title>Predict what happens</question-title>
> Predict the outcomes of the following values of `a` and `b` above. Predicting what you think the code will do is a useful skill to practice
> 1. a = 0; b = -1
> 2. a = 0; b = 10
> 3. a = 4; b = -22
> 4. a = 99; b = 99
>
> > <solution-title></solution-title>
> > 1. at least one part is false
> > 2. at least one part is false
> > 3. both parts are true
> > 4. at least one part is false
> {: .solution}
{: .question}


while `or` is true if at least one part is true:

```python
a = 1
b = -1
if (a < 0) or (b > 0):
    print('at least one test is true')
```


> <tip-title>`True` and `False`</tip-title>
> `True` and `False` are special words in Python called `booleans`,
> which represent truth values. A statement such as `1 < 0` returns
> the value `False`, while `-1 < 0` returns the value `True`.
{: .tip}

`True` and `False` booleans are not the only values in Python that are true and false.
In fact, *any* value can be used in an `if` or `elif`. This is commonly used to
check, for instance, if a string is empty or if some data is provided:

```python
if '':
    print('empty string is true')
if 'word':
    print('word is true')
```

You can also use it to check if a list is empty or full:

```python
if []:
    print('empty list is true')
if [1, 2, 3]:
    print('non-empty list is true')
# The last statement is equivalent to:
if len([1, 2, 3]) > 0:
    print('non-empty list is true')
```

Or you can check if a number is zero, or non-zero:

```python
if 0:
    print('zero is true')
if 1:
    print('one is true')
```

## Inverting Conditions

Sometimes it is useful to check whether some condition is not true.
The Boolean operator `not` can do this explicitly.
After reading and running the code below,
write some `if` statements that use `not` to test the rule
that you formulated in the previous question.
`not` is a `unary` (not `binary`) operator: it only takes a single value

```python
if not '':
    print('empty string is not true')
if not 'word':
    print('word is not true')
if not not True:
    print('not not True is true')
```

## Ranges

Python makes it super easy to check if a number is within a range.

```python
quality_score = 32 # Try out different values!

if quality_score > 40:
    print("Your data is a bit sus")
elif 20 < quality_score <= 40:
    print("Hey that looks ok")
elif 4 < quality_score <= 20:
    print("Oh you did nanopore sequencing")
else:
    print("It shouldn't be *that* bad. Try again.")
```

There are two important points here:

- `20 < x < 40` is equivalent to `20 < x and x < 40`, checking both sides of the condition, to make sure it's greater than one value and smaller than another
- Note that we checked in the second case `20 < x` and then in the third we had to check `x <= 20`. If we had not had a `<=` on one side, what would have happened to 20? It would have gone straight to else!


## Exercises

> <question-title></question-title>
> `if`s, `elif`s and `else`s get evaluated in blocks. Look at the following code and list the lines that are part of a single block.
>
> ```
> 1.  if x:
> 2.      # ..
> 3.  if y:
> 4.      # ..
> 5.  elif z:
> 6.      # ..
> 7.  if q:
> 8.      # ..
> 9.  else:
> 10.     # ..
> 11. elif t:
> 12.     # ..
> 13. else e:
> 14.     # ..
> ```
>
> > <solution-title></solution-title>
> > "Blocks" of if/elif/elses
> > - must start with an `if`
> > - can have 0 or more `elif` conditions
> > - can have 0 or 1 `else` condition (if no else condition is supplied, it's equivalent to `else: <nothing>`)
> >
> > The above blocks are parsed together, you could not insert a `print` anywhere within the blocks, but between the blocks it would work.
> >
> > - 1-2, Just an `if` by itself. There's no elif, or else, so that's the end of that block
> > - 3-6, `if` and `elif` get evaluated, there is no `else`, so that's the end of that block
> > - 7-10, `if` and `else` is fine
> > - 11-14, error! This is missing an `if` case, it will fail with a syntaxerror.
> {: .solution}
{: .question}

```python
# Test code here.
```

> <question-title>How Many Paths?</question-title>
>
> Consider this code:
>
> ```
> if 4 > 5:
>     print('A')
> elif 4 == 5:
>     print('B')
> elif 4 < 5:
>     print('C')
> ```
>
> Which of the following would be printed if you were to run this code?
> Why did you pick this answer?
>
> 1.  A
> 2.  B
> 3.  C
> 4.  B and C
>
> > <solution-title></solution-title>
> > C gets printed because the first two conditions, `4 > 5` and `4 == 5`, are not true,
> > but `4 < 5` is true.
> {: .solution}
{: .question}

```python
# Test code here.
```

> <question-title>Close Enough</question-title>
>
> Write some conditions that print `True` if the variable `a` is within `10` of the variable `b`
> and `False` otherwise.
> Compare your implementation with your partner's:
> do you get the same answer for all possible pairs of numbers?
>
> > <tip-title>abs</tip-title>
> > There is a [built-in function `abs`][abs-function] that returns the absolute value of
> > a number:
> > ```python
> > print(abs(-12))
> > ```
> >
> > ```
> > 12
> > ```
> {: .tip}
>
> > <solution-title>1</solution-title>
> > ```
> > a = 5
> > b = 5.1
> >
> > if abs(a - b) <= 10:
> >     print('True')
> > else:
> >     print('False')
> > ```
> {: .solution}
>
> > <solution-title>2</solution-title>
> > ```
> > print(abs(a - b) <= 10)
> > ```
> >
> > This works because the Booleans `True` and `False`
> > have string representations which can be printed.
> {: .solution}
{: .question}

```python
# Test code here.
```

> <question-title>Pitfalls</question-title>
>
> A *integer* number between 0 and 100 will be provided to this function. Answer these two questions:
>
> - Will it always print something? If not, which value(s) fail?
> - Can you find any numbers the programmer explicitly wanted to handle, that aren't handled as expected?
>
> ```
> num = 42 # Randomly chosen so the code will execute, try changing it around.
> if num > 90:
>     print("great score")
> elif num < 32:
>     print("Very cold")
> elif num >= 86:
>     print("Almost")
> elif num == 86:
>     print("It's exactly this value!")
> elif 32 < num < 58:
>     print("Getting warmer")
> elif 57 < num <= 86:
>     print("Everything else goes here")
> ```
>
> > <solution-title></solution-title>
> > 1. No, it won't. 32 is the only value there that doesn't print anything. You can either do `x < 57` and later `57 <= x` to test the bigger and smaller values, or you can make use `x < 57` and `56 < x`, which have the same results, but **only with integers**. If your code accepted a float, e.g. `56.5`, both of those tests would be true. So `x < 57` and later `57 <= x` is the preferred way to write that.
> > 2. `86` is the most obvious solution to this, the programmer added a check specifically to see if the value was 86, but instead it's caught by the previous case.
> {: .solution}
{: .question}

```python
num = 42 # Randomly chosen so the code will execute, try changing it around.
if num > 90:
    print("great score")
elif num < 32:
    print("Very cold")
elif num >= 86:
    print("Almost")
elif num == 86:
    print("It's exactly this value!")
elif 32 < num < 58:
    print("Getting warmer")
elif 57 < num <= 86:
    print("Everything else goes here")
```

> <tip-title>Why a synthetic example like this?</tip-title>
> Complicated if/elif/else cases are common in code, you need to be able to spot these sort of issues. For example there are large if/else cases in the [Galaxy codebase](https://github.com/galaxyproject/galaxy/blob/9143dd7ca46d150ebfb26febbe187979f682da51/tools/stats/grouping.py#L153-L176), sometimes nested even, and being ale to predict their behaviour is really important to being able to work with the code. [Missing else cases](https://github.com/galaxyproject/galaxy/blob/9143dd7ca46d150ebfb26febbe187979f682da51/tools/stats/grouping.py#L185-L195) are sometimes important, sometimes a bug, sometimes just the code hasn't been implemented yet, which is why we always write good code comments!
{: .tip}


topic: data-science
tutorial: python-exceptions
title: Python - Try & Except
difficulty: introductory
time_estimation: 20M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: e51cafc688c55f15b7be0f146885487d
last_modified: 2026-02-26T21:12:14.901202
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-exceptions/tutorial.html

content:
Try/except are a construct in Python used to catch a potential exception. Sometimes things go wrong in your code! Or in someone else's code in a module. Sometimes some errors might be expected like when you try and read a user supplied file, maybe it isn't available because they've specified the wrong path.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Raise

When you're writing code, sometimes there are errors you might need to handle:

- If you're calculating the mean of a set of numbers, and the list is empty. You don't want to divide by `len(numbers)` and trigger a Zero Division Error.
- If you're opening a file, there maybe a chance that the file is not found.

Using try/excepts allow you to:

- provide friendlier and more useful error messages
- handle expected error cases

For instance, returning to the `mean()` function example, what should `mean([])` return for a hypothetical mean function that calculates the average of a list of numbers. Should it return 0? It probably should return an error. Let's look at some example code:


```python
def mean(numbers):
    return sum(numbers) / len(numbers)

mean([1, 2, 3])
mean([])
```

This raises a `ZeroDivisionError`  but we can make this a more friendly error message by raising our own exception.

```python
def mean(numbers):
    if not numbers:
        raise ValueError("You are calculating the mean of an empty list, which is not possible.")
    return sum(numbers) / len(numbers)

mean([1, 2, 3])
mean([])
```

> <tip-title>Where do ValueError, ZeroDivisionError come from?</tip-title>
> There are loads of different types of exception codes! [The python documentation has a large list](https://docs.python.org/3/library/exceptions.html) of exceptions and some descriptions for when or why those exceptions might be raised.
{: .tip}

Now we get a much more useful error message from the function! Using `raise` is especially important for *library authors*, people writing python modules that we all use. If they provide useful error messages, it helps you as an end user understand what's happening.

## Try & Except

Let's look at how you would handle one of these exceptions, we'll continue with the `mean()` example above.

```python
numbers = []
try:
    result = mean(numbers)
    print(f"Result is {result}")
except ValueError:
    print("We cannot calculate the mean")
```

Here we use `try:` to setup a new block, and this code is **tried**, Python attempts to execute it. Below are one or more `except:` blocks which catch specific errors. Here we have specifically said we know a ValueError can happen, and decided to handle it.

Or for another example, accessing a user supplied file. Oftentimes users will call your program and supply a non-existent, or inacessible file. Here you can use multiple `except` blocks to catch all of those potential errors.

```python
user_supplied_file = 'does-not-exist.txt'
try:
    open(user_supplied_file, 'r')
except FileNotFoundError:
    print(f"The path you supplied ({user_supplied_file}) doesn't exist, please double check it!")
except PermissionError:
    print(f"You supplied a valid file, but it is unreadable. Try changing it's permissions with `chmod +r {user_supplied_file}`")
```

Failing to open a file raises a `FileNotFoundError` which indicates the file isn't available, and `PermissionError` indicates that a file is unreadable. However in practice, sometimes you'll see something like this:

```
# Bad!
try:
    doSomething()
except:
    print("error")
```

This is called a **bare exception**, and will catch any exception, compared with `except ValueError` which only catches value errors. People consider this generally a bad idea, termed *code smell*. (Because it smells (appears) bad!)

## Finally

The last portion of the `try:`/`except:` block is `finally:`, a third block which lets us do cleanup. It's often very nice to your users that if your program fails halfway through, that you cleanup after yourself.

```python
import os

try:
    with open('gene_query.fa', 'w') as handle:
        handle.write(">some fasta sequence we want to search against a database")

    # runQuery('gene_query.fa')

    # But here we have an error! Something goes wrong!
    1 / 0
except:
    # And now our results are invalid.
    print("Something went wrong! Oh no! Check your inputs.")
finally:
    # So we should cleanup this temporary file we created, so it doesn't cause
    # problems or distract the user from the results file.
    # This function will delete a file from your computer:
    os.remove('gene_query.fa')
```

## Fallback

Sometimes we can use `try`/`except` to have a fallback option. Consider the pseudocode below:

```
try:
    runBlast()
except BlastNotAvailable:
    try:
        runBLAT()
    except BLATnotAvailable:
        print("Neither Blast nor BLAT were available.")
```

Sometimes you have a fallback option, some other tool you can use in its place. When that's possible, you can use `try` and `except` to handle those cases and work around potential issues. But this isn't always the case, sometimes you just need to print your error message and stop executing code.


topic: data-science
tutorial: python-functions
title: Python - Functions
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: e7ec47f79b2c90c3322986ccd95c01e4
last_modified: 2026-02-26T21:12:14.901928
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-functions/tutorial.html

content:
Functions are the basic unit of all work in Python! Absolutely everything you do uses functions. Conceptually, functions are super simple. Just like in maths, they take an input, do some transformation, and return an output. As an example, `f(x) = x + 2` is a function that calculates whatever value you request, plus two. But functions are foundational, so, you should understand them well before moving on.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## What is a Function

Functions are a way to re-use some computation you want to do, multiple times. If you don't have a function, you need to re-write the calculation every time, so we use functions to collect those statements and make them easy to re-run. Additionally they let us "parameterise" some computation. Instead of computing the same value every time, we can template it out, we can re-run the computation with new inputs, and see new results.

> > <code-in-title>Math</code-in-title>
> > ```
> > # Define our function
> > f(x) = 3 * x
> > # Compute some value
> > f(3) # is 9
> > ```
> {: .code-in}
>
> > <code-out-title>Python</code-out-title>
> > ```
> > # Define our function
> > def f(x):
> >    return 3 * x
> > # Compute some value
> > f(3)
> > ```
> {: .code-out}
>
{: .code-2col}

We've talked about mathematical functions before, but now we'll talk about more programing-related functions


## Create Functions

Human beings can only keep a few items in working memory at a time. Breaking down larger/more complicated pieces of code in functions helps in understanding and using it. A function can be re-used. Write one time, use many times. (Known as staying {DRY} in the industry.)

Knowing what Americans mean when they talk about temperatures and weather can be difficult, but we can wrap the temperature conversion calculation ($$^{\circ}\text{C} = (^{\circ}\text{F} - 32) * \dfrac{5}{9}$$) up as a function that we can easily re-use.

```python
def fahr_to_celsius(temp):
    return ((temp - 32) * (5/9))
```

![The above function fahr to celsius is shown except annotated. def is labelled "def statement", fahr_to_celsius is noted as the function name. Inside parentheses is temp and an arrow shows it is called parameter names. The next line which is indented is annotated as the function body which has a return statement and the calculation from above.](../../images/python-basics/Figure7_Functions.png)

The function definition opens with the keyword `def` followed by the name of the function `fahr_to_celsius` and a parenthesized list of parameter names `temp`. The body of the function — the statements that are executed when it runs — is indented below the definition line. The body concludes with a `return` keyword followed by the return value.

When we call the function, the values we pass to it are assigned to those variables so that we can use them inside the function. Inside the function, we use a return statement to send a result back to whoever asked for it.

Let’s try running our function.

```python
fahr_to_celsius(32)
print(f"freezing point of water: {fahr_to_celsius(32)}C")
print(f"boiling point of water: {fahr_to_celsius(212)}C")
```

> <tip-title>Formatting Strings</tip-title>
> There are several ways to print out a few values in python. We'd recommend you to use `f-strings` as it's the cleanest and most modern way to do it.
>
> `f`-strings start with an `f` (very descriptive name eh?). Within the text between the single or double quotes (`'`/`"`) you can use curly braces to refer to variables or python code which will be placed there in the string.
>
> ```
> a = 10
> b = f"Here is the value of a: {a}"
> print(b)
> print(f"Here is the value of a: {5 + 5}")
> print(f"Here is the value of a: {function_that_returns_10()}")
> ```
>
> All of those would print out `Here is the value of a: 10`.
>
> f-strings can be a lot fancier for formatting decimal places, but we don't need that for now. Just know:
>
> 1. Start with an `f`
> 2. Use braces to use the value of a variable, a function, or some python expression.
{: .tip}

We’ve successfully called the function that we defined, and we have access to the value that we returned.

Now that we’ve seen how to turn Fahrenheit into Celsius, we can also write the function to turn Celsius into Kelvin:

```python
def celsius_to_kelvin(temp_c):
    return temp_c + 273.15

print(f'freezing point of water in Kelvin: {celsius_to_kelvin(0.)}')
```

> <tip-title>What is `0.`</tip-title>
> That's a float! A `.` in a number makes it a float, rather than an integer.
{: .tip}

What about converting Fahrenheit to Kelvin? We could write out both formulae, but we don’t need to. Instead, we can *compose* the two functions we have already created:

```python
def fahr_to_kelvin(temp_f):
    temp_c = fahr_to_celsius(temp_f)
    temp_k = celsius_to_kelvin(temp_c)
    return temp_k

print(f'boiling point of water in Kelvin: {fahr_to_kelvin(212.0)}')
```

This is our first taste of how larger programs are built: we define basic operations, then combine them in ever-larger chunks to get the effect we want. Real-life functions will usually be larger than the ones shown here — typically half a dozen to a few dozen lines — but they shouldn’t ever be much longer than that, or the next person who reads it won’t be able to understand what’s going on.

### Documentation

Documenting your code is extremely, *extremely*, **extremely** important to do. We all forget what we're doing, it's only normal, so documenting what you're doing is key to being able to restart work later.

```python
def fahr_to_kelvin(temp_f):
    """
    Converts a temperature from Fahrenheit to Kelvin

    temp_f: the temperature in Fahrenheit

    returns the temperature in Celsius
    """
    temp_c = fahr_to_celsius(temp_f)
    temp_k = celsius_to_kelvin(temp_c)
    return temp_k

print(f'boiling point of water in Kelvin: {fahr_to_kelvin(212.0)}')
```

For a function this small, with such a descriptive name (`fahr_to_kelvin`) it feels quite obvious what the function should do, what inputs it takes, what outputs it produces. However, you will thank yourself in the future if you do this now. You may think you will remember what the code does, but, be kind to your future self who is busy and stressed and may not want to spend time reading the code over again to figure out what every single function does.

> <question-title>Converting statements to functions</question-title>
> A lot of what you'll do in programing is to turn a procedure that you want to do, into statements and a function.
>
> Fill in the missing portions of this function, two average numbers. Then use it to find the average of 32326 and 631
>
> > <solution-title></solution-title>
> > ```
> > def average2(a, b):
> >     c = (a + b) / 2
> >     return c
> > ```
> > We call it "average2" here because it will only average two numbers. It will not work for three numbers or a list of them.
> {: .solution}
{: .question}

```python
# Test out solutions here!
def average2(a, b):
    c =
    return c

print(average2(32326, 631))
```

> <question-title>A more complicated example</question-title>
>
> The formula for a 90° triangle can be expressed as: $$c = \sqrt{a^2 + b^2}$$
>
> 1. Write a function which takes `a` and `b`, and calculates `c`
> 2. Name this function "pythagorus"
> 3. Remember to import math, if you haven't already
>
> > <solution-title></solution-title>
> > ```
> > def pythagorus(a, b):
> >     c = math.sqrt(math.pow(a, 2) + math.pow(b, 2))
> >     return c
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!


print(pythagorus(1234, 4321)) # Should return 4493.750883170984
```


### Variable Scope

In composing our temperature conversion functions, we created variables inside of those functions, `temp`, `temp_c`, `temp_f`, and `temp_k`. We refer to these variables as local variables because they no longer exist once the function is done executing. If we try to access their values outside of the function, we will encounter an error:

```python
print(f'Again, temperature in Kelvin was: {temp_k}')
```

If you want to reuse the temperature in Kelvin after you have calculated it with fahr_to_kelvin, you can store the result of the function call in a variable:

```python
temp_kelvin = fahr_to_kelvin(212.0)
print(f'temperature in Kelvin was: {temp_kelvin}')
```

Watch out for scope issues:

- Variables created inside a function will stay inside the function
- Variables created outside of the function can be accessible inside the function, but you should not do this!
- Ensure that variables are properly scoped will prevent later errors when working with modules or testing big projects.


```python
a = 1
b = 2.0

# Location 1

def some_generic_computation(x, y):
    c = x + y
    d = x * y
    e = c / d
    # Location 2
    return e

# Location 3
```

> <question-title>Scope</question-title>
> Given the above code, which variables are accessible at Locations 1, 2, and 3?
>
> > <solution-title></solution-title>
> > 1. a, b
> > 2. a and b are there but you shouldn't use these. x, y, c, d, e are also accessible.
> > 3. a, b.
> {: .solution}
{: .question}

### Defining Default parameters

If we usually want a function to work one way, but occasionally need it to do something else, we can allow people to pass a parameter when they need to but provide a default to make the normal case easier. The example below shows how Python matches values to parameters:

```python
def display(a=1, b=2, c=3):
    print(f'a: {a}, b: {b}, c: {c}')

# no parameters:
display()
# one parameter:
display(55)
# two parameters:
display(55, 66)
```

As this example shows, parameters are matched up from left to right, and any that haven’t been given a value explicitly get their default value. We can override this behavior by naming the value as we pass it in:

```python
# only setting the value of c
display(c=77)
```


> <question-title>Exercise: Signing a message</question-title>
> Let's test out a default argument. Imagine you are printing out a message, and at the bottom it should have a signature.
>
> Inputs:
> - `message`: a variable that is always provided to the function, it has no default.
> - `signature`: a variable that can be *optionally* provided, it should have a default like your name.
>
> You can accomplish this with *three print statements*:
> 1. Print the message
> 2. Print nothing (i.e. `print()`)
> 3. Print a signature variable.
>
> > <solution-title></solution-title>
> > ```
> > def myFunction(message, signature="Your name"):
> >     print(message)
> >     print()
> >     print(signature)
> > ```
> {: .solution}
{: .question}

```python
# Test things out here!
def myFunction # Fix this function!


# Here are some test cases, for you to check if your function works!
myFunction('This is a message')
myFunction('This is a message', signature='Jane Doe')
```


topic: data-science
tutorial: sql-basic
title: Introduction to SQL
difficulty: introductory
time_estimation: 3H
tools_json: []
requirements_json: ["[]"]
tags_json: ["SQL"]
content_hash: 5f8644b505f509e1a5bd63f555b6f442
last_modified: 2026-02-26T21:12:14.911421
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-basic/tutorial.html

content:
This tutorial will introduce you to {SQL} which can be used to query databases!

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


```sql
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c http://swcarpentry.github.io/sql-novice-survey/files/survey.db
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///survey.db")
%load_ext sql
%sql sqlite:///survey.db
%config SqlMagic.displaycon=False
```

# Selecting Data

A relational database
is a way to store and manipulate information.
Databases are arranged as table.
Each table has columns (also known as fields) that describe the data,
and rows (also known as records) which contain the data.

When we are using a spreadsheet,
we put formulas into cells to calculate new values based on old ones.
When we are using a database,
we send commands
(usually called queries)
to a database manager:
a program that manipulates the database for us.
The database manager does whatever lookups and calculations the query specifies,
returning the results in a tabular form
that we can then use as a starting point for further queries.

Queries are written in a language called {SQL},
SQL provides hundreds of different ways to analyze and recombine data.
We will only look at a handful of queries,
but that handful accounts for most of what scientists do.

> <tip-title>Changing Database Managers</tip-title>
>
> Many database managers --- Oracle,
> IBM DB2, PostgreSQL, MySQL, Microsoft Access, and SQLite ---  understand
> SQL but each stores data in a different way,
> so a database created with one cannot be used directly by another.
> However, every database manager
> can import and export data in a variety of formats like .csv, SQL,
> so it *is* possible to move information from one to another.
{: .tip}

Before we get into using {SQL} to select the data, let's take a look at the tables of the database we will use in our examples:

**Person**: people who took readings.

|id      |personal |family
|--------|---------|----------
|dyer    |William  |Dyer
|pb      |Frank    |Pabodie
|lake    |Anderson |Lake
|roe     |Valentina|Roerich
|danforth|Frank    |Danforth

**Site**: locations where readings were taken.

|name |lat   |long   |
|-----|------|-------|
|DR-1 |-49.85|-128.57|
|DR-3 |-47.15|-126.72|
|MSK-4|-48.87|-123.4 |

**Visited**: when readings were taken at specific sites.

|id   |site |dated     |
|-----|-----|----------|
|619  |DR-1 |1927-02-08|
|622  |DR-1 |1927-02-10|
|734  |DR-3 |1930-01-07|
|735  |DR-3 |1930-01-12|
|751  |DR-3 |1930-02-26|
|752  |DR-3 |None      |
|837  |MSK-4|1932-01-14|
|844  |DR-1 |1932-03-22|

**Survey**: the actual readings.  The field `quant` is short for quantitative and indicates what is being measured.  Values are `rad`, `sal`, and `temp` referring to 'radiation', 'salinity' and 'temperature', respectively.

|taken|person|quant|reading|
|-----|------|-----|-------|
|619  |dyer  |rad  |9.82   |
|619  |dyer  |sal  |0.13   |
|622  |dyer  |rad  |7.8    |
|622  |dyer  |sal  |0.09   |
|734  |pb    |rad  |8.41   |
|734  |lake  |sal  |0.05   |
|734  |pb    |temp |-21.5  |
|735  |pb    |rad  |7.22   |
|735  |None  |sal  |0.06   |
|735  |None  |temp |-26.0  |
|751  |pb    |rad  |4.35   |
|751  |pb    |temp |-18.5  |
|751  |lake  |sal  |0.1    |
|752  |lake  |rad  |2.19   |
|752  |lake  |sal  |0.09   |
|752  |lake  |temp |-16.0  |
|752  |roe   |sal  |41.6   |
|837  |lake  |rad  |1.46   |
|837  |lake  |sal  |0.21   |
|837  |roe   |sal  |22.5   |
|844  |roe   |rad  |11.25  |

Notice that three entries --- one in the `Visited` table,
and two in the `Survey` table --- don't contain any actual
data, but instead have a special `None` entry:
we'll return to these missing values.

For now,
let's write an SQL query that displays scientists' names.
We do this using the SQL command `SELECT`,
giving it the names of the columns we want and the table we want them from.
Our query and its output look like this:

```sql
SELECT family, personal FROM Person;
```

The semicolon at the end of the query
tells the database manager that the query is complete and ready to run.
We have written our commands in upper case and the names for the table and columns
in lower case,
but we don't have to:
as the example below shows,
SQL is case insensitive.

```sql
SeLeCt FaMiLy, PeRsOnAl FrOm PeRsOn;
```

You can use SQL's case insensitivity to your advantage. For instance,
some people choose to write SQL keywords (such as `SELECT` and `FROM`)
in capital letters and **field** and **table** names in lower
case. This can make it easier to locate parts of an SQL statement. For
instance, you can scan the statement, quickly locate the prominent
`FROM` keyword and know the table name follows.  Whatever casing
convention you choose, please be consistent: complex queries are hard
enough to read without the extra cognitive load of random
capitalization.  One convention is to use UPPER CASE for SQL
statements, to distinguish them from tables and column names. This is
the convention that we will use for this lesson.

> <question-title>Is a personal and family name column a good design?</question-title>
> If you were tasked with designing a database to store this same data, is storing the name data in
> this way the best way to do it? Why or why not?
>
> Can you think of any names that would be difficult to enter in such a schema?
>
> > <solution-title></solution-title>
> > No, it is generally not. There are a lot of [falsehoods that programmers believe about names](https://shinesolutions.com/2018/01/08/falsehoods-programmers-believe-about-names-with-examples/).
> > The situation is much more complex as you can read in that article, but names vary wildly and
> > generally placing constraints on how names are entered is only likely to frustrate you or your
> > users later on when they need to enter data into that database.
> >
> > In general you should consider using a single text field for the name and allowing users to
> > specify them as whatever they like (if it is a system with registration), or asking what they
> > wish to be recorded (if you are doing this sort of data collection).
> >
> > If you are doing scientific research, you might know that names are generally very poor
> > identifiers of a single human, and in that case consider recording their
> > [ORCiD](https://orcid.org/) which will help you reference that individual when you are
> > publishing later.
> >
> > This is also a good time to consider what data you really *need* to collect. If you are working
> > in the EU under GDPR, do you really need their full legal name? Is that necessary? Do you have a
> > plan for ensuring that data is correct when publishing, if any part of their name has changed
> > since?
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

While we are on the topic of SQL's syntax, one aspect of SQL's syntax
that can frustrate novices and experts alike is forgetting to finish a
command with `;` (semicolon).  When you press enter for a command
without adding the `;` to the end, it can look something like this:

~~~
SELECT id FROM Person
...>
...>
~~~

This is SQL's prompt, where it is waiting for additional commands or
for a `;` to let SQL know to finish.  This is easy to fix!  Just type
`;` and press enter!

Now, going back to our query,
it's important to understand that
the rows and columns in a database table aren't actually stored in any particular order.
They will always be *displayed* in some order,
but we can control that in various ways.
For example,
we could swap the columns in the output by writing our query as:

```sql
SELECT personal, family FROM Person;
```

or even repeat columns:

```sql
SELECT id, id, id FROM Person;
```

As a shortcut,
we can select all of the columns in a table using `*`:

```sql
SELECT * FROM Person;
```

> <question-title>Selecting Site Names</question-title>
>
> Write a query that selects only the `name` column from the `Site` table.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT name FROM Site;
> > ~~~
> >
> > |name      |
> > |----------|
> > |DR-1      |
> > |DR-3      |
> > |MSK-4     |
> {: .solution}
{: .question}


```sql
-- Try solutions here!
```

> <question-title>Query Style</question-title>
>
> Many people format queries as:
>
> ~~~
> SELECT personal, family FROM person;
> ~~~
>
> or as:
>
> ~~~
> select Personal, Family from PERSON;
> ~~~
>
> What style do you find easiest to read, and why?
{: .question}

```sql
-- Try solutions here!
```


# Sorting and Removing Duplicates

In beginning our examination of the Antarctic data, we want to know:

* what kind of quantity measurements were taken at each site;
* which scientists took measurements on the expedition;

To determine which measurements were taken at each site,
we can examine the `Survey` table.
Data is often redundant,
so queries often return redundant information.
For example,
if we select the quantities that have been measured
from the `Survey` table,
we get this:

```sql
SELECT quant FROM Survey;
```


This result makes it difficult to see all of the different types of
`quant` in the Survey table.  We can eliminate the redundant output to
make the result more readable by adding the `DISTINCT` keyword to our
query:

```sql
SELECT DISTINCT quant FROM Survey;
```

If we want to determine which visit (stored in the `taken` column)
have which `quant` measurement,
we can use the `DISTINCT` keyword on multiple columns.
If we select more than one column,
distinct *sets* of values are returned
(in this case *pairs*, because we are selecting two columns):

```sql
SELECT DISTINCT taken, quant FROM Survey;
```

Notice in both cases that duplicates are removed
even if the rows they come from didn't appear to be adjacent in the database table.


Our next task is to identify the scientists on the expedition by looking at the `Person` table.
As we mentioned earlier,
database records are not stored in any particular order.
This means that query results aren't necessarily sorted,
and even if they are,
we often want to sort them in a different way,
e.g., by their identifier instead of by their personal name.
We can do this in SQL by adding an `ORDER BY` clause to our query:

```sql
SELECT * FROM Person ORDER BY id;
```

|id     |personal |family  |
|-------|---------|--------|
|danfort|Frank    |Danforth|
|dyer   |William  |Dyer    |
|lake   |Anderson |Lake    |
|pb     |Frank    |Pabodie |
|roe    |Valentina|Roerich |

By default, when we use `ORDER BY`,
results are sorted in ascending order of the column we specify
(i.e.,
from least to greatest).

We can sort in the opposite order using `DESC` (for "descending"):

> <tip-title>A note on ordering</tip-title>
>
> While it may look that the records are consistent every time we ask for them in this lesson, that is because no one has changed or modified any of the data so far. Remember to use `ORDER BY` if you want the rows returned to have any sort of consistent or predictable order.
{: .tip}
```sql
SELECT * FROM person ORDER BY id DESC;
```

(And if we want to make it clear that we're sorting in ascending order,
we can use `ASC` instead of `DESC`.)


In order to look at which scientist measured quantities during each visit,
we can look again at the `Survey` table.
We can also sort on several fields at once.
For example,
this query sorts results first in ascending order by `taken`,
and then in descending order by `person`
within each group of equal `taken` values:

```sql
SELECT taken, person, quant FROM Survey ORDER BY taken ASC, person DESC;
```

This query gives us a good idea of which scientist was involved in which visit,
and what measurements they performed during the visit.

Looking at the table, it seems like some scientists specialized in
certain kinds of measurements.  We can examine which scientists
performed which measurements by selecting the appropriate columns and
removing duplicates.

```sql
SELECT DISTINCT quant, person FROM Survey ORDER BY quant ASC;
```

> <question-title>Finding Distinct Dates</question-title>
>
> Write a query that selects distinct dates from the `Visited` table.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT DISTINCT dated FROM Visited;
> > ~~~
> >
> > |dated     |
> > |----------|
> > |1927-02-08|
> > |1927-02-10|
> > |1930-01-07|
> > |1930-01-12|
> > |1930-02-26|
> > |&nbsp;    |
> > |1932-01-14|
> > |1932-03-22|
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Displaying Full Names</question-title>
>
> Write a query that displays the full names of the scientists in the `Person` table,
> ordered by family name.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT personal, family FROM Person ORDER BY family ASC;
> > ~~~
> >
> > |personal  |family    |
> > |----------|----------|
> > |Frank     |Danforth  |
> > |William   |Dyer      |
> > |Anderson  |Lake      |
> > |Frank     |Pabodie   |
> > |Valentina |Roerich   |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <tip-title>Is sorting names useful?</tip-title>
> If you are someone with a name which falls at the end of the alphabet, you've likely been
> penalised for this your entire life. Alphabetically sorting names should always be looked at
> critically and through a lens to whether you are fairly reflecting everyone's contributions,
> rather than just the default sort order.
>
> There are many options, either by some metric of contribution that everyone could agree on, or
> better, consider random sorting, like the GTN uses with our [Hall of Fame]({% link hall-of-fame.md %})
> page where we intentionally order randomly to tell contributors that no one persons
> contributions matter more than anothers.
>
> > The evidence provided in a variety of studies leaves no doubt that an
> > alphabetical author ordering norm disadvantages researchers with
> > last names toward the end of the alphabet. There is furthermore con-
> > vincing evidence that researchers are aware of this and that they
> > react strategically to such alphabetical discrimination, for example
> > with their choices of who to collaborate with. See {% cite Weber_2018 %} for more.
> {: .quote}
{: .tip}

> <tip-title>Name collation</tip-title>
> When you are sorting things in SQL, you need to be aware of something called collation which can
> affect your results if you have values that are not the letters A-Z. Collating is the process of
> sorting values, and this affects many human languages when storing data in a database.
>
> Here is a Dutch example. In the old days their alphabet contained a `ÿ` which was later replaced
> with `ĳ`, a digraph of two characters squished together. This is commonly rendered as `ij`
> however, two separate characters, due to the internet and widespread use of keyboards featuring
> mainly ascii characters. However, it is still the 25th letter of their alphabet.
>
> ```
> sqlite> create table nl(value text);
> sqlite> insert into nl values ('appel'), ('beer'), ('index'), ('ijs'), ('jammer'), ('winkel'), ('zon');
> sqlite> select * from nl order by value;
> appel
> beer
> index
> ijs
> jammer
> winkel
> zon
> ```
>
> Find a dutch friend and ask them if this is the correct order for this list. Unfortunately it
> isn't. Even though it is `ij` as two separate characters, it should be sorted as if it was `ĳ` or
> `ÿ`, before `z`. Like so: appel, beer, index, jammer, winkel, ijs, zon
>
> While there is not much you can do about it now (you're just beginning!) it is something you
> should be aware of. When you later need to know about this, you will find the term 'collation'
> useful, and you'll find the procedure is different for every database engine.
{: .tip}

# Filtering

One of the most powerful features of a database is
the ability to filter data,
i.e.,
to select only those records that match certain criteria.
For example,
suppose we want to see when a particular site was visited.
We can select these records from the `Visited` table
by using a `WHERE` clause in our query:

```sql
SELECT * FROM Visited WHERE site = 'DR-1';
```

The database manager executes this query in two stages.
First,
it checks at each row in the `Visited` table
to see which ones satisfy the `WHERE`.
It then uses the column names following the `SELECT` keyword
to determine which columns to display.

This processing order means that
we can filter records using `WHERE`
based on values in columns that aren't then displayed:

```sql
SELECT id FROM Visited WHERE site = 'DR-1';
```

![SQL Filtering in Action]`(../../images/carpentries-sql/sql-filter.svg)

We can use many other Boolean operators to filter our data.
For example,
we can ask for all information from the DR-1 site collected before 1930:

```sql
SELECT * FROM Visited WHERE site = 'DR-1' AND dated < '1930-01-01';
```

> <tip-title>Date Types</tip-title>
>
> Most database managers have a special data type for dates.
> In fact, many have two:
> one for dates,
> such as "May 31, 1971",
> and one for durations,
> such as "31 days".
> SQLite doesn't:
> instead,
> it stores dates as either text
> (in the ISO-8601 standard format "YYYY-MM-DD HH:MM:SS.SSSS"),
> real numbers
> ([Julian days](https://en.wikipedia.org/wiki/Julian_day), the number of days since November 24, 4714 BCE),
> or integers
> ([Unix time](https://en.wikipedia.org/wiki/Unix_time), the number of seconds since midnight, January 1, 1970).
> If this sounds complicated,
> it is,
> but not nearly as complicated as figuring out
> [historical dates in Sweden](https://en.wikipedia.org/wiki/Swedish_calendar).
{: .tip}

> <tip-title>'30: 1930 or 2030?</tip-title>
> Storing the year as the last two digits causes problems in databases, and is part of what caused
> [Y2K](https://en.wikipedia.org/wiki/Year_2000_problem). Be sure to use the databases' built in
> format for storing dates, if it is available as that will generally avoid any major issues.
>
> Similarly there is a ["Year 2038 problem"](https://en.wikipedia.org/wiki/Year_2000_problem#Year_2038_problem),
> as the timestamps mentioned above that count seconds since Jan 1, 1970 were running out of space
> on 32-bit machines. Many systems have since migrated to work around this with 64-bit timestamps.
{: .tip}

If we want to find out what measurements were taken by either Lake or Roerich,
we can combine the tests on their names using `OR`:

```sql
SELECT * FROM Survey WHERE person = 'lake' OR person = 'roe';
```

Alternatively,
we can use `IN` to see if a value is in a specific set:

```sql
SELECT * FROM Survey WHERE person IN ('lake', 'roe');
```

We can combine `AND` with `OR`,
but we need to be careful about which operator is executed first.
If we *don't* use parentheses,
we get this:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND person = 'lake' OR person = 'roe';
```

which is salinity measurements by Lake,
and *any* measurement by Roerich.
We probably want this instead:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND (person = 'lake' OR person = 'roe');
```

We can also filter by partial matches.  For example, if we want to
know something just about the site names beginning with "DR" we can
use the `LIKE` keyword.  The percent symbol acts as a
wildcard, matching any characters in that
place.  It can be used at the beginning, middle, or end of the string
[See this page on wildcards](https://www.w3schools.com/sql/sql_wildcards.asp) for more information:

```sql
SELECT * FROM Visited WHERE site LIKE 'DR%';
```


Finally,
we can use `DISTINCT` with `WHERE`
to give a second level of filtering:

```sql
SELECT DISTINCT person, quant FROM Survey WHERE person = 'lake' OR person = 'roe';
```

But remember:
`DISTINCT` is applied to the values displayed in the chosen columns,
not to the entire rows as they are being processed.

> <tip-title>Growing Queries</tip-title>
>
> What we have just done is how most people "grow" their {SQL} queries.
> We started with something simple that did part of what we wanted,
> then added more clauses one by one,
> testing their effects as we went.
> This is a good strategy --- in fact,
> for complex queries it's often the *only* strategy --- but
> it depends on quick turnaround,
> and on us recognizing the right answer when we get it.
>
> The best way to achieve quick turnaround is often
> to put a subset of data in a temporary database
> and run our queries against that,
> or to fill a small database with synthesized records.
> For example,
> instead of trying our queries against an actual database of 20 million Australians,
> we could run it against a sample of ten thousand,
> or write a small program to generate ten thousand random (but plausible) records
> and use that.
{: .tip}

> <question-title>Fix This Query</question-title>
>
> Suppose we want to select all sites that lie within 48 degrees of the equator.
> Our first query is:
>
> ~~~
> SELECT * FROM Site WHERE (lat > -48) OR (lat < 48);
> ~~~
>
> Explain why this is wrong,
> and rewrite the query so that it is correct.
>
> > <solution-title></solution-title>
> >
> > Because we used `OR`, a site on the South Pole for example will still meet
> > the second criteria and thus be included. Instead, we want to restrict this
> > to sites that meet _both_ criteria:
> >
> > ~~~
> > SELECT * FROM Site WHERE (lat > -48) AND (lat < 48);
> > ~~~
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Finding Outliers</question-title>
>
> Normalized salinity readings are supposed to be between 0.0 and 1.0.
> Write a query that selects all records from `Survey`
> with salinity values outside this range.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT * FROM Survey WHERE quant = 'sal' AND ((reading > 1.0) OR (reading < 0.0));
> > ~~~
> >
> > |taken     |person    |quant     |reading   |
> > |----------|----------|----------|----------|
> > |752       |roe       |sal       |41.6      |
> > |837       |roe       |sal       |22.5      |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title> Matching Patterns</question-title>
>
> Which of these expressions are true?
>
> 1. `'a' LIKE 'a'`
> 2. `'a' LIKE '%a'`
> 3. `'beta' LIKE '%a'`
> 4. `'alpha' LIKE 'a%%'`
> 5. `'alpha' LIKE 'a%p%'`
>
> > <solution-title></solution-title>
> >
> > 1. True because these are the same character.
> > 2. True because the wildcard can match _zero_ or more characters.
> > 3. True because the `%` matches `bet` and the `a` matches the `a`.
> > 4. True because the first wildcard matches `lpha` and the second wildcard matches zero characters (or vice versa).
> > 5. True because the first wildcard matches `l` and the second wildcard matches `ha`.
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <tip-title>Case-insensitive matching</tip-title>
> But what about if you don't care about if it's `ALPHA` or `alpha` in the database, and you are
> using a language that has a notion of case (unlike e.g. Chinese, Japenese)?
>
> Then you can use the `ILIKE` operator for 'case Insensitive LIKE'.
> for example the following are true:
>
> - `'a' ILIKE 'A'`
> - `'AlPhA' ILIKE '%lpha'`
{: .tip}


# Calculating New Values

After carefully re-reading the expedition logs,
we realize that the radiation measurements they report
may need to be corrected upward by 5%.
Rather than modifying the stored data,
we can do this calculation on the fly
as part of our query:

```sql
SELECT 1.05 * reading FROM Survey WHERE quant = 'rad';
```

When we run the query,
the expression `1.05 * reading` is evaluated for each row.
Expressions can use any of the fields,
all of usual arithmetic operators,
and a variety of common functions.
(Exactly which ones depends on which database manager is being used.)
For example,
we can convert temperature readings from Fahrenheit to Celsius
and round to two decimal places:

```sql
SELECT taken, round(5 * (reading - 32) / 9, 2) FROM Survey WHERE quant = 'temp';
```


As you can see from this example, though, the string describing our
new field (generated from the equation) can become quite unwieldy. {SQL}
allows us to rename our fields, any field for that matter, whether it
was calculated or one of the existing fields in our database, for
succinctness and clarity. For example, we could write the previous
query as:

```sql
SELECT taken, round(5 * (reading - 32) / 9, 2) as Celsius FROM Survey WHERE quant = 'temp';
```

We can also combine values from different fields,
for example by using the string concatenation operator `||`:

```sql
SELECT personal || ' ' || family FROM Person;
```

But of course that can also be solved by simply having a single name field which avoids other
issues.

> <question-title> Fixing Salinity Readings</question-title>
>
> After further reading,
> we realize that Valentina Roerich
> was reporting salinity as percentages.
> Write a query that returns all of her salinity measurements
> from the `Survey` table
> with the values divided by 100.
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT taken, reading / 100 FROM Survey WHERE person = 'roe' AND quant = 'sal';
> > ~~~
> >
> > |taken     |reading / 100|
> > |----------|-------------|
> > |752       |0.416        |
> > |837       |0.225        |
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Unions</question-title>
>
> The `UNION` operator combines the results of two queries:
>
> ~~~
> SELECT * FROM Person WHERE id = 'dyer' UNION SELECT * FROM Person WHERE id = 'roe';
> ~~~
>
> |id  |personal |family |
> |----|-------- |-------|
> |dyer|William  |Dyer   |
> |roe |Valentina|Roerich|
>
> The `UNION ALL` command is equivalent to the `UNION` operator,
> except that `UNION ALL` will select all values.
> The difference is that `UNION ALL` will not eliminate duplicate rows.
> Instead, `UNION ALL` pulls all rows from the query
> specifics and combines them into a table.
> The `UNION` command does a `SELECT DISTINCT` on the results set.
> If all the records to be returned are unique from your union,
> use `UNION ALL` instead, it gives faster results since it skips the `DISTINCT` step.
> For this section, we shall use UNION.
>
> Use `UNION` to create a consolidated list of salinity measurements
> in which Valentina Roerich's, and only Valentina's,
> have been corrected as described in the previous challenge.
> The output should be something like:
>
> |taken|reading|
> |-----|-------|
> |619  |0.13   |
> |622  |0.09   |
> |734  |0.05   |
> |751  |0.1    |
> |752  |0.09   |
> |752  |0.416  |
> |837  |0.21   |
> |837  |0.225  |
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT taken, reading FROM Survey WHERE person != 'roe' AND quant = 'sal' UNION SELECT taken, reading / 100 FROM Survey WHERE person = 'roe' AND quant = 'sal' ORDER BY taken ASC;
> > ~~~
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Selecting Major Site Identifiers</question-title>
>
> The site identifiers in the `Visited` table have two parts
> separated by a '-':
>
> ~~~
> SELECT DISTINCT site FROM Visited;
> ~~~
>
> |site |
> |-----|
> |DR-1 |
> |DR-3 |
> |MSK-4|
>
> Some major site identifiers (i.e. the letter codes) are two letters long and some are three.
> The "in string" function `instr(X, Y)`
> returns the 1-based index of the first occurrence of string Y in string X,
> or 0 if Y does not exist in X.
> The substring function `substr(X, I, [L])`
> returns the substring of X starting at index I, with an optional length L.
> Use these two functions to produce a list of unique major site identifiers.
> (For this data,
> the list should contain only "DR" and "MSK").
>
> > <solution-title></solution-title>
> > ```
> > SELECT DISTINCT substr(site, 1, instr(site, '-') - 1) AS MajorSite FROM Visited;
> > ```
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

# Missing Data

Real-world data is never complete --- there are always holes.
Databases represent these holes using a special value called `null`.
`null` is not zero, `False`, or the empty string;
it is a one-of-a-kind value that means "nothing here".
Dealing with `null` requires a few special tricks
and some careful thinking.

By default, the Python SQL interface does not display NULL values in its output, instead it shows `None`.

To start,
let's have a look at the `Visited` table.
There are eight records,
but #752 doesn't have a date --- or rather,
its date is null:

```sql
SELECT * FROM Visited;
```

Null doesn't behave like other values.
If we select the records that come before 1930:

```sql
SELECT * FROM Visited WHERE dated < '1930-01-01';
```

we get two results,
and if we select the ones that come during or after 1930:

```sql
SELECT * FROM Visited WHERE dated >= '1930-01-01';
```

we get five,
but record #752 isn't in either set of results.
The reason is that
`null<'1930-01-01'`
is neither true nor false:
null means, "We don't know,"
and if we don't know the value on the left side of a comparison,
we don't know whether the comparison is true or false.
Since databases represent "don't know" as null,
the value of `null<'1930-01-01'`
is actually `null`.
`null>='1930-01-01'` is also null
because we can't answer to that question either.
And since the only records kept by a `WHERE`
are those for which the test is true,
record #752 isn't included in either set of results.

Comparisons aren't the only operations that behave this way with nulls.
`1+null` is `null`,
`5*null` is `null`,
`log(null)` is `null`,
and so on.
In particular,
comparing things to null with = and != produces null:

```sql
SELECT * FROM Visited WHERE dated = NULL;
```

produces no output, and neither does:

```sql
SELECT * FROM Visited WHERE dated != NULL;
```

To check whether a value is `null` or not,
we must use a special test `IS NULL`:

```sql
SELECT * FROM Visited WHERE dated IS NULL;
```

or its inverse `IS NOT NULL`:

```sql
SELECT * FROM Visited WHERE dated IS NOT NULL;
```

Null values can cause headaches wherever they appear.
For example,
suppose we want to find all the salinity measurements
that weren't taken by Lake.
It's natural to write the query like this:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND person != 'lake';
```

but this query filters omits the records
where we don't know who took the measurement.
Once again,
the reason is that when `person` is `null`,
the `!=` comparison produces `null`,
so the record isn't kept in our results.
If we want to keep these records
we need to add an explicit check:

```sql
SELECT * FROM Survey WHERE quant = 'sal' AND (person != 'lake' OR person IS NULL);
```


We still have to decide whether this is the right thing to do or not.
If we want to be absolutely sure that
we aren't including any measurements by Lake in our results,
we need to exclude all the records for which we don't know who did the work.

In contrast to arithmetic or Boolean operators, aggregation functions
that combine multiple values, such as `min`, `max` or `avg`, *ignore*
`null` values. In the majority of cases, this is a desirable output:
for example, unknown values are thus not affecting our data when we
are averaging it. Aggregation functions will be addressed in more
detail in [the next section](#).

> <question-title>Sorting by Known Date</question-title>
>
> Write a query that sorts the records in `Visited` by date,
> omitting entries for which the date is not known
> (i.e., is null).
>
> > <solution-title></solution-title>
> >
> > ~~~
> > SELECT * FROM Visited WHERE dated IS NOT NULL ORDER BY dated ASC;
> > ~~~
> >
> > |id        |site      |dated     |
> > |----------|----------|----------|
> > |619       |DR-1      |1927-02-08|
> > |622       |DR-1      |1927-02-10|
> > |734       |DR-3      |1930-01-07|
> > |735       |DR-3      |1930-01-12|
> > |751       |DR-3      |1930-02-26|
> > |837       |MSK-4     |1932-01-14|
> > |844       |DR-1      |1932-03-22|
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>NULL in a Set</question-title>
>
> What do you expect the following query to produce?
>
> ~~~
> SELECT * FROM Visited WHERE dated IN ('1927-02-08', NULL);
> ~~~
>
> What does it actually produce?
> > <solution-title></solution-title>
> >
> > You might expect the above query to return rows where dated is either '1927-02-08' or NULL.
> > Instead it only returns rows where dated is '1927-02-08', the same as you would get from this
> > simpler query:
> >
> > ~~~
> > SELECT * FROM Visited WHERE dated IN ('1927-02-08');
> > ~~~
> >
> > The reason is that the `IN` operator works with a set of *values*, but NULL is by definition
> > not a value and is therefore simply ignored.
> >
> > If we wanted to actually include NULL, we would have to rewrite the query to use the IS NULL condition:
> >
> > ~~~
> > SELECT * FROM Visited WHERE dated = '1927-02-08' OR dated IS NULL;
> > ~~~
> {: .solution}
{: .question}

```sql
-- Try solutions here!
```

> <question-title>Pros and Cons of Sentinels</question-title>
>
> Some database designers prefer to use
> a sentinel value
> to mark missing data rather than `null`.
> For example,
> they will use the date "0000-00-00" to mark a missing date,
> or -1.0 to mark a missing salinity or radiation reading
> (since actual readings cannot be negative).
> What does this simplify?
> What burdens or risks does it introduce?
{: .question}

```sql
-- Try solutions here!
```

{% include _includes/quiz.html id="recap.yml" %}


topic: data-science
tutorial: python-venv
title: Virtual Environments For Software Development
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: b479120bca8c0065bc6a7789e54141d7
last_modified: 2026-02-26T21:12:14.905655
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-venv/tutorial.html

content:
"Virtual Environments" allow you to easily manage your installed Python packages and prevent conflicts between different project's dependencies. In general most modern projects should use `conda` for dependency management, but `venv` can be convenient for Python-only projects.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}


If you have a python project you are using, you will often see something like
following two lines somewhere at the top.

```python
from matplotlib import pyplot as plt
import numpy as np
```

This means that our code requires two *external libraries* (also called third-party packages or dependencies) -
`numpy` and `matplotlib`.
Python applications often use external libraries that don’t come as part of the standard Python distribution. This means
that you will have to use a *package manager* tool to install them on your system.
Applications will also sometimes need a
specific version of an external library (e.g. because they require that a particular
bug has been fixed in a newer version of the library), or a specific version of Python interpreter.
This means that each Python application you work with may require a different setup and a set of dependencies so it
is important to be able to keep these configurations separate to avoid confusion between projects.
The solution for this problem is to create a self-contained *virtual
environment* per project, which contains a particular version of Python installation plus a number of
additional external libraries.

Virtual environments are not just a feature of Python - all modern programming languages use them to isolate code
of a specific project and make it easier to develop, run, test and share code with others. In this tutorial, we learn how
to set up a virtual environment to develop our code and manage our external dependencies.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Virtual Environments

So what exactly are virtual environments, and why use them?

A Python virtual environment is an **isolated working copy** of a specific version of
Python interpreter together with specific versions of a number of external libraries installed into that
virtual environment. A virtual environment is simply a *directory with a particular
structure* which includes links to and enables multiple side-by-side installations of
different Python interpreters or different versions of the same external library to coexist on your machine and only one to be selected for each of our projects. This allows you to work on a particular
project without worrying about affecting other projects on your machine.

As more external libraries are added to your Python project over time, you can add them to
its specific virtual environment and avoid a great deal of confusion by having separate (smaller) virtual environments
for each project rather than one huge global environment with potential package version clashes. Another big motivator
for using virtual environments is that they make sharing your code with others much easier (as we will see shortly).
Here are some typical scenarios where the usage of virtual environments is highly recommended (almost unavoidable):

- You have an older project that only works under Python 2. You do not have the time to migrate the project to Python 3
  or it may not even be possible as some of the third party dependencies are not available under Python 3. You have to
  start another project under Python 3. The best way to do this on a single machine is to set up two separate Python virtual
  environments.
- One of your Python 3 projects is locked to use a particular older version of a third party dependency. You cannot use the
  latest version of the
  dependency as it breaks things in your project. In a separate branch of your project, you want to try and fix problems
  introduced by the new version of the dependency without affecting the working version of your project. You need to set up
  a separate virtual environment for your branch to 'isolate' your code while testing the new feature.

You do not have to worry too much about specific versions of external libraries that your project depends on most of the time.
Virtual environments enable you to always use the latest available version without specifying it explicitly.
They also enable you to use a specific older version of a package for your project, should you need to.

> <tip-title>A Specific Python or Package Version is Only Ever Installed Once</tip-title>
> Note that you will not have a separate Python or package installations for each of your projects - they will only
> ever be installed once on your system but will be referenced from different virtual environments.
{: .tip}

### Managing Python Virtual Environments

There are several commonly used command line tools for managing Python virtual environments:
- `venv`, available by default from the standard `Python` distribution from `Python 3.3+`
- `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions
- `pipenv`, created to fix certain shortcomings of `virtualenv`
- `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community)
- `poetry`, a modern Python packaging tool which handles virtual environments automatically

While there are pros and cons for using each of the above, all will do the job of managing Python
virtual environments for you and it may be a matter of personal preference which one you go for.
In this course, we will use `venv` to create and manage our
virtual environment (which is the preferred way for Python 3.3+).

Until you encounter the needs of a project which goes beyond what is available
in the Python ecosystem, e.g. when you depend on external packages like htslib
or bioinformatics tools that are simply not distributed as part of PyPI, then
`venv` is a good choice to get started with.


### Managing Python Packages

Part of managing your (virtual) working environment involves installing, updating and removing external packages
on your system. The Python package manager tool `pip` is most commonly used for this - it interacts
 and obtains the packages from the central repository called [Python Package Index (PyPI)](https://pypi.org/).
`pip` can now be used with all Python distributions (including Anaconda).

> <tip-title>A Note on Anaconda and `conda`</tip-title>
> Anaconda is an open source Python
> distribution commonly used for scientific programming - it conveniently installs Python, package and environment management `conda`, and a 
> number of commonly used scientific computing packages so you do not have to obtain them separately. 
> `conda` is an independent command line tool (available separately from the Anaconda distribution too) with dual functionality: (1) it is a package manager that helps you find Python packages from
> remote package repositories and install them on your system, and (2) it is also a virtual environment manager. So, you can use `conda` for both tasks instead of using `venv` and `pip`.
{: .tip}

### Many Tools for the Job

Installing and managing Python distributions, external libraries and virtual environments is, well,
complex. There is an abundance of tools for each task, each with its advantages and disadvantages, and there are different
ways to achieve the same effect (and even different ways to install the same tool!).
Note that each Python distribution comes with its own version of
`pip` - and if you have several Python versions installed you have to be extra careful to use the correct `pip` to
manage external packages for that Python version.

`venv` and `pip` are considered the *de facto* standards for virtual environment and package management for Python 3.
However, the advantages of using Anaconda and `conda` are that you get (most of the) packages needed for
scientific code development included with the distribution. If you are only collaborating with others who are also using
Anaconda, you may find that `conda` satisfies all your needs. It is good, however, to be aware of all these tools,
and use them accordingly. As you become more familiar with them you will realise that equivalent tools work in a similar
way even though the command syntax may be different (and that there are equivalent tools for other programming languages
too to which your knowledge can be ported).

![Python environment hell XKCD comic  showing boxes like pip, easy_install, homebrew 2.7, anaconda, homebrew 3.6, /usr/local/Cellar, ~/python/, and a chaotic mess of arrows moving between them all. At the bottom is the text: My python environment has become so degraded that my laptop has been declared a superfund site. (A superfund site is generally an environmental disaster area.)](../../images/xkcd/python_environment.png "Python Environment Hell from XKCD 1987 (CC-BY-NC 2.5)")

Let us have a look at how we can create and manage virtual environments from the command line using `venv` and manage packages using `pip`.

### Creating a `venv` Environment
Creating a virtual environment with `venv` is done by executing the following command:

```bash
python3 -m venv /path/to/new/virtual/environment
```

where `/path/to/new/virtual/environment` is a path to a directory where you want to place it - conventionally within
your software project so they are co-located.
This will create the target directory for the virtual environment (and any parent directories that don’t exist already).

For our project, let's create a virtual environment called `venv` off the project root:

```bash
python3 -m venv venv
```

If you list the contents of the newly created `venv` directory, on a Mac or Linux system 
(slightly different on Windows as explained below) you should see something like:

```bash
ls -l venv
```

So, running the `python3 -m venv venv` command created the target directory called `venv`
containing:

- `pyvenv.cfg` configuration file with a home key pointing to the Python installation from which the command was run,
- `bin` subdirectory (called `Scripts` on Windows) containing a symlink of the Python interpreter binary used to create the
environment and the standard Python library,
- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) to contain its own independent set of installed Python packages isolated from other projects,
- various other configuration and supporting files and subdirectories.

> <tip-title>Naming Virtual Environments</tip-title>
> What is a good name to use for a virtual environment? Using "venv" or ".venv" as the
> name for an environment and storing it within the project's directory seems to be the recommended way -
> this way when you come across such a subdirectory within a software project,
> by convention you know it contains its virtual environment details.
> A slight downside is that all different virtual environments
> on your machine then use the same name and the current one is determined by the context of the path
> you are currently located in. A (non-conventional) alternative is to
> use your project name for the name of the virtual environment, with the downside that there is nothing to indicate
> that such a directory contains a virtual environment. In our case, we have settled to use the name "venv" since it is 
> not a hidden directory and we want it to be displayed by the command line when listing directory contents (hence, 
> no need for the "." in its name that would, by convention, make it hidden). In the future,
> you will decide what naming convention works best for you. Here are some references for each of the naming conventions:
> - [The Hitchhiker's Guide to Python](https://docs.python-guide.org/dev/virtualenvs/) notes that "venv" is the general convention used globally
> - [The Python Documentation](https://docs.python.org/3/library/venv.html) indicates that ".venv" is common
> - ["venv" vs ".venv" discussion](https://discuss.python.org/t/trying-to-come-up-with-a-default-directory-name-for-virtual-environments/3750)
{: .tip}

Once you’ve created a virtual environment, you will need to activate it:

```bash
source venv/bin/activate
```

Activating the virtual environment will change your command line’s prompt to show what virtual environment
you are currently using (indicated by its name in round brackets at the start of the prompt),
and modify the environment so that running Python will get you the particular
version of Python configured in your virtual environment.

You can verify you are using your virtual environment's version of Python by checking the path using `which`:
```bash
which python3
```

When you’re done working on your project, you can exit the environment with:

```bash
deactivate
```

If you've just done the `deactivate`, ensure you reactivate the environment ready for the next part:

```bash
source venv/bin/activate
```

> <tip-title>Python Within A Virtual Environment</tip-title>
> 
> Within a virtual environment, commands `python` and `pip` will refer to the version of Python you created the environment with. If you create a virtual environment with `python3 -m venv venv`, `python` will refer to `python3` and `pip` will refer to `pip3`. 
>
> On some machines with Python 2 installed, `python` command may refer to the copy of Python 2 installed outside of the virtual environment instead, which can cause confusion. You can always check which version of Python you are using in your virtual environment with the command `which python` to be absolutely sure. We continue using `python3` and `pip3` in this material to avoid confusion for those users, but commands `python` and `pip` may work for you as expected.
{: .tip}

### Installing External Libraries in an Environment with `pip`

We noticed earlier that our code depends on two *external libraries* - `numpy` and `matplotlib`. In order
for the code to run on your machine, you need to
install these two dependencies into your virtual environment.

To install the latest version of a package with `pip` you use pip's `install` command and specify the package’s name, e.g.:

```bash
pip3 install numpy
pip3 install matplotlib
```

or like this to install multiple packages at once for short:

```bash
pip3 install numpy matplotlib
```

> <tip-title>How About `python3 -m pip install`?</tip-title>
> Why are we not using `pip` as an argument to `python3` command, in the same way we did with `venv` 
> (i.e. `python3 -m venv`)? `python3 -m pip install` should be used according to the 
> [official Pip documentation](https://pip.pypa.io/en/stable/user_guide/#running-pip); other official documentation
> still seems to have a mixture of usages. Core Python developer Brett Cannon offers a 
> [more detailed explanation](https://snarky.ca/why-you-should-use-python-m-pip/) of edge cases when the two options may produce 
> different results and recommends `python3 -m pip install`. We kept the old-style command (`pip3 install`) as it seems more 
> prevalent among developers at the moment - but it may be a convention that will soon change and certainly something you should consider. 
{: .tip}

If you run the `pip3 install` command on a package that is already installed, `pip` will notice this and do nothing.

To install a specific version of a Python package give the package name followed by `==` and the version number, e.g.
`pip3 install numpy==1.21.1`.

To specify a minimum version of a Python package, you can
do `pip3 install numpy>=1.20`.

To upgrade a package to the latest version, e.g. `pip3 install --upgrade numpy`.

To display information about a particular installed package do:

```bash
pip3 show numpy
```

To list all packages installed with `pip` (in your current virtual environment):

```bash
pip3 list
```

To uninstall a package installed in the virtual environment do: `pip3 uninstall package-name`.
You can also supply a list of packages to uninstall at the same time.

### Exporting/Importing an Environment with `pip`

You are collaborating on a project with a team so, naturally, you will want to share your environment with your
collaborators so they can easily 'clone' your software project with all of its dependencies and everyone
can replicate equivalent virtual environments on their machines. `pip` has a handy way of exporting,
saving and sharing virtual environments.

To export your active environment - use `pip freeze` command to
produce a list of packages installed in the virtual environment.
A common convention is to put this list in a `requirements.txt` file:

```bash
pip3 freeze > requirements.txt
cat requirements.txt
```

The first of the above commands will create a `requirements.txt` file in your current directory.
The `requirements.txt` file can then be committed to a version control system and
get shipped as part of your software and shared with collaborators and/or users. They can then replicate your environment and
install all the necessary packages from the project root as follows:

```bash
pip3 install -r requirements.txt
```

As your project grows - you may need to update your environment for a variety of reasons. For example, one of your project's dependencies has
just released a new version (dependency version number update), you need an additional package for data analysis
(adding a new dependency) or you have found a better package and no longer need the older package (adding a new and
removing an old dependency). What you need to do in this case (apart from installing the new and removing the
packages that are no longer needed from your virtual environment) is update the contents of the `requirements.txt` file
accordingly by re-issuing `pip freeze` command and propagate the updated `requirements.txt` file to your collaborators
via your code sharing platform (e.g. GitHub).

> <tip-title>Official Documentation</tip-title>
> For a full list of options and commands, consult the [official `venv` documentation](https://docs.python.org/3/library/venv.html)
> and the [Installing Python Modules with `pip` guide](https://docs.python.org/3/installing/index.html#installing-index). Also check out the guide ["Installing packages using `pip` and virtual environments"](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-packages-using-pip-and-virtual-environments).
{: .tip}

## Running Python Scripts From Command Line
Congratulations! Your environment is now activated and set up to run your script
from the command line.


topic: data-science
tutorial: python-glob
title: Python - Globbing
difficulty: intermediate
time_estimation: 15M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: b92d9b7742f1b580a74dc6d4092f134d
last_modified: 2026-02-26T21:12:14.902247
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-glob/tutorial.html

content:
Globbing is the term used in computer science when we have a bunch of files and we want to list all of them matching some pattern.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Setup

We'll start by creating some files for use in the rest of this tutorial

```python
import os
import subprocess

dirs = ['a', 'a/b', 'c', 'c/e', 'd', '.']
files = ['a.txt', 'a.csv', 'b.csv', 'b.txt', 'e.glm']

for d in dirs:
    # Create some directories
    os.makedirs(d, exist_ok=True)
    # Create some files
    for f in files:
        subprocess.check_output(['touch', os.path.join(d, f)])
```

Now we should have a pretty full folder!

# Finding Files

We can use the glob module to find files:

```python
import glob
print(glob.glob('*.csv'))
print(glob.glob('*.txt'))
```

Here we use an asterisk (`*`) as a wildcard, it matches any bit of text (but not into folders!) to all matching files. Here we list all matching `csv` or `txt` files. This is great to find files matching a pattern.

We can also use asterisks anywhere in the glob, it doesn't just have to be the filename portion:

```python
print(glob.glob('a*'))
```

Here we even see a third entry: the directory.

# Finding files in directories

Until now we've found only files in a single top level directory, but what if we wanted to find files in subdirectories?

Only need a single directory? Just include that!

```python
print(glob.glob('a/*.csv'))
```

But if you need more levels, or want to look in *all* folders, then you need the double wildcard! With two asterisks `**` we can search recursively through directories for files:

```python
print(glob.glob('**/a.csv'))
```

# Exercise

> <question-title>Where in the world is the CSV?</question-title>
>
> 1. How would you find all `.csv` files?
> 2. How would you find all `.txt` files?
> 3. How would you find all files starting with the letter 'e'?
>
> > <solution-title></solution-title>
> >
> > 1. `glob.glob('**/*.csv')`
> > 2. `glob.glob('**/*.txt')`
> > 3. `glob.glob('**/e*')`
> {: .solution}
{: .question}

```python
# Try things out here!
```

# Pitfalls

Some analyses (especially simultaions) can be dependent on data input order or data sorting. This was recently seen in {% cite Bhandari_Neupane_2019 %} where the data files used were sorted one way on Windows, and another on Linux, resulting in different results for the same code and the same datasets! Yikes!

If you know your analyses are dependent on file ordering, then you can use `sorted()` to make sure the data is provided in a uniform way every time.

```python
print(sorted(glob.glob('**/a.csv')))
```

If you're not sure if your results will be dependent, you can try sorting anyway. Or better yet, randomising the list of inputs to make sure your code behaves properly in any scenario.


topic: data-science
tutorial: python-basics
title: Introduction to Python
difficulty: introductory
time_estimation: 5H
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 38ba164a33052724b5e4f6b854b231b2
last_modified: 2026-02-26T21:12:14.900757
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-basics/tutorial.html

content:
In this lesson, we will be using Python 3 with some of its most popular scientific libraries. We will be using JupyterNotebook, a Python interpreter that comes with everything we need for the lesson.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which is licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Overview

Python was developed by Guido van Rossum. Guido van Rossum started implementing Python in 1989. Python is a very simple programming language so even if you are new to programming, you can learn python without facing any issues.

Features of Python language:
1. **Readable**: Python is a very readable language.

2. **Easy to Learn**: Learning python is easy as this is a expressive and high level programming language, which means it is easy to understand the language and thus easy to learn.

3. **Cross platform**: Python is available and can run on various operating systems such as Mac, Windows, Linux, Unix etc. This makes it a cross platform and portable language.

4. **Open Source**: Python is a open source programming language.

5. **Large standard library**: Python comes with a large standard library that has some handy codes and functions which we can use while writing code in Python.

6. **Free**: Python is free to download and use. This means you can download it for free and use it in your application.

7. **Supports exception handling**: Python supports exception handling which means we can write less error prone code and can test various scenarios that can cause an exception later on.

8. **Automatic memory management**: Python supports automatic memory management which means the memory is cleared and freed automatically. You do not have to bother clearing the memory.


> <tip-title>Why Jupyter?</tip-title>
>
> JupyterLab is a User Interface including notebooks. A user can open several notebooks or files as tabs in the same window, like an IDE. JupyterNotebook is a web-based interactive computational environment for creating Jupyter notebook documents. It supports several languages like Python (IPython), Julia, R etc. and is largely used for data analysis, data visualization and further interactive, exploratory computing.
>
> JupyterNotebook has several advantages:
>
> - You can easily type, edit, and copy and paste blocks of code.
> - Tab complete allows you to easily access the names of things you are using and learn more about them.
> - It allows you to annotate your code with links, different sized text, bullets, etc. to make it more accessible to you and your collaborators.
> - It allows you to display figures next to the code that produces them to tell a complete story of the analysis.
>
> Each notebook contains one or more cells that contain code, text, or images. Each notebook can be exported (File, Export as, Executable script) as Python script that can be run from the command line.
>
> We will be using JupyterNotebook in Galaxy and as a result you need to frequently save the notebook in the workspace. This is both for good practice and to protect you in case you accidentally close the browser. Your environment will still run, so it will contain the last saved notebook you have.  Furthermore, you need to download a notebook, before you delete or close a notebook in your history or you will lose it.
{: .tip}


# Python Fundamentals

## Variables

Any Python interpreter can be used as a calculator:

```python
3 + 5 * 4
```

This is great but not very interesting. To do anything useful with data, we need to assign its value to a variable. In Python, we can assign a value to a variable, using the equals sign ```=```. For example, we can track the weight of a patient who weighs 60 kilograms by assigning the value 60 to a variable ```weight_kg```:

```python
weight_kg = 60
```

From now on, whenever we use `weight_kg`, Python will substitute the value we assigned to it.

In Python, variable names:

- can include letters, digits, and underscores
- cannot start with a digit
- are case sensitive

This means that, for example:

- `weight0` is a valid variable name, whereas `0weight` is not
- `weight` and `Weight` are different variables



### Types of data

Python knows various types of data. Three common ones are:

- integer numbers
- floating point numbers, and
- strings.

In the example above, variable `weight_kg` has an integer value of `60`. If we want to more precisely track the weight of our patient, we can use a floating point value by executing:

```python
weight_kg = 60.3
```

To create a string, we add single or double quotes around some text. To identify and track a patient throughout our study, we can assign each person a unique identifier by storing it in a string:

```python
patient_id = '001'
```

### Using Variables in Python

Once we have data stored with variable names, we can make use of it in calculations. We may want to store our patient’s weight in pounds as well as kilograms:

```python
weight_lb = 2.2 * weight_kg
```

We might decide to add a prefix to our patient identifier:

```python
patient_id = 'inflam_' + patient_id
```

### Variables Persist Between Cells
Be aware that it is the order of execution of cells that is important in a Jupyter notebook, not the order in which they appear. Python will remember all the code that was run previously, including any variables you have defined, irrespective of the order in the notebook. Therefore if you define variables lower down the notebook and then (re)run cells further up, those defined further down will still be present. As an example, create two cells with the following content, in this order:

```python
print(myval)
myval = 1
```
If you execute this in order, the first cell will give an error. However, if you run the first cell after the second cell, it will print out 1. To prevent confusion, it can be helpful to use the Kernel -> Restart & Run All option which clears the interpreter and runs everything from a clean slate going top to bottom.


> <question-title>Variables and values</question-title>
>
> What values do the variables `mass` and `age` have after each of the following statements? Test your answer by executing the lines.
> ```
> mass = 47.5
> age = 122
> mass = mass * 2.0
> age = age - 20
> ```
>
> > <solution-title></solution-title>
> >
> > `mass` holds a value of 47.5, `age` does not exist
> > `mass` still holds a value of 47.5, `age` holds a value of 122
> > `mass` now has a value of 95.0, `age`'s value is still 122
> > `mass` still has a value of 95.0, `age` now holds 102
> >
> {: .solution}
{: .question}


> <question-title>Variables</question-title>
>
> Python allows you to assign multiple values to multiple variables in one line by separating the variables and values with commas. What does the following program print out?
> ```
> first, second = 'Grace', 'Hopper'
> third, fourth = second, first
> print(third, fourth)
> ```
>
> > <solution-title></solution-title>
> >
> > Hopper Grace
> >
> {: .solution}
{: .question}


> <question-title>Variables and data types</question-title>
>
> What are the data types of the following variables?
>
> ```
> planet = 'Earth'
> apples = 5
> distance = 10.5
> ```
>
> > <solution-title></solution-title>
> >
> > ```
> > type(planet)
> > type(apples)
> > type(distance)
> > ```
> >
> {: .solution}
{: .question}



## Lists

Lists are built into the language so we do not have to load a library to use them. We create a list by putting values inside square brackets and separating the values with commas:

```python
odds = [1, 3, 5, 7]
print('odds are:', odds)
```


We can access elements of a list using indices – numbered positions of elements in the list. These positions are numbered starting at 0, so the first element has an index of 0.

```python
print('first element:', odds[0])
print('last element:', odds[3])
print('"-1" element:', odds[-1])
```

Yes, we can use negative numbers as indices in Python. When we do so, the index `-1` gives us the last element in the list, `-2` the second to last, and so on. Because of this, `odds[3]` and `odds[-1]` point to the same element here.

There is one important difference between lists and strings: we can change the values in a list, but we cannot change individual characters in a string. For example:

```python
names = ['Curie', 'Darwing', 'Turing']  # typo in Darwin's name
print('names is originally:', names)
names[1] = 'Darwin'  # correct the name
print('final value of names:', names)
```


works, but the following does not:

```python
name = 'Darwin'
name[0] = 'd'
```

Lists in Python can contain elements of different types. Example:

```python
sample_ages = [10, 12.5, 'Unknown']
```

There are many ways to change the contents of lists besides assigning new values to individual elements:

We can append new elements to a list

```python
odds.append(11)
print('odds after adding a value:', odds)
```

We can pop an element off the end of a list

```python
removed_element = odds.pop(0)
print('odds after removing the first element:', odds)
print('removed_element:', removed_element)
```

Or we can reverse the list

```python
odds.reverse()
print('odds after reversing:', odds)
```


### Mutable vs Immutable data
Data which can be modified in place is called mutable, while data which cannot be modified is called immutable. Strings and numbers are immutable. This does not mean that variables with string or number values are constants, but when we want to change the value of a string or number variable, we can only replace the old value with a completely new value.

Lists and arrays, on the other hand, are mutable: we can modify them after they have been created. We can change individual elements, append new elements, or reorder the whole list. For some operations, like sorting, we can choose whether to use a function that modifies the data in-place or a function that returns a modified copy and leaves the original unchanged.

Be careful when modifying data in-place. If two variables refer to the same list, and you modify the list value, it will change for both variables!

```python
salsa = ['peppers', 'onions', 'cilantro', 'tomatoes']
my_salsa = salsa        # <-- my_salsa and salsa point to the *same* list data in memory
salsa[0] = 'hot peppers'
print('Ingredients in my salsa:', my_salsa)
```

If you want variables with mutable values to be independent, you must make a copy of the value when you assign it.

```python
salsa = ['peppers', 'onions', 'cilantro', 'tomatoes']
my_salsa = salsa.copy()        # <-- makes a *copy* of the list
salsa[0] = 'hot peppers'
print('Ingredients in my salsa:', my_salsa)
```


Because of pitfalls like this, code which modifies data in place can be more difficult to understand. However, it is often far more efficient to modify a large data structure in place than to create a modified copy for every small change. You should consider both of these aspects when writing your code.

### Nested lists
Since a list can contain any Python variables, it can even contain other lists.

For example, we could represent the products in the shelves of a small grocery shop:

```python
x = [['pepper', 'zucchini', 'onion'],
   ['cabbage', 'lettuce', 'garlic'],
   ['apple', 'pear', 'banana']]
```

Here is an example of how indexing a list of lists `x` works:

```python
print([x[0]])
print(x[0])
print(x[0][0])
```


### Slicing
Subsets of lists can be accessed by specifying ranges of values in brackets. This is commonly referred to as “slicing” the list.

```python
chromosomes = ['X', 'Y', '2', '3', '4']
autosomes = chromosomes[2:5]
print('autosomes:', autosomes)
last = chromosomes[-1]
print('last:', last)
```


> <question-title>Negative indices</question-title>
>
> Use slicing to access only the last four characters of a string or entries of a list.
>
> ```
> string_for_slicing = 'Observation date: 02-Feb-2013'
> list_for_slicing = [['fluorine', 'F'], ['chlorine', 'Cl'], ['bromine', 'Br'], ['iodine', 'I'], ['astatine', 'At']]
> ```
> Your output should be:
> '2013'
> [['chlorine', 'Cl'], ['bromine', 'Br'], ['iodine', 'I'], ['astatine', 'At']]
>
> Would your solution work regardless of whether you knew beforehand the length of the string or list (e.g. if you wanted to apply the solution to a set of lists of different lengths)? If not, try to change your approach to make it more robust.
> Hint: Remember that indices can be negative as well as positive.
>
> > <solution-title></solution-title>
> >
> > Use negative indices to count elements from the end of a container (such as list or string):
> > ```
> > string_for_slicing[-4:]
> > list_for_slicing[-4:]
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Slicing</question-title>
>
> So far we’ve seen how to use slicing to take single blocks of successive entries from a sequence. But what if we want to take a subset of entries that aren’t next to each other in the sequence?
>
> You can achieve this by providing a third argument to the range within the brackets, called the step size. The example below shows how you can take every third entry in a list:
>
> ```
> primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37]
> subset = primes[0:12:3]
> print('subset', subset)
> ```
>
> Notice that the slice taken begins with the first entry in the range, followed by entries taken at equally-spaced intervals (the steps) thereafter. What if you wanted to begin the subset with the third entry? Use the previous example to write your solution that gives the following output.
>
> > <solution-title></solution-title>
> >
> > You would need to specify that as the starting point of the sliced range:
> >
> > ```
> > subset = primes[2:12:3]
> > print('subset', subset)
> > ```
> >
> {: .solution}
{: .question}



## Strings

### Use an index to get a single character from a string.

The characters (individual letters, numbers,and so on) in a string are ordered. For example, the string 'AB' is not the same as 'BA'. Because of this ordering, we can treat the string as a list of characters. Each position in the string (first, second, etc.) is given a number. This number is called an index or sometimes a subscript. Indices are numbered from 0.You can use the position’s index in square brackets to get the character at that position.

```python
atom_name = 'helium'
print(atom_name[0])
```

### Use a slice to get a substring.
A part of a string is called a substring. A substring can be as short as a single character. An item in a list is called an element. Whenever we treat a string as if it were a list, the string’s elements are its individual characters. A slice is a part of a string (or, more generally, any list-like thing). We take a slice by using [start:stop], where start is replaced with the index of the first element we want and stop is replaced with the index of the element just after the last element we want. Mathematically, you might say that a slice selects [start:stop). The difference between stop and start is the slice’s length. Taking a slice does not change the contents of the original string. Instead, the slice is a copy of part of the original string.

```python
atom_name = 'sodium'
print(atom_name[0:3])
```

You can use the built-in function `len` to find the length of a string.

```python
print(len('helium'))
```

Nested functions are evaluated from the inside out, like in mathematics.

> <question-title>Slicing</question-title>
>
> Use what you have learnt in this tutorial to answer the following questions:
> a. What does the following program print?
> ```
> atom_name = 'carbon'
> print('atom_name[1:3] is:', atom_name[1:3])
> ```
> b. What does `thing[low:high]` do?
> c. What does `thing[low:]` (without a value after the colon) do?
> d. What does `thing[:high]` (without a value before the colon) do?
> e. What does `thing[:]` (just a colon) do?
> f. What does `thing[number:some-negative-number]` do?
> g. What happens when you choose a high value which is out of range? (i.e., try `atom_name[0:15]`)
>
> > <solution-title></solution-title>
> >
> > a. `atom_name[1:3] is: ar`
> > b. `thing[low:high]` returns a slice from low to the value before high
> > c. `thing[low:]` returns a slice from low all the way to the end of thing
> > d. `thing[:high]` returns a slice from the beginning of thing to the value before high
> > e. `thing[:]` returns all of thing
> > f. `thing[number:some-negative-number]` returns a slice from number to some-negative-number values from the end of thing
> > g. If a part of the slice is out of range, the operation does not fail. `atom_name[0:15]` gives the same result as `atom_name[0:]`.
> >
> {: .solution}
{: .question}



### You can use the “+” and “*” operators on strings/lists.

“Adding” character strings concatenates them.

```python
full_name = 'Ahmed' + ' ' + 'Walsh'
print(full_name)
```


Multiplying a character string by an integer "N" creates a new string that consists of that character string repeated N times.

Since multiplication is repeated addition.

```python
separator = '=' * 10
print(separator)
```


The same rules apply for lists. Consider the following example:

```python
counts = [2, 4, 6, 8, 10]
repeats = counts * 2
print(repeats)
```


It’s equivalent to:

```python
counts + counts
```

Strings have a length (but numbers don’t).The built-in function `len` counts the number of characters in a string.

```python
print(len(full_name))
```


But numbers don’t have a length (not even zero). For example, the following command returns an error message.

```python
print(len(52))
```


## Type conversion

Python converts automatically integers to floats, when needed, but you must convert numbers to strings or vice versa when operating on them.
It is not allowed to add numbers and strings. For example `print(1 + '2')` is ambiguous: should `1 + '2'` be `3` or `'12'`?

Some types can be converted to other types by using the type name as a function.

```python
print(1 + int('2'))
print(str(1) + '2')
```

> <question-title>Data types</question-title>
>
> What type of value is 3.25 + 4?
>
> > <solution-title></solution-title>
> >
> > It is a float: integers are automatically converted to floats as necessary.
> {: .solution}
{: .question}

> <question-title>Division</question-title>
>
> In Python 3, the `//` operator performs integer (whole-number) floor division, the `/` operator performs floating-point division, and the `%` (or *modulo*) operator calculates and returns the remainder from integer division:
>
> > <code-in-title>Python</code-in-title>
> > ```
> > print('5 // 3:', 5 // 3)
> > print('5 / 3:', 5 / 3)
> > print('5 % 3:', 5 % 3)
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > 5 // 3: 1
> > 5 / 3: 1.6666666666666667
> > 5 % 3: 2
> > ```
> {: .code-out}
>
> If `num_subjects` is the number of subjects taking part in a study, and `num_per_survey` is the number that can take part in a single survey, write an expression that calculates the number of surveys needed to reach everyone once.
>
> > <solution-title></solution-title>
> >
> > We want the minimum number of surveys that reaches everyone once, which is the rounded up value of `num_subjects/ num_per_survey`. This is equivalent to performing a floor division with `//` and adding 1. Before the division we need to subtract 1 from the number of subjects to deal with the case where `num_subjects` is evenly divisible by `num_per_survey`.
> >
> > ```
> > num_subjects = 600
> > num_per_survey = 42
> > num_surveys = (num_subjects - 1) // num_per_survey + 1
> >
> > print(num_subjects, 'subjects,', num_per_survey, 'per survey:', num_surveys)
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Typecasting</question-title>
>
> Where reasonable, `float()` will convert a string to a floating point number, and `int()` will convert a floating point number to an integer:
>
> > <code-in-title>Python</code-in-title>
> > ```
> > print("string to float:", float("3.4"))
> > print("float to int:", int(3.4))
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > string to float: 3.4
> > float to int: 3
> > ```
> {: .code-out}
>
> If the conversion doesn’t make sense, however, an error message will occur.
> Given this information, what do you expect the following program to do? What does it actually do? Why do you think it does that?
> ```
> print("fractional string to int:", int("3.4"))
> ```
>
> > <solution-title></solution-title>
> >
> > Python 3 throws an error. If you ask Python to perform two consecutive typecasts, you must convert it explicitly in code.
> >
> > ```
> > int(float("3.4"))
> > ```
> >
> {: .solution}
{: .question}

> <question-title>Typecasting</question-title>
>
> Which of the following will return the floating point number `2.0`? Note: there may be more than one right answer.
> ```
> first = 1.0
> second = "1"
> third = "1.1"
> ```
>
> 1. `first + float(second)`
> 2. `float(second) + float(third)`
> 3. `first + int(third)`
> 4. `first + int(float(third))`
> 5. `int(first) + int(float(third))`
> 6. `2.0 * second`
>
> > <solution-title></solution-title>
> >
> > Answer: 1 and 4
> {: .solution}
{: .question}


> <question-title>Imaginary numbers</question-title>
>
> Python provides complex numbers, which are written as `1.0+2.0j`. If `val` is a complex number, its real and imaginary parts can be accessed using dot notation as `val.real` and `val.imag`.
>
> ```
> complex = 6 + 2j
> print(complex.real)
> print(complex.imag)
> ```
>
> Output:
> 6.0
> 2.0
>
> 1. Why do you think Python uses `j` instead of i for the imaginary part?
> 2. What do you expect `1+2j + 3` to produce?
> 3. What do you expect `4j` to be? What about `4 j` or `4 + j`?
>
> > <solution-title></solution-title>
> >
> >1. Standard mathematics treatments typically use i to denote an imaginary number. However, from media reports it was an early convention established from electrical engineering that now presents a technically expensive area to change.
> >2. `(4+2j)`
> >3. `4j` and Syntax Error: invalid syntax. In the latter cases, `j` is considered a variable and the statement depends on if `j` is defined and if so, its assigned value.
> >
> {: .solution}
{: .question}


## Built-in Python functions

To carry out common tasks with data and variables in Python, the language provides us with several built-in functions. To display information to the screen, we use the `print` function:

```python
print(weight_lb)
print(patient_id)
```


When we want to make use of a function, referred to as calling the function, we follow its name by parentheses. The parentheses are important: if you leave them off, the function doesn’t actually run! Sometimes you will include values or variables inside the parentheses for the function to use. In the case of `print`, we use the parentheses to tell the function what value we want to display. `print` automatically puts a single space between outputs to separate them and wraps around to a new line at the end.

We can display multiple things at once using only one `print` call:

```python
print(patient_id, 'weight in kilograms:', weight_kg)
```


We can also call a function inside of another function call. For example, Python has a built-in function called `type` that tells you a value’s data type:

```python
print(type(60.3))
print(type(patient_id))
```


Moreover, we can do arithmetic with variables right inside the `print` function:

```python
print('weight in pounds:', 2.2 * weight_kg)
```

The above command, however, did not change the value of `weight_kg`:

```python
print(weight_kg)
```

To change the value of the `weight_kg` variable, we have to assign `weight_kg` a new value using the equals `=` sign:

```python
weight_kg = 65.0
print('weight in kilograms is now:', weight_kg)
```

A function may take zero or more arguments.  An *argument* is a value passed into a function. You must always use parentheses, even if they’re empty, so that Python knows a function is being called.

Every function call produces some result. If the function doesn’t have a useful result to return, it usually returns the special value `None`. `None` is a Python object that stands in anytime there is no value.

```python
result = print('example')
print('result of print is', result)
```

Commonly-used built-in functions include `max`, `min`, and `round`. `max` and `min` work on character strings as well as numbers. From “larger” and “smaller”, they use the order: (0-9, A-Z, a-z) to compare letters.
```python
print(max(1, 2, 3))
print(min('a', 'A', '0'))
```

Functions may have default values for some arguments. `round` will round off a floating-point number. By default, it rounds to zero decimal places.
```python
round(3.712)
```

We can specify the number of decimal places we want.
```python
round(3.712, 1)
```


### Functions attached to objects are called methods

Methods have parentheses like functions, but come after the variable. Some methods are used for internal Python operations, and are marked with double underlines.

```python
my_string = 'Hello world!'  # creation of a string object

print(len(my_string))       # the len function takes a string as an argument and returns the length of the string

print(my_string.swapcase()) # calling the swapcase method on the my_string object

print(my_string.__len__())  # calling the internal __len__ method on the my_string object, used by len(my_string)
```


You might even see them chained together. They operate left to right.

```python
print(my_string.isupper())          # Not all the letters are uppercase
print(my_string.upper())            # This capitalizes all the letters

print(my_string.upper().isupper())  # Now all the letters are uppercase
```

### Use the built-in function `help` to get help for a function.

Every built-in function has online documentation.

```python
help(round)
```


## Conditionals

An if statement (more properly called a conditional statement) controls whether some block of code is executed or not.
The first line opens with if and ends with a colon and the block of code to be executed is indented. An example is showed below.

```python
num = 37
if num > 100:
    print('greater')
else:
    print('not greater')
print('done')
```

If the expression that follows the if statement is true, the body of the `if` (i.e., the set of lines indented underneath it) is executed, and `“greater”` is printed. If it is false, the body of the `else` is executed instead, and `“not greater”` is printed. Only one or the other is ever executed before continuing on with program execution to print `“done”`:

![A flow chart diagram is shown with a decision box of number > 100. For true it branches to print 'greater'. For false it branches to print 'not greater'. Both branches re-connect to print done.](../../images/python-basics/Figure5_Conditionals.png)

Conditional statements don’t have to include an `else`.  If there isn’t one, Python simply does nothing if the expression is false:

```python
num = 53
print('before conditional...')
if num > 100:
    print(num, 'is greater than 100')
print('...after conditional')
```

We can also chain several expressions together using `elif`, which is short for “else if”. The following Python code uses `elif` to print the sign of a number.

```python
num = -3

if num > 0:
    print(num, 'is positive')
elif num == 0:
    print(num, 'is zero')
else:
    print(num, 'is negative')
```


### Comparison operators in Python
The operators used for comparing values in conditionals are the following:

- `>` : greater than
- `<` : less than
- `==` : equal to
- `!=` : does not equal
- `>=` : greater than or equal to
- `<=` : less than or equal to


### Logical operators in Python
We can also combine expressions using `and` and `or`. `and` is only true if both parts are true:

```python
if (1 > 0) and (-1 >= 0):
    print('both parts are true')
else:
    print('at least one part is false')
```


while `or` is true if at least one part is true:

```python
if (1 < 0) or (1 >= 0):
    print('at least one test is true')
```


### True and False
`True` and `False` are special words in Python called booleans, which represent truth values. A statement such as `1 < 0` returns the value `False`, while `-1 < 0` returns the value `True`.


> <question-title>Conditionals</question-title>
>
> What does this program print?
>
> ```
> pressure = 71.9
> if pressure > 50.0:
>     pressure = 25.0
> elif pressure <= 50.0:
>     pressure = 0.0
> print(pressure)
> ```
>
> > <solution-title></solution-title>
> >
> > 25
> {: .solution}
{: .question}


> <question-title>Conditionals</question-title>
>
> Write some conditions that print `True` if the variable `a` is within 10% of the variable `b` and `False` otherwise. Compare your implementation with your partner’s: do you get the same answer for all possible pairs of numbers?
> Hint: There is a built-in function `abs()` that returns the absolute value of a number.
>
> > <solution-title></solution-title>
> >
> > ```
> > a = 5
> > b = 5.1
> > if abs(a - b) <= 0.1 * abs(b):
> >    print('True')
> > else:
> >    print('False')
> > ```
> >
> {: .solution}
{: .question}


## For loops

Doing calculations on the values in a list one by one is very time consuming.

```python
odds = [1, 3, 5, 7, 9, 11]
print(odds[0])
print(odds[1])
print(odds[2])
print(odds[3])
print(odds[4])
print(odds[5])

```


A for loop tells Python to execute some statements once for each value in a list, a character string, or some other collection.
“for each thing in this group, do these operations”. The for loop equivalent to the previous code is:

```python
for num in odds:
    print(num)
```
The improved version uses a for loop to repeat an operation — in this case, printing — once for each thing in a sequence. The general form of a loop is:

```python
for variable in collection:
    # do things using variable, such as print
```

Using the odds example above, the loop might look like this:
![A cartoon schematic shows a  small array named odds with the odd numbers from 1 to 11. A for loop is written "for num in odds" and for each of these items in the array print(num) happens, and the output is shown below, the odd numbers printed out.](../../images/python-basics/Figure6_For_loop.png)

where each number (`num`) in the variable odds is looped through and printed one number after another. The other numbers in the diagram denote which loop cycle the number was printed in (1 being the first loop cycle, and 6 being the final loop cycle).

We can call the loop variable anything we like, but there must be a colon at the end of the line starting the loop, and we must indent anything we want to run inside the loop. Unlike many other languages, there is no command to signify the end of the loop body (e.g. end for); what is indented after the for statement belongs to the loop. Python uses indentation to show nesting. Any consistent indentation is legal, but almost everyone uses four spaces.

When looping through a list, the position index and corresponding value can be retrieved at the same time using the `enumerate()` function.

```python
for i, v in enumerate(['tic', 'tac', 'toe']):
... print(i, v)
```


To loop over two or more lists at the same time, the entries can be paired with the `zip()` function.

```python
questions = ['name', 'quest', 'favorite color']
answers = ['lancelot', 'the holy grail', 'blue']
for q, a in zip(questions, answers):
   print('What is your {0}?  It is {1}.'.format(q, a))
```


We can choose any name we want for variables. It is a good idea to choose variable names that are meaningful, otherwise it would be more difficult to understand what the loop is doing.

Here’s another loop that repeatedly updates a variable:

```python
length = 0
names = ['Curie', 'Darwin', 'Turing']
for value in names:
    length = length + 1
print('There are', length, 'names in the list.')
print('After the loop, name is', name)
```

It is worth tracing the execution of this little program step by step. Since there are three names in `names`, the statement on line 4 will be executed three times. The first time around, `length` is `0` (the value assigned to it on line 1) and `value` is `Curie`. The statement adds 1 to the old value of `length`, producing 1, and updates `length` to refer to that new value. The next time around, `value` is `Darwin` and `length` is `1`, so `length` is updated to be `2`. After one more update, `length` is `3`; since there is nothing left in `names` for Python to process, the loop finishes and the print function on line 5 tells us our final answer.

Note that a loop variable is a variable that is being used to record progress in a loop. It still exists after the loop is over, stores the value assigned to it last, and we can re-use variables previously defined as loop variables as well.


> <question-title>range of numbers</question-title>
>
> Python has a built-in function called `range()` that generates a sequence of numbers. `range` can accept 1, 2, or 3 parameters.
> If one parameter is given, `range` generates a sequence of that length, starting at zero and incrementing by 1. For example, `range(3)` produces the numbers `0, 1, 2`.
> If two parameters are given, `range` starts at the first and ends just before the second, incrementing by one. For example, `range(2, 5)` produces `2, 3, 4`.
> If `range` is given 3 parameters, it starts at the first one, ends just before the second one, and increments by the third one. For example, `range(3, 10, 2)` produces `3, 5, 7, 9`.
> Using `range`, write a loop that uses `range` to print the first 3 natural numbers:
> ```
> 1
> 2
> 3
> ```
>
> > <solution-title></solution-title>
> >
> >  ```
> >  for i in range(1, 4):
> >     print(i)
> >  ```
> {: .solution}
{: .question}


> <question-title>Number of iterations</question-title>
>
> Given the following loop:
> ```
> word = 'oxygen'
> for char in word:
>    print(char)
> ```
> How many times is the for loop executed?
>
> > <solution-title></solution-title>
> >
> > The body of the loop is executed 6 times.
> {: .solution}
{: .question}


> <question-title>Exponentiation</question-title>
>
> Exponentiation is built into Python:
>
> ```
> print(5 ** 3)
> ```
>
> Output:
> 125
>
> Write a loop that calculates the same result as `5 ** 3` using multiplication (and without exponentiation).
>
> > <solution-title></solution-title>
> >
> > ```
> > result = 1
> > for number in range(0, 3):
> >     result = result * 5
> > print(result)
> > ```
> {: .solution}
{: .question}


> <question-title>Iterations over a list</question-title>
>
> Write a loop that calculates the sum of elements in a list by adding each element and printing the final value, so `[124, 402, 36]` prints `562`
>
> > <solution-title></solution-title>
> >
> > ```
> > numbers = [124, 402, 36]
> > summed = 0
> > for num in numbers:
> >     summed = summed + num
> > print(summed)
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Polynomial</question-title>
>
> Suppose you have encoded a polynomial as a list of coefficients in the following way: the first element is the constant term, the second element is the coefficient of the linear term, the third is the coefficient of the quadratic term, etc.
>
> ```
> x = 5
> coefs = [2, 4, 3]
> y = coefs[0] * x**0 + coefs[1] * x**1 + coefs[2] * x**2
> print(y)
> ```
>
> Output:
> 97
>
> Write a loop using `enumerate(coefs)` which computes the value `y` of any polynomial, given `x` and `coefs`.
>
> > <solution-title></solution-title>
> >
> > ```
> > y = 0
> > for idx, coef in enumerate(coefs):
> >     y = y + coef * x**idx
> > ```
> >
> {: .solution}
{: .question}


> <question-title>For loops and conditionals</question-title>
>
> Fill in the blanks so that this program creates a new list containing zeroes where the original list’s values were negative and ones where the original list’s values were positive.
> ```
> original = [-1.5, 0.2, 0.4, 0.0, -1.3, 0.4]
> result = ____
> for value in original:
>     if ____:
>         result.append(0)
>     else:
>         ____
>     print(result)
> ```
>
> Output:
> [0, 1, 1, 1, 0, 1]
>
> > <solution-title></solution-title>
> >
> > ```
> > original = [-1.5, 0.2, 0.4, 0.0, -1.3, 0.4]
> > result = []
> > for value in original:
> >     if value < 0:
> >         result.append(0)
> >     else:
> >         result.append(1)
> >     print(result)
> > ```
> >
> {: .solution}
{: .question}



## While loops
With the while loop we can execute a set of statements as long as an expression is true. The following example prints `i` as long as `i` is less than 6:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  print(odds[i])
  i = i + 1
```


Remember to increment `i`, or else the loop will continue forever. The while loop requires relevant variables to be ready, in this example we need to define an indexing variable, `i`, which we set to 0.

With the `break` statement we can stop the loop even if the while condition is true:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  print(odds[i])
  if odds[i] == 3:
    break
  i = i + 1
```


With the `continue` statement we can stop the current iteration, and continue with the next:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  i = i + 1
  if odds[i] == 3:
    continue
  print(odds[i])

```


With the `else` statement we can run a block of code once when the condition no longer is true:

```python
odds = [1, 3, 5, 7, 9, 11]
i = 0
while odds[i] < 6:
  print(odds[i])
  i = i + 1
else:
  print("condition is no longer True")
```



## Create Functions
Human beings can only keep a few items in working memory at a time. Breaking down larger/more complicated pieces of code in functions helps in understanding and using it. A function can be re-used. Write one time, use many times.

```python
def fahr_to_celsius(temp):
    return ((temp - 32) * (5/9))
```

![The above function fahr to celsius is shown except annotated. def is labelled "def statement", fahr_to_celsius is noted as the function name. Inside parentheses is temp and an arrow shows it is called parameter names. The next line which is indented is annotated as the function body which has a return statement and the calculation from above.](../../images/python-basics/Figure7_Functions.png)

The function definition opens with the keyword `def` followed by the name of the function `fahr_to_celsius` and a parenthesized list of parameter names `temp`. The body of the function — the statements that are executed when it runs — is indented below the definition line. The body concludes with a `return` keyword followed by the return value.

When we call the function, the values we pass to it are assigned to those variables so that we can use them inside the function. Inside the function, we use a return statement to send a result back to whoever asked for it.

Let’s try running our function.

```python
fahr_to_celsius(32)
print('freezing point of water:', fahr_to_celsius(32), 'C')
print('boiling point of water:', fahr_to_celsius(212), 'C')
```


We’ve successfully called the function that we defined, and we have access to the value that we returned.

Now that we’ve seen how to turn Fahrenheit into Celsius, we can also write the function to turn Celsius into Kelvin:

```python
def celsius_to_kelvin(temp_c):
    return temp_c + 273.15

print('freezing point of water in Kelvin:', celsius_to_kelvin(0.))
```


What about converting Fahrenheit to Kelvin? We could write out the formula, but we don’t need to. Instead, we can compose the two functions we have already created:

```python
def fahr_to_kelvin(temp_f):
    temp_c = fahr_to_celsius(temp_f)
    temp_k = celsius_to_kelvin(temp_c)
    return temp_k

print('boiling point of water in Kelvin:', fahr_to_kelvin(212.0))
```


This is our first taste of how larger programs are built: we define basic operations, then combine them in ever-larger chunks to get the effect we want. Real-life functions will usually be larger than the ones shown here — typically half a dozen to a few dozen lines — but they shouldn’t ever be much longer than that, or the next person who reads it won’t be able to understand what’s going on.

### Variable Scope
In composing our temperature conversion functions, we created variables inside of those functions, `temp`, `temp_c`, `temp_f`, and `temp_k`. We refer to these variables as local variables because they no longer exist once the function is done executing. If we try to access their values outside of the function, we will encounter an error:

```python
print('Again, temperature in Kelvin was:', temp_k)
```

If you want to reuse the temperature in Kelvin after you have calculated it with fahr_to_kelvin, you can store the result of the function call in a variable:

```python
temp_kelvin = fahr_to_kelvin(212.0)
print('temperature in Kelvin was:', temp_kelvin)
```


### Defining Default parameters
If we usually want a function to work one way, but occasionally need it to do something else, we can allow people to pass a parameter when they need to but provide a default to make the normal case easier. The example below shows how Python matches values to parameters:

```python
def display(a=1, b=2, c=3):
    print('a:', a, 'b:', b, 'c:', c)

print('no parameters:')
display()
print('one parameter:')
display(55)
print('two parameters:')
display(55, 66)
```


As this example shows, parameters are matched up from left to right, and any that haven’t been given a value explicitly get their default value. We can override this behavior by naming the value as we pass it in:

```python
print('only setting the value of c')
display(c=77)
```

> <question-title>Variable scope</question-title>
>
> What does the following piece of code display when run — and why?
>
> ```
> f = 0
> k = 0
>
> def f2k(f):
>     k = ((f - 32) * (5.0 / 9.0)) + 273.15
>     return k
>
> print(f2k(8))
> print(f2k(41))
> print(f2k(32))
> print(k)
> ```
>
> > <solution-title></solution-title>
> >
> > Output:
> > 259.81666666666666
> > 278.15
> > 273.15
> > 0
> >
> > `k` is 0 because the `k` inside the function `f2k` doesn’t know about the `k` defined outside the function. When the `f2k` function is called, it creates a local variable `k`. The function does not return any values and does not alter `k` outside of its local copy. Therefore the original value of `k` remains unchanged.
> >
> {: .solution}
{: .question}



## Libraries

A library is a collection of files (called modules) that contains functions for use by other programs. It may also contain data values (e.g., numerical constants) and other things. A library’s contents are supposed to be related, but there’s no way to enforce that. The Python standard library is an extensive suite of modules that comes with Python itself. Many additional libraries are available from PyPI (the Python Package Index).

### Libraries and modules
A library is a collection of modules, but the terms are often used interchangeably, especially since many libraries only consist of a single module, so don’t worry if you mix them.

### A program must import a library module before using it.
You can use `import` to load a library module into a program’s memory, then refer to things from the `module as module_name.thing_name`. Python uses `.` to mean “part of”. For example, using `math`, one of the modules in the standard library:

```python
import math

print('pi is', math.pi)
print('cos(pi) is', math.cos(math.pi))
```


You can use `help` to learn about the contents of a library module. it works just like help for a function.
```python
help(math)
```
You can import specific items from a library module to shorten programs. You can use `from ... import ...` to load only specific items from a library module. Then refer to them directly without library name as prefix.

```python
from math import cos, pi

print('cos(pi) is', cos(pi))
```


You can create an alias for a library module when importing it to shorten programs. Use `import ... as ...` to give a library a short alias while importing it. Then refer to items in the library using that shortened name.

```python
import math as m

print('cos(pi) is', m.cos(m.pi))
```



# Programming style
A consistent coding style helps others (including our future selves) read and understand code more easily. Code is read much more often than it is written, and as the [Zen of Python](https://www.python.org/dev/peps/pep-0020/) states, “Readability counts”. Python proposed a standard style through one of its first Python Enhancement Proposals (PEP), [PEP8](https://www.python.org/dev/peps/pep-0008/).

Some points worth highlighting:

* document your code and ensure that assumptions, internal algorithms, expected inputs, expected outputs, etc., are clear
* use clear, semantically meaningful variable names
* use white-space, not tabs, to indent lines (tabs can cause problems across different text editors, operating systems, and version control systems)

# Python community

Python supports a large and diverse community across academia and industry.

* The [Python 3 documentation](https://docs.python.org/3/) covers the core language and the standard library.
* [PyCon](https://pycon.org/) is the largest annual conference for the Python community.
* [SciPy](https://scipy.org/) is a rich collection of scientific utilities. It is also the name of [a series of annual conferences](https://conference.scipy.org/).
* [Jupyter](https://jupyter.org/) is the home of Project Jupyter.
* [Pandas](https://pandas.pydata.org/) is the home of the Pandas data library.
* [Stack Overflow’s](https://stackoverflow.com/questions/tagged/python?tab=Votes) general Python section can be helpful, as well as the sections on [NumPy](https://stackoverflow.com/questions/tagged/numpy?tab=Votes), [SciPy](https://stackoverflow.com/questions/tagged/scipy?tab=Votes), and [Pandas](https://stackoverflow.com/questions/tagged/pandas?tab=Votes).


