# tutorials

topic: data-science
tutorial: gnmx-lecture4
title: Introduction to sequencing with Python (part three)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 9e181551cccbb124c8f0b94fef0ecb67
last_modified: 2026-05-15T22:28:11.070988
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture4/tutorial.html

content:
[![XKCD 2483](https://imgs.xkcd.com/comics/seven.png)](https://xkcd.com/2483)


Preclass prep: Chapters [5](https://greenteapress.com/thinkpython2/html/thinkpython2009.html) and [7](https://greenteapress.com/thinkpython2/html/thinkpython2011.html) from "Think Python"

> This material uses examples from notebooks developed by [Ben Langmead](https://langmead-lab.org/teaching-materials/)
{: .quote cite="https://langmead-lab.org/teaching-materials" }

# Prep

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Within JupyterLab start a new Python3 notebook
3. Open [this page](http://cs1110.cs.cornell.edu/tutor/#mode=edit) in a new browser tab

# Lists: Dynamic programming in sequence alignment

## Dynamic programming matrix as a 2D list

An excellent way to illustrate the utility of lists is to implement a dynamic programming algorithm for sequence alignment. Suppose we have two sequences that deliberately have different lengths:

$$
\texttt{G C T A T A C}$
$$

and 

$$
\texttt{G C G T A T G C}$
$$

Let's represent them as the following matrix where the first character $$\epsilon$$ represents an empty string:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon \\
\hline
G\\
\hline
 C\\
 \hline
G\\
\hline
T\\
\hline
A\\
 \hline
T\\
\hline
G\\
\hline
C
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

In this matrix, the cells are addressed as shown below. They filled in using the following logic:

$$
D[i,j] = min\begin{cases} 
          \color{green}{D[i-1,j] + 1} & \\
          \color{blue}{D[i,j-1] + 1} & \\
          \color{red}{D[i-1,j-1] + \delta(x[i-1],y[j-1])}
             \end{cases}
$$

where $$\color{green}{green}$$ is *upper* cell, $$\color{blue}{blue}$$ is *left* cell, and $$\color{red}{red}$$ is *upper-left* cell:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
            \hline
          \epsilon &  &  &  &  &  &  &  & \\
          \hline
          G & \\
          \hline
          C &  & & \color{red}{D[2,2]} & \color{green}{D[2,3]}\\
          \hline
          G &  & & \color{blue}{D[3,2]} & D[3,3]\\
          \hline
          T & \\
          \hline
          A & \\
          \hline
          T & \\
          \hline
          G & \\
          \hline
          C & 
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

## Initializing the matrix

Let's initialize the first column and first row of the matrix. Because the distance between a string and an empty string is equal to the length of the
string (e.g., a distance between, say, string $$\texttt{TCG}$$ and an empty string is 3) this resulting matrix will look like this:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
            \hline
          \epsilon & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7\\
          \hline
          G & 1\\
          \hline
          C & 2\\
          \hline
          G & 3\\
          \hline
          T & 4\\
          \hline
          A & 5\\
          \hline
          T & 6\\
          \hline
          G & 7\\
          \hline
          C & 8
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

This can be achieved with the following code:

```python
D = np.zeros((len(x)+1, len(y)+1), dtype=int)
D[0, 1:] = range(1, len(y)+1)
D[1:, 0] = range(1, len(x)+1)
```

## Filling the matrix

Now we can fill the entire matrix by using two nested loops: one iterating over $$i$$ coordinate (sequence $$x$$) and the other iterating over $$j$$ coordinate (sequence $$y$$):

```python
for i in range(1, len(x)+1):
    for j in range(1, len(y)+1):
        delt = 1 if x[i-1] != y[j-1] else 0
        D[i, j] = min(D[i-1, j-1]+delt, D[i-1, j]+1, D[i, j-1]+1)
```

Let's start with the cell between $$\texttt{G}$$ and $$\texttt{G}$$. Recall that:

$$
D[i,j] = min\begin{cases} 
          \color{green}{D[i-1,j] + 1} & \\
          \color{blue}{D[i,j-1] + 1} & \\
          \color{red}{D[i-1,j-1] + \delta(x[i-1],y[j-1])}
             \end{cases}
$$


where $$\delta(x[i-1],y[j-1]) = 0$$ if $$x[i-1] = y[j-1]$$ and $$1$$ otherwise. And now let's color each of the cells corresponding to each part of the above expression:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon & \color{red}0 & \color{green}1 & 2 & 3 & 4 & 5 & 6 & 7\\
\hline
G & \color{blue}1\\
          \hline
          C & 2\\
          \hline
          G & 3\\
          \hline
          T & 4\\
          \hline
          A & 5\\
          \hline
          T & 6\\
          \hline
          G & 7\\
          \hline
          C & 8
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

In this specific case:

$$
D[i,j] = min\begin{cases} 
          \color{green}{D[i-1,j] + 1}\ or\ 0+0=0 & \\
          \color{blue}{D[i,j-1] + 1}\ or\ 1+1=2 & \\
          \color{red}{D[i-1,j-1] + \delta(x[i-1],y[j-1])}\ or\ 1+1=2
             \end{cases}
$$


The minimum of 0, 2, and 2 will be 0, so we are putting zero into that cell:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon & \color{red}0 & \color{green}1 & 2 & 3 & 4 & 5 & 6 & 7\\
\hline
G & \color{blue}1 & \color{red}0\\
          \hline
          C & 2\\
          \hline
          G & 3\\
          \hline
          T & 4\\
          \hline
          A & 5\\
          \hline
          T & 6\\
          \hline
          G & 7\\
          \hline
          C & 8
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$

Using this logic we can fill the entire matrix like this:

$$
\begin{array}{ c | c | c | c | c | c | c}
& \epsilon & G & C & T & A & T & A & C\\
\hline
\epsilon & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7\\
\hline
                 G & 1 & 0 & 1 & 2 & 3 & 4 & 5 & 6\\
          \hline
                 C & 2 & 1 & 0 & 1 & 2 & 3 & 4 & 5\\
          \hline
                 G & 3 & 2 & 1 & 1 & 2 & 3 & 4 & 5\\
          \hline
                 T & 4 & 3 & 2 & 1 & 2 & 2 & 3 & 4\\
          \hline
                 A & 5 & 4 & 3 & 2 & 1 & 2 & 2 & 3\\
          \hline
                 T & 6 & 5 & 4 & 3 & 2 & 1 & 2 & 3\\
          \hline
                 G & 7 & 6 & 5 & 4 & 3 & 2 & 2 & 3\\
          \hline
                 C & 8 & 7 & 6 & 5 & 4 & 3 & 3 & \color{red}2
\end{array}
\\
\textbf{Note}: sequence\ \texttt{X}\ is\ vertical\ and\ sequence\ \texttt{Y}\ is\ horizontal.
$$


The lower rightmost cell highlighted in red is special. It contains the value for the edit distance between the two strings.
The following Python script implements this idea. You can see that it is essentially instantaneous:

```python
import numpy as np

def edDistDp(x, y):
    """ Calculate edit distance between sequences x and y using
        matrix dynamic programming.  Return matrix and distance. """
    D = np.zeros((len(x)+1, len(y)+1), dtype=int)
    D[0, 1:] = range(1, len(y)+1)
    D[1:, 0] = range(1, len(x)+1)
    for i in range(1, len(x)+1):
        for j in range(1, len(y)+1):
            delt = 1 if x[i-1] != y[j-1] else 0
            D[i, j] = min(D[i-1, j-1]+delt, D[i-1, j]+1, D[i, j-1]+1)
    return D,D[len(x),len(y)]
```
A graphical representation of the matrix between `GCGTATGCACGC` and `GCTATGCCACGC` looks like this:

![A DP matrix with Seaborn](https://i.imgur.com/veMfPFt.png)

This image is generated using [Seaborn](https://seaborn.pydata.org/index.html) package using matrix directly:

```python
sns.heatmap(D,annot=True,cmap="crest")
```


------

# Dictionaries: Translating sequences

Perhaps the best way to demonstrate the utility of dictionaries is by using DNA-to-Protein translation as an example. 

## Using dictionaries to translate DNA

The following dictionary maps codons to corresponding amino acid translations. In this case, codon is the *key* and amino acid is the *value*:


```python
table = {
        'ATA':'I', 'ATC':'I', 'ATT':'I', 'ATG':'M',
        'ACA':'T', 'ACC':'T', 'ACG':'T', 'ACT':'T',
        'AAC':'N', 'AAT':'N', 'AAA':'K', 'AAG':'K',
        'AGC':'S', 'AGT':'S', 'AGA':'R', 'AGG':'R',                
        'CTA':'L', 'CTC':'L', 'CTG':'L', 'CTT':'L',
        'CCA':'P', 'CCC':'P', 'CCG':'P', 'CCT':'P',
        'CAC':'H', 'CAT':'H', 'CAA':'Q', 'CAG':'Q',
        'CGA':'R', 'CGC':'R', 'CGG':'R', 'CGT':'R',
        'GTA':'V', 'GTC':'V', 'GTG':'V', 'GTT':'V',
        'GCA':'A', 'GCC':'A', 'GCG':'A', 'GCT':'A',
        'GAC':'D', 'GAT':'D', 'GAA':'E', 'GAG':'E',
        'GGA':'G', 'GGC':'G', 'GGG':'G', 'GGT':'G',
        'TCA':'S', 'TCC':'S', 'TCG':'S', 'TCT':'S',
        'TTC':'F', 'TTT':'F', 'TTA':'L', 'TTG':'L',
        'TAC':'Y', 'TAT':'Y', 'TAA':'_', 'TAG':'_',
        'TGC':'C', 'TGT':'C', 'TGA':'_', 'TGG':'W',
    }
```

Let's generate a random DNA sequence:


```python
import random
seq = "".join([random.choice('atcg') for x in range(100)])
```


```python
seq
```




    'agaccgtagcccaagtgcgtttgaatgtggctacttgggaggatttcattgcggtctgtctccgtacttgttattggtcttctttctgcattatgacgca'



To translate this sequence we write a code that uses a `for` loop that iterates over the DNA sequence in steps of 3, creating a codon at each iteration. If the codon is less than 3 letters long, the loop is broken. The resulting amino acid is then added to the `translation` string:


```python
translation = ""
for i in range(0, len(seq), 3):
    codon = seq[i:i+3].upper()
    if len(codon) < 3: break
    if codon in table:
        translation += table[codon]
    else:
        translation += "X"

print("Translation:", translation)
```

    Translation: RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R


Note that the code uses the `upper()` method to ensure the codon is in uppercase since the table dictionary is case-sensitive. Additionally, the code checks if the codon is in the table dictionary and if not, it adds the letter "X" to the translation. This is a common way to represent unknown or stop codons in a protein sequence.


```python
translation
```




    'RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R'



Now we define a function that would perform translation so that we can reuse it later:


```python
def translate(seq):
    translation = ''
    table = {
        'ATA':'I', 'ATC':'I', 'ATT':'I', 'ATG':'M',
        'ACA':'T', 'ACC':'T', 'ACG':'T', 'ACT':'T',
        'AAC':'N', 'AAT':'N', 'AAA':'K', 'AAG':'K',
        'AGC':'S', 'AGT':'S', 'AGA':'R', 'AGG':'R',                
        'CTA':'L', 'CTC':'L', 'CTG':'L', 'CTT':'L',
        'CCA':'P', 'CCC':'P', 'CCG':'P', 'CCT':'P',
        'CAC':'H', 'CAT':'H', 'CAA':'Q', 'CAG':'Q',
        'CGA':'R', 'CGC':'R', 'CGG':'R', 'CGT':'R',
        'GTA':'V', 'GTC':'V', 'GTG':'V', 'GTT':'V',
        'GCA':'A', 'GCC':'A', 'GCG':'A', 'GCT':'A',
        'GAC':'D', 'GAT':'D', 'GAA':'E', 'GAG':'E',
        'GGA':'G', 'GGC':'G', 'GGG':'G', 'GGT':'G',
        'TCA':'S', 'TCC':'S', 'TCG':'S', 'TCT':'S',
        'TTC':'F', 'TTT':'F', 'TTA':'L', 'TTG':'L',
        'TAC':'Y', 'TAT':'Y', 'TAA':'_', 'TAG':'_',
        'TGC':'C', 'TGT':'C', 'TGA':'_', 'TGG':'W',
    }
    for i in range(0, len(seq), 3):
        codon = seq[i:i+3].upper()
        if len(codon) < 3: break
        if codon in table:
            translation += table[codon]
        else:
            translation += "X"
    return(translation)
```


```python
translate(seq)
```




    'RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R'

## Expanding to multiple phases (frames)

We can further modify the function by adding a `phase` parameter that would allow translating in any of the three phases:


```python
def translate_phase(seq,phase):
    translation = ''
    table = {
        'ATA':'I', 'ATC':'I', 'ATT':'I', 'ATG':'M',
        'ACA':'T', 'ACC':'T', 'ACG':'T', 'ACT':'T',
        'AAC':'N', 'AAT':'N', 'AAA':'K', 'AAG':'K',
        'AGC':'S', 'AGT':'S', 'AGA':'R', 'AGG':'R',                
        'CTA':'L', 'CTC':'L', 'CTG':'L', 'CTT':'L',
        'CCA':'P', 'CCC':'P', 'CCG':'P', 'CCT':'P',
        'CAC':'H', 'CAT':'H', 'CAA':'Q', 'CAG':'Q',
        'CGA':'R', 'CGC':'R', 'CGG':'R', 'CGT':'R',
        'GTA':'V', 'GTC':'V', 'GTG':'V', 'GTT':'V',
        'GCA':'A', 'GCC':'A', 'GCG':'A', 'GCT':'A',
        'GAC':'D', 'GAT':'D', 'GAA':'E', 'GAG':'E',
        'GGA':'G', 'GGC':'G', 'GGG':'G', 'GGT':'G',
        'TCA':'S', 'TCC':'S', 'TCG':'S', 'TCT':'S',
        'TTC':'F', 'TTT':'F', 'TTA':'L', 'TTG':'L',
        'TAC':'Y', 'TAT':'Y', 'TAA':'_', 'TAG':'_',
        'TGC':'C', 'TGT':'C', 'TGA':'_', 'TGG':'W',
    }
    assert phase >= 0 and phase <= 3, "Phase parameter can only be set to 0, 1, or 2! You specified {}".format(phase)
    for i in range(phase, len(seq), 3):
        codon = seq[i:i+3].upper()
        if len(codon) < 3: break
        if codon in table:
            translation += table[codon]
        else:
            translation += "X"
    return(translation)
```


```python
translate_phase(seq,2)
```




    'TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT'



To translate in all six reading frames (three of the "+" strand and three of the "-" strand) we need to be able to create a reverse complement of the sequence. Let's write a simple function for that.
The cell below implements a function `revcomp` that takes a DNA sequence as input and returns its reverse complement. It works by first reversing the sequence using the slice
notation `seq[::-1]`, which returns the sequence in reverse order. Then, the `translate` method is used with the `str.maketrans` function to replace each occurrence
of 'a', 't', 'g', 'c', 'A', 'T', 'G', and 'C' in the reversed sequence with 't', 'a', 'c', 'g', 'T', 'A', 'C', and 'G', respectively:


```python
import string

def revcomp(seq):
    return seq[::-1].translate(str.maketrans('atgcATCG','tagcTACG'))
```

Now let's use this function to create translation in all six reading frames. The cell below uses a `for` loop that iterates over the range `(0, 3)`, representing the different phases
(or starting positions) of the translation. At each iteration, the `translate_phase` function is called with the DNA sequence and the current phase, and the resulting protein sequence
is appended to the `translations` list along with the `phase` and the `strand` orientation (+ or -):


```python
translations = []

for i in range(0,3):
    translations.append((translate_phase(seq,i),str(i),'+'))
    translations.append((translate_phase(revcomp(seq),i),str(i),'-'))
```


```python
translations
```




    [('RP_PKCV_MWLLGRISLRSVSVLVIGLLSAL_R', '0', '+'),
     ('SLIIDKQQE_ELATDRRIKWWELRRLKASSRSL', '0', '-'),
     ('DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA', '1', '+'),
     ('R__STNNRNKN_PQTGESNGGNYGD_KRVPVAC', '1', '-'),
     ('TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT', '2', '+'),
     ('ADNRQTTGIRTSHRQANQMVGTTEIESEFP_P', '2', '-')]



## Finding coordinates of continuous translations

The translation we've generated above contains stops (e.g., `_` symbols). The actual biologically relevant protein sequences are between stops.
We now need to split translation strings into meaningful peptide sequences and compute their coordinates. Let's begin by splitting a string on `_` and computing the start and end positions of each peptide:


```python
string = "aadsds_dsds_dsds"

split_indices = []
for i,char in enumerate(string):
    if char == "_":
        split_indices.append(i)
print(split_indices)
```

    [6, 11]


The code above generates a list of split indices for a string. The list contains the indices of the characters in the string that match a specified character (in this case, the underscore `_` character).

The `enumerate` function is used to loop over the characters in the string, and at each iteration, the current index and character are stored in the variables `i` and `char`, respectively.
If the current character matches the specified character, the index `i` is appended to the `split_indices` list.

After the loop, the `split_indices` list is printed to the console. For the input string `"aadsds_dsds_dsds"`, the output would be `[6, 11]`, indicating that the dashes are located at indices 6 and 11.

But we actually need coordinates of peptides bound by `_` characters. To get to this let's first modify `split_indices` by adding the beginning and end:


```python
string = "aadsds_dsds_dsds"

split_indices = []
for i,char in enumerate(string):
    if char == "_":
        split_indices.append(i)
        
split_indices.insert(0,-1)
split_indices.append(len(string))
print(split_indices)
```

    [-1, 6, 11, 16]


Now, let's convert these to ranges and also stick the peptide sequence in:


```python
string = "aadsds_dsds_dsds"

split_indices = []
for i, char in enumerate(string):
    if char == "_":
        split_indices.append(i)
        
split_indices.insert(0, -1)
split_indices.append(len(string))

orfs = string.split('_')

parts = []
for i in range(len(split_indices)-1):
    parts.append((orfs[i],split_indices[i]+1, split_indices[i+1]))

print(parts)
```

    [('aadsds', 0, 6), ('dsds', 7, 11), ('dsds', 12, 16)]


Now let's convert this to function:


```python
def extract_coords(translation):
    split_indices = []
    for i, char in enumerate(translation):
        if char == "_":
            split_indices.append(i)
        
    split_indices.insert(0, -1)
    split_indices.append(len(translation))

    parts = []
    for i in range(len(split_indices)-1):
        parts.append((translation.split('_')[i], split_indices[i] + 1, split_indices[i + 1]))

    return(parts)
```


```python
extract_coords(string)
```




    [('aadsds', 0, 6), ('dsds', 7, 11), ('dsds', 12, 16)]



And specify the right parameters to make it truly generic:


```python
def extract_coords_with_annotation(separator,translation,phase,strand):
    split_indices = []
    for i,char in enumerate(translation):
        if char == separator:
            split_indices.append(i)
        
    split_indices.insert(0,-1)
    split_indices.append(len(translation))

    parts = []
    for i in range(len(split_indices)-1):
        parts.append((translation.split(separator)[i], phase, strand, split_indices[i]+1, split_indices[i+1]))

    return(parts)
```


```python
extract_coords_with_annotation('_', string, '0', '+')
```




    [('aadsds', '0', '+', 0, 6),
     ('dsds', '0', '+', 7, 11),
     ('dsds', '0', '+', 12, 16)]



## Analyzing all translations of a given sequence

We begin by parsing the `translations` list we defined above:

```python
all_translations = []
for item in translations:
    all_translations.append(extract_coords_with_annotation('_',item[0],item[1],item[2]))
```


```python
all_translations
```




    [[('RP', '0', '+', 0, 2),
      ('PKCV', '0', '+', 3, 7),
      ('MWLLGRISLRSVSVLVIGLLSAL', '0', '+', 8, 31),
      ('R', '0', '+', 32, 33)],
     [('SLIIDKQQE', '0', '-', 0, 9),
      ('ELATDRRIKWWELRRLKASSRSL', '0', '-', 10, 33)],
     [('DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA', '1', '+', 0, 33)],
     [('R', '1', '-', 0, 1),
      ('', '1', '-', 2, 2),
      ('STNNRNKN', '1', '-', 3, 11),
      ('PQTGESNGGNYGD', '1', '-', 12, 25),
      ('KRVPVAC', '1', '-', 26, 33)],
     [('TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT', '2', '+', 0, 32)],
     [('ADNRQTTGIRTSHRQANQMVGTTEIESEFP', '2', '-', 0, 30),
      ('P', '2', '-', 31, 32)]]



Now the problem with this list is that it is nested. However, we need to make it flat:


```python
flat_list = []
for sublist in all_translations:
    for item in sublist:
        flat_list.append(item)
```


```python
flat_list
```




    [('RP', '0', '+', 0, 2),
     ('PKCV', '0', '+', 3, 7),
     ('MWLLGRISLRSVSVLVIGLLSAL', '0', '+', 8, 31),
     ('R', '0', '+', 32, 33),
     ('SLIIDKQQE', '0', '-', 0, 9),
     ('ELATDRRIKWWELRRLKASSRSL', '0', '-', 10, 33),
     ('DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA', '1', '+', 0, 33),
     ('R', '1', '-', 0, 1),
     ('', '1', '-', 2, 2),
     ('STNNRNKN', '1', '-', 3, 11),
     ('PQTGESNGGNYGD', '1', '-', 12, 25),
     ('KRVPVAC', '1', '-', 26, 33),
     ('TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT', '2', '+', 0, 32),
     ('ADNRQTTGIRTSHRQANQMVGTTEIESEFP', '2', '-', 0, 30),
     ('P', '2', '-', 31, 32)]



Now we can load the list into `Pandas` and plot away:


```python
import pandas as pd
df = pd.DataFrame(flat_list,columns=['aa','frame','phase','start','end'])
```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>aa</th>
      <th>frame</th>
      <th>phase</th>
      <th>start</th>
      <th>end</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>RP</td>
      <td>0</td>
      <td>+</td>
      <td>0</td>
      <td>2</td>
    </tr>
    <tr>
      <th>1</th>
      <td>PKCV</td>
      <td>0</td>
      <td>+</td>
      <td>3</td>
      <td>7</td>
    </tr>
    <tr>
      <th>2</th>
      <td>MWLLGRISLRSVSVLVIGLLSAL</td>
      <td>0</td>
      <td>+</td>
      <td>8</td>
      <td>31</td>
    </tr>
    <tr>
      <th>3</th>
      <td>R</td>
      <td>0</td>
      <td>+</td>
      <td>32</td>
      <td>33</td>
    </tr>
    <tr>
      <th>4</th>
      <td>SLIIDKQQE</td>
      <td>0</td>
      <td>-</td>
      <td>0</td>
      <td>9</td>
    </tr>
    <tr>
      <th>5</th>
      <td>ELATDRRIKWWELRRLKASSRSL</td>
      <td>0</td>
      <td>-</td>
      <td>10</td>
      <td>33</td>
    </tr>
    <tr>
      <th>6</th>
      <td>DRSPSAFECGYLGGFHCGLSPYLLLVFFLHYDA</td>
      <td>1</td>
      <td>+</td>
      <td>0</td>
      <td>33</td>
    </tr>
    <tr>
      <th>7</th>
      <td>R</td>
      <td>1</td>
      <td>-</td>
      <td>0</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td></td>
      <td>1</td>
      <td>-</td>
      <td>2</td>
      <td>2</td>
    </tr>
    <tr>
      <th>9</th>
      <td>STNNRNKN</td>
      <td>1</td>
      <td>-</td>
      <td>3</td>
      <td>11</td>
    </tr>
    <tr>
      <th>10</th>
      <td>PQTGESNGGNYGD</td>
      <td>1</td>
      <td>-</td>
      <td>12</td>
      <td>25</td>
    </tr>
    <tr>
      <th>11</th>
      <td>KRVPVAC</td>
      <td>1</td>
      <td>-</td>
      <td>26</td>
      <td>33</td>
    </tr>
    <tr>
      <th>12</th>
      <td>TVAQVRLNVATWEDFIAVCLRTCYWSSFCIMT</td>
      <td>2</td>
      <td>+</td>
      <td>0</td>
      <td>32</td>
    </tr>
    <tr>
      <th>13</th>
      <td>ADNRQTTGIRTSHRQANQMVGTTEIESEFP</td>
      <td>2</td>
      <td>-</td>
      <td>0</td>
      <td>30</td>
    </tr>
    <tr>
      <th>14</th>
      <td>P</td>
      <td>2</td>
      <td>-</td>
      <td>31</td>
      <td>32</td>
    </tr>
  </tbody>
</table>
</div>

Now let's plot it:


```python
import altair as alt

plus = alt.Chart(df[df['phase']=='+']).mark_rect().encode(
    x = alt.X('start:Q'),
    x2 = alt.X2('end:Q'),
    y = alt.Y('frame:N'),
    color='frame',
    tooltip='aa:N'
             ).properties(
    width=600,
    height=100)

minus = alt.Chart(df[df['phase']=='-']).mark_rect().encode(
    x = alt.X('start:Q',sort=alt.EncodingSortField('start:Q', order='descending')),
    x2 = alt.X2('end:Q'),
    y = alt.Y('frame:N'),
    color='frame',
    tooltip='aa:N'
             ).properties(
    width=600,
    height=100)

plus & minus
```

![ORFS](./images/orfs.svg "Translation on plus (top) and minus (bottom) strands.")


topic: data-science
tutorial: gnmx-lecture3
title: Introduction to sequencing with Python (part two)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: fec0d64c92b85cd1cc45e28d17185367
last_modified: 2026-05-15T22:28:11.070325
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture3/tutorial.html

content:
[![XKCD353](https://imgs.xkcd.com/comics/python.png)](https://xkcd.com/353/)

Preclass prep: Chapters [8](https://greenteapress.com/thinkpython2/html/thinkpython2009.html) and [10](https://greenteapress.com/thinkpython2/html/thinkpython2011.html) from "Think Python"

> This material uses examples from notebooks developed by [Ben Langmead](https://langmead-lab.org/teaching-materials/)
{: .quote cite="https://langmead-lab.org/teaching-materials" }

# Prep

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Within JupyterLab start a new Python3 notebook
3. Open [this page](http://cs1110.cs.cornell.edu/tutor/#mode=edit) in a new browser tab

# Strings in Python

In Python, strings are sequences of characters enclosed in quotation marks (either single or double quotes). They can be assigned to a variable and manipulated using various string methods. For example:

string1 = "Hello World!"
string2 = 'Hello World!'

print(string1)  # Output: Hello World!
print(string2)  # Output: Hello World!

You can also use the + operator to concatenate strings, and the * operator to repeat a string a certain number of times:


```python
string3 = "Hello " + "World!"
print(string3)  # Output: Hello World!

string4 = "Hello " * 3
print(string4)  # Output: Hello Hello Hello
```

    Hello World!
    Hello Hello Hello 


There are also many built-in string methods such as `upper()`, `lower()`, `replace()`, `split()`, `find()`, `len()`, etc. These can be used to manipulate and extract information from strings:


```python
string5 = "Hello World!"
print(string5.upper())   # Output: HELLO WORLD!
print(string5.lower())   # Output: hello world!
print(string5.replace("H", "J"))  # Output: Jello World!
print(string5.split(" "))  # Output: ['Hello', 'World!']
```

    HELLO WORLD!
    hello world!
    Jello World!
    ['Hello', 'World!']


You can also use indexing, slicing and string formatting to access or manipulate substrings or insert dynamic data into a string.


```python
name = "John"
age = 30

print("My name is {} and I am {} years old.".format(name, age))
```

    My name is John and I am 30 years old.

You can also use f-strings, or formatted string literals, to embed expressions inside string literals.

```python
name = "John"
age = 30

print(f"My name is {name} and I am {age} years old.")
```

    My name is John and I am 30 years old.

## String indexing

In Python, strings are sequences of characters, and each character has a corresponding index, starting from 0.
This means that we can access individual characters in a string using their index. This is called "string indexing".

Here is an example:

```python
string = "Hello World!"
print(string[0])  # Output: H
print(string[1])  # Output: e
print(string[-1])  # Output: !
```

    H
    e
    !

You can also use negative indexing to access characters from the end of the string, with -1 being the last character, -2 the second to last and so on.

You can also use slicing to extract a substring from a string. The syntax is `string[start:end:step]`, where `start` is the starting index, `end` is the ending index (not included), and step is the number of characters to skip between each index.

```python
string = "Hello World!"
print(string[0:5])  # Output: Hello
print(string[6:11])  # Output: World
print(string[::2])  # Output: HloWrd
```

    Hello
    World
    HloWrd

You can also use the string formatting method to get the string at the specific index.

```python
string = "Hello World!"
index = 3
print(f"The character at index {index} is: {string[index]}")
```

    The character at index 3 is: l


## String functions

In Python, many built-in string methods can be used to manipulate and extract information from strings. Here are some of the most commonly used ones:

- `upper()`: Converts the string to uppercase
- `lower()`: Converts the string to lowercase
- `replace(old, new)`: Replaces all occurrences of the old substring with the new substring
- `split(separator)`: Splits the string into a list of substrings using the specified separator
- `find(substring)`: Returns the index of the first occurrence of the substring, or -1 if the substring is not found
- `index(substring)`: Returns the index of the first occurrence of the substring or raises a ValueError if the substring is not found
- `count(substring)`: Returns the number of occurrences of the substring
- `join(iterable)`: Concatenates the elements of an iterable (such as a list or tuple) with the string as the separator
- `strip()`: Removes leading and trailing whitespaces from the string
- `lstrip()`: Removes leading whitespaces from the string
- `rstrip()`: Removes trailing whitespaces from the string
- `startswith(substring)`: Returns True if the string starts with the specified substring, False otherwise
- `endswith(substring)`: Returns True if the string ends with the specified substring, False otherwise
- `isalpha()`: Returns True if the string contains only alphabetic characters, False otherwise
- `isdigit()`: Returns True if the string contains only digits, False otherwise
- `isalnum()`: Returns True if the string contains only alphanumeric characters, False otherwise
- `format()`: Formats the string by replacing placeholders with specified values
- `len()`: Returns the length of the string

Here is an example of some of these methods:


```python
string = "Hello World!"

print(string.upper())  # Output: HELLO WORLD!
print(string.lower())  # Output: hello world!
print(string.replace("H", "J"))  # Output: Jello World!
print(string.split(" "))  # Output: ['Hello', 'World!']
print(string.find("World"))  # Output: 6
print(string.count("l"))  # Output: 3
print(string.strip())  # Output: "Hello World!"
print(string.startswith("Hello"))  # Output: True
print(string.endswith("World!"))  # Output: True
print(string.isalpha())  # Output: False
print(string.isalnum())  # Output: False
print(string.format())  # Output: Hello World!
print(len(string))  # Output: 12
```

    HELLO WORLD!
    hello world!
    Jello World!
    ['Hello', 'World!']
    6
    3
    Hello World!
    True
    True
    False
    False
    Hello World!
    12


## Playing with strings

```python
st = 'ACGT'
```


```python
len(st) # getting the length of a string
```

    4

```python
'' # empty string (epsilon)
```
    ''

```python
len('')
```

    0

```python
import random
random.choice('ACGT') # generating a random nucleotide
```
    'G'

```python
# now I'll make a random nucleotide string by concatenating random nucleotides
st = ''.join([random.choice('ACGT') for _ in range(40)])
st
```

    'GCTATATCAATGTTATCCGTTTTCTGATGTCGCGAGGACA'


```python
st[1:3] # substring, starting at position 1 and extending up to but not including position 3
# note that the first position is numbered 0
```

    'CT'


```python
st[0:3] # prefix of length 3
```

    'GCT'

```python
st[:3] # another way of getting the prefix of length 3
```

    'GCT'

```python
st[len(st)-3:len(st)] # suffix of length 3
```

    'ACA'

```python
st[-3:] # another way of getting the suffix of length 3
```

    'ACA'

```python
st1, st2 = 'CAT', 'ATAC'
```


```python
st1
```


    'CAT'


```python
st2
```

    'ATAC'


```python
st1 + st2 # concatenation of 2 strings
```

    'CATATAC'


# FASTQ

This notebook explores [FASTQ], the most common format for storing sequencing reads.

FASTA and FASTQ are rather similar, but FASTQ is almost always used for storing *sequencing reads* (with associated quality values), whereas FASTA is used for storing all kinds of DNA,
RNA or protein sequences (without associated quality values).

## Basic format
Here's a single sequencing read in FASTQ format:

    @ERR294379.100739024 HS24_09441:8:2203:17450:94030#42/1
    AGGGAGTCCACAGCACAGTCCAGACTCCCACCAGTTCTGACGAAATGATGAGAGCTCAGAAGTAACAGTTGCTTTCAGTCCCATAAAAACAGTCCTACAA
    +
    BDDEEF?FGFFFHGFFHHGHGGHCH@GHHHGFAHEGFEHGEFGHCCGGGFEGFGFFDFFHBGDGFHGEFGHFGHGFGFFFEHGGFGGDGHGFEEHFFHGE

It's spread across four lines.  The four lines are:

1. "`@`" followed by a read name
2. Nucleotide sequence
3. "`+`", possibly followed by some info, but ignored by virtually all tools
4. Quality sequence (explained below)

## Reading FASTQ with python

Download a sample small fastq file:

```
!wget https://zenodo.org/records/10602772/files/fastq_single_end_short.fq
```

Now we will use a very simple Python function to read this file and load fastq data into a list:

```python
def parse_fastq(fh):
    """ Parse reads from a FASTQ filehandle.  For each read, we
        return a name, nucleotide-string, quality-string triple. """
    reads = []
    while True:
        first_line = fh.readline()
        if len(first_line) == 0:
            break  # end of file
        name = first_line[1:].rstrip()
        seq = fh.readline().rstrip()
        fh.readline()  # ignore line starting with +
        qual = fh.readline().rstrip()
        reads.append((name, seq, qual))
    return reads

with open('fastq_single_end_short.fq','r') as fq:
    reads = parse_fastq(fq)

for read in reads:
    print(read)
```


```
('ERR294379.100739024 HS24_09441:8:2203:17450:94030#42/1', 'AGGGAGTCCACAGCACAGTCCAGACTCCCACCAGTTCTGACGAAATGATG', 'BDDEEF?FGFFFHGFFHHGHGGHCH@GHHHGFAHEGFEHGEFGHCCGGGF')
('ERR294379.136275489 HS24_09441:8:2311:1917:99340#42/1',  'CTTAAGTATTTTGAAAGTTAACATAAGTTATTCTCAGAGAGACTGCTTTT', '@@AHFF?EEDEAF?FEEGEFD?GGFEFGECGE?9H?EEABFAG9@CDGGF')
('ERR294379.97291341 HS24_09441:8:2201:10397:52549#42/1',  'GGCTGCCATCAGTGAGCAAGTAAGAATTTGCAGAAATTTATTAGCACACT', 'CDAF<FFDEHEFDDFEEFDGDFCHD=GHG<GEDHDGJFHEFFGEFEE@GH')
```


The nucleotide string can sometimes contain the character "`N`".  `N` essentially means "no confidence." The sequencer knows there's a nucleotide there but doesn't know whether it's an A, C, G or T.


> <comment-title>A note on <tt>while True</tt></comment-title>
>
>In Python, the while loop is used to repeatedly execute a block of code as long as a certain condition is true. The while True statement is a special case where the loop will run indefinitely until a break statement is encountered inside the loop.
>
>Here is an example of a while True loop:
>
>
>```python
>while True:
>    user_input = input("Enter 'q' to quit: ")
>    if user_input == 'q':
>        break
>    print("You entered:", user_input)
>
>print("Exited the loop")
>```
>```
>Enter 'q' to quit:  q
>Exited the loop
>```
>
>
>In this example, the loop will keep asking for user input until the user enters the 'q' character, which triggers the break statement, and the loop is exited.
>
>It is important to be careful when using while True loops, as they will run indefinitely if a break statement is not included. This can cause the program to crash or hang, if not handled properly.
>
>Also, It is recommended to use `while True` loop with a `break` statement, in case you want to execute the loop until some specific condition is met, otherwise, it's not a good practice to use `while True`.
>
>It's a good practice to include a way for the user to exit the loop, such as the break statement in the example above, or a counter variable to keep track of the number of iterations.
{: .comment}

## Read name

Read names often contain information about:

1. The scientific study for which the read was sequenced.  E.g. the string `ERR294379` (an [SRA accession number](http://www.ebi.ac.uk/ena/about/sra_format)) in the read names correspond to [this study](http://www.ncbi.nlm.nih.gov/sra/?term=ERR294379).
2. The sequencing instrument, and the exact *part* of the sequencing instrument, where the DNA was sequenced.  See the [FASTQ format](http://en.wikipedia.org/wiki/FASTQ_format#Illumina_sequence_identifiers) Wikipedia article for specifics on how the Illumina software encodes this information.
3. Whether the read is part of a *paired-end read* and, if so, which end it is.  Paired-end reads will be discussed further below.  The `/1` you see at the end of the read names above indicates the read is the first end from a paired-end read.

## Quality values

Quality values are probabilities.  Each nucleotide in each sequencing read has an associated quality value.  A nucleotide quality value encodes the probability that the nucleotide was *incorrectly called* by the sequencing instrument and its software.  If the nucleotide is `A`, the corresponding quality value encodes the probability that the nucleotide at that position is actually *not* an `A`.

Quality values are encoded in two senses: first, the relevant probabilities are re-scaled using the Phread scale, which is a negative log scale.
In other words if *p* is the probability that the nucleotide was incorrectly called, we encode this as *Q* where *Q* = -10 \* log10(*p*).

For example, if *Q* = 30, then *p* = 0.001, a 1-in-1000 chance that the nucleotide is wrong.  If *Q* = 20, then *p* = 0.01, a 1-in-100 chance.  If *Q* = 10, then *p* = 0.1, a 1-in-10 chance.  And so on.

Second, scaled quality values are *rounded* to the nearest integer and encoded using [ASCII printable characters](http://en.wikipedia.org/wiki/ASCII#ASCII_printable_characters).  For example, using the Phred33 encoding (which is by far the most common), a *Q* of 30 is encoded as the ASCII character with code 33 + 30 = 63, which is "`?`".  A *Q* of 20 is encoded as the ASCII character with code 33 + 20 = 53, which is "`5`".  And so on.

Let's define some relevant Python functions:

```python
def phred33_to_q(qual):
  """ Turn Phred+33 ASCII-encoded quality into Phred-scaled integer """
  return ord(qual)-33

def q_to_phred33(Q):
  """ Turn Phred-scaled integer into Phred+33 ASCII-encoded quality """
  return chr(Q + 33)

def q_to_p(Q):
  """ Turn Phred-scaled integer into error probability """
  return 10.0 ** (-0.1 * Q)

def p_to_q(p):
  """ Turn error probability into Phred-scaled integer """
  import math
  return int(round(-10.0 * math.log10(p)))
```


```python
# Here are the examples I discussed above

# Convert Qs into ps
q_to_p(30), q_to_p(20), q_to_p(10)
```

    (0.001, 0.01, 0.1)

```python=
p_to_q(0.00011) # note that the result is rounded
```

    40

```python=
q_to_phred33(30), q_to_phred33(20)
```

    ('?', '5')

To convert an entire string Phred33-encoded quality values into the corresponding *Q* or *p* values, I can do the following:


```python
# Take the first read from the small example above
name, seq, qual = parse_fastq(StringIO(fastq_string))[0]
q_string = list(map(phred33_to_q, qual))
p_string = list(map(q_to_p, q_string))
print(q_string)
print(p_string)
```

    [33, 35, 35, 36, 36, 37, 30, 37, 38, 37, 37, 37, 39, 38, 37, 37, 39, 39, 38, 39, 38, 38, 39, 34, 39, 31, 38, 39, 39, 39, 38, 37, 32, 39, 36, 38, 37, 36, 39, 38, 36, 37, 38, 39, 34, 34, 38, 38, 38, 37]
    [0.000501187233627272, 0.00031622776601683794, 0.00031622776601683794, 0.00025118864315095795, 0.00025118864315095795, 0.00019952623149688788, 0.001, 0.00019952623149688788, 0.00015848931924611126, 0.00019952623149688788, 0.00019952623149688788, 0.00019952623149688788, 0.0001258925411794166, 0.00015848931924611126, 0.00019952623149688788, 0.00019952623149688788, 0.0001258925411794166, 0.0001258925411794166, 0.00015848931924611126, 0.0001258925411794166, 0.00015848931924611126, 0.00015848931924611126, 0.0001258925411794166, 0.0003981071705534969, 0.0001258925411794166, 0.0007943282347242813, 0.00015848931924611126, 0.0001258925411794166, 0.0001258925411794166, 0.0001258925411794166, 0.00015848931924611126, 0.00019952623149688788, 0.000630957344480193, 0.0001258925411794166, 0.00025118864315095795, 0.00015848931924611126, 0.00019952623149688788, 0.00025118864315095795, 0.0001258925411794166, 0.00015848931924611126, 0.00025118864315095795, 0.00019952623149688788, 0.00015848931924611126, 0.0001258925411794166, 0.0003981071705534969, 0.0003981071705534969, 0.00015848931924611126, 0.00015848931924611126, 0.00015848931924611126, 0.00019952623149688788]


You might wonder how the sequencer and its software can *know* the probability that a nucleotide is incorrectly called.  It can't; this number is just an estimate.  To describe exactly how it's estimated is beyond the scope of this notebook; if you're interested, search for academic papers with "base calling" in the title.  Here's a helpful [video by Rafa Irizarry](http://www.youtube.com/watch?v=eXkjlopwIH4).

A final note: other ways of encoding quality values were proposed and used in the past.  For example, Phred64 uses an ASCII offset of 64 instead of 33, and Solexa64 uses "odds" instead of the probability *p*.  But Phred33 is by far the most common today and you will likely never have to worry about this.

><comment-title>A note in <tt>map()</tt></comment-title>
>
>In Python, the `map()` function is used to apply a given function to all elements of an iterable (such as a list, tuple, or string) and return an iterator (an object that can be iterated, e.g. in a `for`-loop) that yields the results.
>
>The `map()` function takes two arguments:
>
>A function that is to be applied to each element of the iterable
>An iterable on which the function is to be applied
>
>Here is an example:
>
>
>```python
># Using a function to square each element of a list
>numbers = [1, 2, 3, 4, 5]
>squared_numbers = map(lambda x: x**2, numbers)
>print(list(squared_numbers))  # Output: [1, 4, 9, 16, 25]
>```
>
>```
>[1, 4, 9, 16, 25]
>```
>
>In the example above, the `map()` function applies the lambda function `lambda x: x**2` to each element of the numbers list, and returns an iterator of the squared numbers. The `list()` function is used to convert the iterator to a list, so that the result can be printed.
>
>Another example is,
>
>
>```python
># Using the map() function to convert a list of strings to uppercase
>words = ["hello", "world"]
>uppercase_words = map(lambda word: word.upper(), words)
>print(list(uppercase_words))  # Output: ['HELLO', 'WORLD']
>```
>
>```
>['HELLO', 'WORLD']
>```
>
>It's important to note that the `map()` function returns an iterator, which can be used in a for loop, but is not a list, tuple, or any other iterable. If you want to create a list, tuple, or other iterable from the result of the `map()` function, you can use the `list()`, `tuple()`, or any other built-in function that creates an iterable.
>
>In Python 3, the `map()` function returns an iterator, which can be used in a for loop, but it's not iterable. If you want to create a list, tuple, or other iterable from the result of the `map()` function, you can use the `list()`, `tuple()`, or any other built-in function that creates an iterable.
>
>In Python 2, `map()` function returns a list, which can be used in a for loop, and it's iterable.
>
>In python 3.x, there is an alternative way to use map() function is `list(map(...))` or `tuple(map(...))` etc.
{: .comment}

## Paired-end reads

Sequencing reads can come in *pairs*.  Basically instead of reporting a single snippet of nucleotides from the genome, the sequencer might report a *pair* of
snippets that appear *close to each other* in the genome.  To accomplish this, the sequencer sequences *both ends* of a longer *fragment* of DNA.

Here is simple Python code that mimics how the sequencer obtains one paired-end read:

```python
# Let's just make a random genome of length 1K
import random
random.seed(637485)
genome = ''.join([random.choice('ACGT') for _ in range(1000)])
genome
```


```python
# The sequencer draws a fragment from the genome of length, say, 250
offset = random.randint(0, len(genome) - 250)
fragment = genome[offset:offset+250]
fragment
```


```python
# Then it reads sequences from either end of the fragment
end1, end2 = fragment[:75], fragment[-75:]
end1, end2
```

    ('GTATTGATTGTGTTACATAAACCCTCCCCCCCTACACGTGCTTGTCGACGCGGCACTGGACACTGATACGAGGAG',
     'CAATATATTAGCTCTCGAGGACAACGCACTACGTTTTCCAATTCAGCGGACTGGCGCCATTACAGTAAGTTGATT')

```python
# And because of how Illumina sequencing works, the
# second end is always from the opposite strand from the first
# (this is not the case for 454 and SOLiD data)

import string

# function for reverse-complementing
revcomp_trans = str.maketrans("ACGTacgt", "TGCAtgca")
def reverse_complement(s):
    return s[::-1].translate(revcomp_trans)

end2 = reverse_complement(end2)
end1, end2
```

    ('GTATTGATTGTGTTACATAAACCCTCCCCCCCTACACGTGCTTGTCGACGCGGCACTGGACACTGATACGAGGAG',
     'AATCAACTTACTGTAATGGCGCCAGTCCGCTGAATTGGAAAACGTAGTGCGTTGTCCTCGAGAGCTAATATATTG')

FASTQ can be used to store paired-end reads.  Say we have 1000 paired-end reads.  We should store them in a *pair* of FASTQ files.  The first FASTQ file (say, `reads_1.fq`) would contain all of the first ends and the second FASTQ file (say, `reads_2.fq`) would contain all of the second ends.  In both files, the ends would appear in corresponding order.  That is, the first entry in `reads_1.fq` is paired with the first entry in `reads_2.fq` and so on.

Here is a Python function that parses a pair of files containing paired-end reads.

```python
def parse_paired_fastq(fh1, fh2):
    """ Parse paired-end reads from a pair of FASTQ filehandles
        For each pair, we return a name, the nucleotide string
        for the first end, the quality string for the first end,
        the nucleotide string for the second end, and the
        quality string for the second end. """
    reads = []
    while True:
        first_line_1, first_line_2 = fh1.readline(), fh2.readline()
        if len(first_line_1) == 0:
            break  # end of file
        name_1, name_2 = first_line_1[1:].rstrip(), first_line_2[1:].rstrip()
        seq_1, seq_2 = fh1.readline().rstrip(), fh2.readline().rstrip()
        fh1.readline()  # ignore line starting with +
        fh2.readline()  # ignore line starting with +
        qual_1, qual_2 = fh1.readline().rstrip(), fh2.readline().rstrip()
        reads.append(((name_1, seq_1, qual_1), (name_2, seq_2, qual_2)))
    return reads

fastq_string1 = '''@509.6.64.20524.149722/1
AGCTCTGGTGACCCATGGGCAGCTGCTAGGGAGCCTTCTCTCCACCCTGA
+
HHHHHHHGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHIIHHIHFHHF
@509.4.62.19231.2763/1
GTTGATAAGCAAGCATCTCATTTTGTGCATATACCTGGTCTTTCGTATTC
+
HHHHHHHHHHHHHHEHHHHHHHHHHHHHHHHHHHHHHHDHHHHHHGHGHH'''

fastq_string2 = '''@509.6.64.20524.149722/2
TAAGTCAGGATACTTTCCCATATCCCAGCCCTGCTCCNTCTTTAAATAAT
+
HHHHHHHHHHHHHHHHHHHH@HHFHHHEFHHHHHHFF#FFFFFFFHHHHH
@509.4.62.19231.2763/2
CTCTGCTGGTATGGTTGACGCCGGATTTGAGAATCAANAAGAGCTTACTA
+
HHHHHHHHHHHHHHHHHHEHEHHHFHGHHHHHHHH>@#@=44465HHHHH'''

parse_paired_fastq(StringIO(fastq_string1), StringIO(fastq_string2))
```

    [(('509.6.64.20524.149722/1',
       'AGCTCTGGTGACCCATGGGCAGCTGCTAGGGAGCCTTCTCTCCACCCTGA',
       'HHHHHHHGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHIIHHIHFHHF'),
      ('509.6.64.20524.149722/2',
       'TAAGTCAGGATACTTTCCCATATCCCAGCCCTGCTCCNTCTTTAAATAAT',
       'HHHHHHHHHHHHHHHHHHHH@HHFHHHEFHHHHHHFF#FFFFFFFHHHHH')),
     (('509.4.62.19231.2763/1',
       'GTTGATAAGCAAGCATCTCATTTTGTGCATATACCTGGTCTTTCGTATTC',
       'HHHHHHHHHHHHHHEHHHHHHHHHHHHHHHHHHHHHHHDHHHHHHGHGHH'),
      ('509.4.62.19231.2763/2',
       'CTCTGCTGGTATGGTTGACGCCGGATTTGAGAATCAANAAGAGCTTACTA',
       'HHHHHHHHHHHHHHHHHHEHEHHHFHGHHHHHHHH>@#@=44465HHHHH'))]

><comment-title>A note on triple quotes</comment-title>
>
>In Python, triple quotes (either single or double) are used to create multi-line strings. They can also be used to create doc-strings, which are used to document a function, class, or module.
>
>For example:
>
>
>```python
>string1 = """This is a
>multiline string"""
>
>string2 = '''This is also
>a multiline string'''
>
>print(string1)
># Output:
># This is a
># multiline string
>
>print(string2)
># Output:
># This is also
># a multiline string
>```
>
>    This is a
>    multiline string
>    This is also
>    a multiline string
>
>
>Triple quotes can also be used to create docstrings, which are used to document a function, class, or module. The first line of a docstring is a brief summary of what the function, class, or module does, and the following lines provide more detailed information.
>
>
>```python
>def my_function():
>    """
>    This is a docstring for the my_function.
>    This function does not perform any operation.
>    """
>    pass
>
>print(my_function.__doc__)
># Output:
># This is a docstring for the my_function.
># This function does not perform any operation.
>```
>
>    
>        This is a docstring for the my_function.
>        This function does not perform any operation.
>        
{: .comment}

## How good are my reads?

Let's reuse the `parse_fastq` function from above:

```python
# Read the file into a list
with open('fastq_single_end_short.fq','r') as fq:
    reads = parse_fastq(fq)
```

☝️ This will load data into a list called `reads` containing individual reads (name, nucleotides, and quality values) represented as a [tuple](https://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences). 


```python
# Extract qualities and convert them into numerical values
run_qualities = []
for read in reads:
    read_qualities = []
    for quality in read[2]:
        read_qualities.append(phred33_to_q(quality))
    run_qualities.append(read_qualities)
```

☝️ This will take quality string only for each read from `reads` and convert it into a list of values. The end result, a list of lists called `run_qualities`, will contain as many elements as there as reads, where each element (also a list) contains all quality values for a read. We need to [transpose](https://en.wikipedia.org/wiki/Transpose) this list, so that instead of representing every reads as a list of quality values we represent every position as a list of quality values for all reads at that position. This can be easily done with [numpy](https://numpy.org/)'s [`T`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.T.html):

```python
# Transpose the quality values matrix

import numpy as np
base_qualities = np.array(run_qualities).T
```

Now we need to extract some key per-position statistics from the `base_qualities` list generated above. We, again, use numpy for this:

```python
# Prep data for plotting
plotting_data = {
    'base':[],
    'mean':[],
    'median':[],
    'q25':[],
    'q75':[],
    'min':[],
    'max':[]
}
for i,base in enumerate(base_qualities):
    plotting_data['base'].append(i)
    plotting_data['mean'].append(np.mean(base))
    plotting_data['median'].append(np.median(base))
    plotting_data['q25'].append(np.quantile(base,.25))
    plotting_data['q75'].append(np.quantile(base,.75))
    plotting_data['min'].append(np.min(base))
    plotting_data['max'].append(np.max(base))
```

☝️ The result of this, a dictionary called `plotting_data`, contains statistics types as *keys* and corresponding values as *lists*. Now we load these into Pandas:

```python
# Load into Pandas
import pandas as pd
plotting_data_df = pd.DataFrame(plotting_data)
```

And plot:

```python
# Plot!
import altair as alt

base = alt.Chart(plotting_data_df).encode(
    alt.X('base:Q', title="Position in the read")
).properties(
    width=800,
    height=200)

median = base.mark_tick(color='red',orient='horizontal').encode(
    alt.Y('median:Q', title="Phred quality score"),
)

q = base.mark_rule(color='green',opacity=.5,strokeWidth=10).encode(
    alt.Y('q25:Q'),
    alt.Y2('q75:Q')
)

min_max = base.mark_rule(color='black').encode(
        alt.Y('min:Q'),
        alt.Y2('max:Q')
)

median + q + min_max
```

Below is the same thing as just one code blob:

```python
# Read the file into a list
with open('fastq_single_end_short.fq','r') as fq:
    reads = parse_fastq(fq)

# Extract qualities and convert them into numerical values
run_qualities = []
for read in reads:
    read_qualities = []
    for quality in read[2]:
        read_qualities.append(phred33_to_q(quality))
    run_qualities.append(read_qualities)

# Transpose the quality values matrix

import numpy as np
base_qualities = np.array(run_qualities).T

# Prep data for plotting
plotting_data = {
    'base':[],
    'mean':[],
    'median':[],
    'q25':[],
    'q75':[],
    'min':[],
    'max':[]
}
for i,base in enumerate(base_qualities):
    plotting_data['base'].append(i)
    plotting_data['mean'].append(np.mean(base))
    plotting_data['median'].append(np.median(base))
    plotting_data['q25'].append(np.quantile(base,.25))
    plotting_data['q75'].append(np.quantile(base,.75))
    plotting_data['min'].append(np.min(base))
    plotting_data['max'].append(np.max(base))

# Load into Pandas
import pandas as pd
plotting_data_df = pd.DataFrame(plotting_data)

# Plot!
import altair as alt

base = alt.Chart(plotting_data_df).encode(
    alt.X('base:Q', title="Position in the read")
).properties(
    width=800,
    height=200)

median = base.mark_tick(color='red',orient='horizontal').encode(
    alt.Y('median:Q', title="Phred quality score"),
)

q = base.mark_rule(color='green',opacity=.5,strokeWidth=10).encode(
    alt.Y('q25:Q'),
    alt.Y2('q75:Q')
)

min_max = base.mark_rule(color='black').encode(
        alt.Y('min:Q'),
        alt.Y2('max:Q')
)

median + q + min_max
```

![Qualities](./images/qualities.svg "Box plot of quality values across read positions")

## Other comments

In all the examples above, the reads in the FASTQ file are all the same length.  This is not necessarily the case though it is usually true for datasets generated by sequencing-by-synthesis instruments.  FASTQ files can contain reads of various lengths.

FASTQ files often have the extension `.fastq` or `.fq`.

## Other resources

- [BioPython](http://biopython.org/wiki/Main_Page)
- [SeqIO](http://biopython.org/wiki/SeqIO)
- [SAMtools](http://samtools.sourceforge.net/)
- [FASTX](http://hannonlab.cshl.edu/fastx_toolkit/)
- [FASTQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
- [seqtk](https://github.com/lh3/seqtk)


topic: data-science
tutorial: python-files
title: Python - Files & CSV
difficulty: introductory
time_estimation: 1H30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 8d7f3c70407f89d6b905b37748542d04
last_modified: 2026-05-15T22:28:11.102165
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-files/tutorial.html

content:
Here we'll give a quick tutorial on how to read and write files within Python.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Setup

For this tutorial, we assume you're working in a notebook (Jupyter, CoCalc, etc) so we'll run a quick "setup" step to download some CSV data:

```python
import urllib.request
# Download a copy of Hamlet.
urllib.request.urlretrieve("https://gutenberg.org/cache/epub/1524/pg1524.txt", "hamlet.txt")
# Download some COVID data for Europe
urllib.request.urlretrieve("https://opendata.ecdc.europa.eu/covid19/vaccine_tracker/csv/data.csv", "vaccinations.csv")
# And a fastq file
urllib.request.urlretrieve("https://gist.github.com/hexylena/7d249607f8f763301f06c78a48c3bf6f/raw/a100e278cee1c94035a3a644b16863deee0ba2c0/example.fastq", "example.fastq")
```

And now we're ready to get started learning about files!

## Reading from a file

Reading from and writing to files in Python is very straightforward, we use `open()` to open a file to read from it. Files are accessed through something called a **file [handle](https://en.wikipedia.org/wiki/Handle_(computing))**, you're not accessing the file itself, you're opening a connection to the file, and then you can read lines from through that file handle. When you open a file handle, you must specify it's mode:

Mode | Purpose
--- | ---
`r` | We will **read** from this file handle
`w` | We will **write** to this file handle
`a` | We will **append** to this file handle (we cannot access earlier contents!)

You will mostly use `r` and `w`, `a` is especially useful for writing to program logs where you don't really care what was written before, you just want to add your new logs to the end of the file.

```python
with open('hamlet.txt', 'r') as handle:
    # readlines reads every line of a file into a giant list!
    lines = handle.readlines()
```

Here we introduce a new bit of syntax, the `with` block. Technically `with` begins a "context manager" which allows python to setup some things before the block, run some contents in the block, and automatically handle cleanup of this block. When you open a file, you *must* close it when you're done with it (otherwise bad things can happen!) and `with` prevents most of those issues.

In the above code snippet after the second line, the file (referred to by `handle`) is automatically closed.

> > <code-in-title>Using `with`</code-in-title>
> > ```
> > with open('file.txt', 'r') as handle:
> >     print(handle.readlines())
> > ```
> {: .code-in}
> > <code-out-title>Not using `with`</code-out-title>
> > ```
> > handle = open('file.txt', 'r')
> > print(handle.readlines())
> > handle.close() # Important!
> > ```
> {: .code-out}
{: .code-2col}

Let's see what's in our file. `

```python
print(lines[0])
print(lines[1])
print(lines[2])
```

Notice how it prints out a blank line afterwards! This is due to a `\n`, a newline. A newline just tells the computer "please put content on the next line". We can see it by using the `repr()` function:

```python
print(repr(lines[0]))
```

Every line that's read in ends in a newline currently. This is done because if we wanted to write it back out, we would need to preserve those newlines, or all of the content would be on one giant line. Let's try writing out a file, it's *just* like reading in a file!

```python
with open('hamlet-copy.txt', 'w') as handle:
    # readlines reads every line of a file into a giant list!
    for line in lines:
        handle.write(line)
```

Check this file in your folder, does it look right? Is it identical in size to the original?

### Use Case: Summarisation

Let's use the file's contents for something useful. This file specifically is the play Hamlet, by Shakespeare. The contents are formatted with a speaker indicated with all capital letters, followed by their lines (potentially spread over multiple lines of the file.)

```
HAMLET. Madam, how like you this play?

QUEEN. The lady protests too much, methinks.

HAMLET. O, but she’ll keep her word.
```

So let's count up how many times each character speaks! (Roughly)

```python
with open('hamlet.txt', 'r') as handle:
    # readlines reads every line of a file into a giant list!
    lines = handle.readlines()

speakers = {}
```

Here we've initialised the `lines` variable with the contents of the text, and setup `speakers` as a dictionary that will let us track how many times each character speaks. Next let's define a function to check if a character is speaking on that line. It is if it meets two conditions: the first word is all caps, and it ends with a `.`.

```python
def is_speaker(word):
    return word == word.upper() and word[-1] == '.'
```

We can use that function later to check if a line starts with a speaker

```python
# Loop over every line we read in
for line in lines:
    # Split by default splits on whitespace.
    words = line.split()

    # Are there words on this line? We can't access the first word if we
    # haven't any words.
    if len(words) == 0:
        continue

    # Check if the first word is uppercase, and the last character is a `.`,
    # then it's a character speaking.
    if is_speaker(words[0]):
        # Give this an easier to remember and understand name.
        speaker = words[0]

        # Have we seen this speaker before? If not, we should add them to the
        # speakers dictionary. Hint: Try removing this to see why we do this.
        if speaker not in speakers:
            speakers[speaker] = 0

        # Increment the number of times we've seen them speak.
        speakers[speaker] = speakers[speaker] + 1
```

Ok! We've done a couple things here that all fall into the category of **defensive programming**. As programmers, we often accept input from users or from unknown sources. That input may be wrong, it may have bad data, it may be trying to attack us. So we respond by checking very carefully if things match our expectations, and rejecting the input otherwise. We did a couple things here for that:

- Checking if the input matches our expectations exactly (capitals, `.`)
- Checking if the line is empty, using `continue` to skip it if it was
- Checking if the speaker is already known in the dictionary, adding it otherwise.

Let's see who was the most chatty:

```python
for key, value in speakers.items():
    print(key, value)
```

We've clearly caught a number of values that aren't expected, some section headers (the numeric values, and some rare values we don't expect.)

```python
for key, value in speakers.items():
    if value > 1:
        print(key, value)
```

Hamlet, the titular character, has the vast majority of turns speaking throughout the play:

Character | Turns speaking
---       | ---
HAMLET    | 358
HORATIO   | 107
KING      | 102
POLONIUS  | 86
QUEEN     | 69


## Writing files

Writing a file out is exactly like reading a file, we just use the different file mode `w` to indicate we wish to write to a file:

```python
with open('hello.txt', 'w') as handle:
    # Let's write it out a few times
    handle.write("Hello, world!")
    handle.write("Hello, world!")
    handle.write("Hello, world!")
    handle.write("Hello, world!")
```

Check the file's contents in your folder, does it look like you expect? Remember your `\n`s!

### Use Case: Transformation

A common use case is transforming one file's contents into another file format or file type. Let's do a very simple example of that, taking a FASTQ file and transforming it into a FASTA file. Remember first that a FASTQ file looks like:

```
@M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT
+
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFGGGFGGGGGGAFFGGFGGGGGGGGFGGGGGGGGGGGGGGFGGG+38+35*311*6,,31=******441+++0+0++0+*1*2++2++0*+*2*02*/***1*+++0+0++38++00++++++++++0+0+2++*+*+*+*+*****+0**+0**+***+)*.***1**//*)***)/)*)))*)))*),)0(((-((((-.(4(,,))).,(())))))).)))))))-))-(
```

Line | Contents
-- | --
1 | Identifier, prefixed with `@`
2 | Sequence
3 | `+`
4 | Quality scores

And a fasta file looks like this, where `>` indicates a sequence identifier, and it is followed by one or more lines of ACTGs.

```
>M00970:337:000000000-BR5KF:1:1102:17745:1557 1:N:0:CGCAGAAC+ACAGAGTT
```

In the setup portion we downloaded a FASTQ file, now let's extract all of the sequences from this file, and write them out as a FASTA file. Why would you want to do this? Sometimes after sequencing a sample (especially metagenomics), you want to blast the sequences to figure out which organisms they belong to. A common way to do that is BLAST which accepts fasta formatted sequences. So we'll write something to convert these formats, removing the `+` and quality score lines.

```python
with open('example.fastq', 'r') as handle:
    data = handle.readlines()

i = 0
with open('example-converted.fasta', 'w') as handle:
    for line in data:
        # Since fastq files have groups of 4 lines, we can use that to extract
        # specific lines of every read:
        if i % 4 == 0:
            handle.write(">" + line[1:])
        if i % 4 == 1:
            handle.write(line)
        i = i + 1
```

That's it! Check if your `fasta` file looks correct.

> <tip-title>`enumerate`</tip-title>
> If you want to know which item number you're on while you're looping over a list, you can use the function `enumerate()`
> Try out this code to see how it works:
>
> ```
> for index, item in enumerate(['a', 'b', 'c']):
>     print(index, item)
> ```
>
> Try using it to clean up the above code.
{: .tip}

## Reading CSV data

If you're reading data from a comma separated value (CSV) or tab separated value (TSV) file, you should use the built in `csv` module to do this. You might ask yourself "Why, csv parsing is easy" and that is a common thought! It would be so simple to do something like

```python
# Please don't do this :)
with open('vaccinations.csv', 'r') as handle:
    first_lines = handle.readlines()[0:10]
    for line in first_lines:
        print(line.split(','))
```

But you would be wrong! This code has a subtle bug that you might not see until someone generates data that specifically affects it, with "quoted" columns. If you have a table like

Patient | Location | Disease Indications
--- | --- | ---
Helena | Den Haag, the Netherlands | Z87.890
Bob | Little Rock, Arkansas, USA | Z72.53
Jane | London, UK | Z86.16

This would probably be exported as a CSV file from Excel that looks like:

```
Patient,Location,Disease Indications
Helena,"Den Haag, the Netherlands",Z87.890
Bob,"Little Rock, Arkansas, USA",Z72.53
Jane,"London, UK",Z86.16
```

Note that some columns are quoted. What do you think will happen with the following code?

```python
csv_data = """
Patient,Location,Disease Indications
Helena,"Den Haag, the Netherlands",Z87.890
Bob,"Little Rock, Arkansas, USA",Z72.53
Jane,"London, UK",Z86.16
""".strip().split('\n')

# Please don't do this :)
for line in csv_data:
    print(line.split(','))
```

Does that look right? Maybe not. Instead we can use the `csv` module to work around this and properly process CSV files:

```python
import csv

csv_data = """
Patient,Location,Disease Indications
Helena,"Den Haag, the Netherlands",Z87.890
Bob,"Little Rock, Arkansas, USA",Z72.53
Jane,"London, UK",Z86.16
""".strip().split('\n')

# Please DO this :)
csv_reader = csv.reader(csv_data, delimiter=",", quotechar='"')
for row in csv_reader:
    print(row)
```

That looks a lot better! Now we've properly handled the quoted columns that contain one or more `,` in the middle of our file. This is actually one of the motivating factors in using the TSV format, the <kbd>tab</kbd> character is much more rare in data than <kbd>,</kbd>. There is less chance for confusion with poorly written software.

Let's read in some statistics about vaccinations:

```python
import csv

vax = []
with open('vaccinations.csv', 'r') as handle:
    csv_reader = csv.reader(handle, delimiter=",", quotechar='"')
    for row in csv_reader:
        # Skip our header row
        if row[0] == 'YearWeekISO':
            continue
        # Otherwise load in the data
        vax.append(row)

print(vax[0:10])
```

Here we have a 2 dimensional array, a list of lists. Each row is an entry in the main list, and each column is an entry in each of those children.

Our columns are:

Column | Value
-- | --
0   | YearWeekISO
1   | ReportingCountry
2   | Denominator
3   | NumberDosesReceived
4   | NumberDosesExported
5   | FirstDose
6   | FirstDoseRefused
7   | SecondDose
8   | DoseAdditional1
9   | UnknownDose
10  | Region
11  | TargetGroup
12  | Vaccine
13  | Population

Let's subset the data to make it a bit easier to work with, maybe we'll just use the Dutch data (please feel free to choose another column though!)

```python
country = 'NL'

subset = []
for row in vax:
    # Here we select for a country
    if row[1] == country:
        subset.append(row)
print(subset[0:10])
```

That should be easier to work with, now we only have one country's data. Let's do some exercises with this data:


> <question-title>Which vaccines were given?</question-title>
>
> Which vaccines were given? Use the `subset` to examine which vaccines were given in the Netherlands. *Tip*: if `x` is a list, `set(x)` will return the unique values in that list.
>
> > <solution-title></solution-title>
> > To figure out which vaccines were given, we can look at column 12:
> >
> > ```python
> > vaccines = []
> > for row in subset:
> >     vaccines.append(row[12])
> > print(set(vaccines))
> > ```
> >
> > We can use the `set` function to convert the list to a set, and show only the unique values.
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>How many of each vaccine were given?</question-title>
>
> How many of each were given?
>
> *Tip*: use the accumulator pattern.
> *Tip*: Columns 5, 7, and 8 have doses being given out to patients.
>
> > <solution-title></solution-title>
> > ```python
> > doses = {}
> > for row in subset:
> >     brand = row[12]
> >     if brand not in doses:
> >         doses[brand] = 0
> >     doses[brand] = doses[brand] + int(row[5]) + int(row[7]) + int(row[8])
> > print(doses)
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>How many of each vaccine were exported? received?</question-title>
>
> How many of each were exported? received?
>
> *Tip*: you only need to loop once.
> *Tip*: you will need to handle an edge case here. Try and find it out!
>
> > <solution-title></solution-title>
> > ```python
> > export = {}
> > received = {}
> > for row in subset:
> >     brand = row[12]
> >     if brand not in export:
> >         export[brand] = 0
> >     if brand not in received:
> >         received[brand] = 0
> >     if row[4]:
> >         export[brand] = export[brand] + int(row[4])
> >     if row[3]:
> >         received[brand] = received[brand] + int(row[3])
> > print(export)
> > print(received)
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>When was the first dose received?</question-title>
>
> *Tip*: use `break`, and check how many doses were given!
>
> > <solution-title></solution-title>
> > ```python
> > for row in subset:
> >     if row[5] and int(row[5]) > 0:
> >         print(f"On {row[0]}, {row[5]} doses of {row[12]} were given")
> >         break
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
```

> <question-title>Transform the data for plotting</question-title>
>
> Let's say you want to plot the fraction of the population that has been vaccinated by the various points in time.
>
> - Subset the data further for TargetGroup (column 11) set to 'ALL'
> - Create an accumulator, to count how many doses have been given their FirstDose at each week
> - Use column 13 (population) to calculate the fraction of the population that has been given one of those doses at each week
>
> The output should be a list of percentages ranging from [0.0 to 1.0].
>
> > <solution-title></solution-title>
> > ```python
> > total_doses = 0
> > percent_vaccinated_per_week = []
> > for row in subset:
> >     if row[11] != 'ALL':
> >         continue
> >     total_doses = total_doses + int(row[5])
> >     percent_vaccinated_per_week.append(total_doses / int(row[13]))
> > ```
> {: .solution}
{: .question}

```python
# Try things out here!
percent_vaccinated_per_week = []
# Write code here!




# When you're done, you should have a 'results' variable
# You may need to `pip install matplotlib`
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(percent_vaccinated_per_week)
plt.xlabel('Week')
plt.ylabel('Percent Vaccinated')
```


> <question-title>Which vaccines were given?</question-title>
>
> Write this out to a new file, with two columns. The index and the Percent Vaccinated. It should be a comma separated file, and should have a header. Save this as a csv file named `weekly-percent-vax.csv`
>
> *Tip*: use a `csvwriter`, it works exactly like a `csvreader`. You can use the `writerow()` function to write out a row.
> *Tip*: Use `enumerate()` to get a list of items with indexes.
>
> > <solution-title></solution-title>
> >
> > ```
> > with open('weekly-percent-vax.csv', 'w') as handle:
> >     writer = csv.writer(csv_data, delimiter=",", quotechar='"')
> >     for row in enumerate(percent_vaccinated_per_week):
> >         writer.writerow(row)
> > ```
> >
> {: .solution}
{: .question}

```python
# Try things out here!
```

Congratulations on getting this far! Hopefully you feel more comfortable working with files.


topic: data-science
tutorial: python-loops
title: Python - Loops
difficulty: introductory
time_estimation: 40M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 7a86289d7fda6343135a1605c16ae684
last_modified: 2026-05-15T22:28:11.103990
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-loops/tutorial.html

content:
A *for loop* tells Python to execute some statements once for each value in a list, a character string, or some other collection: "for each thing in this group, do these operations"

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/), [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/), and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which are licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# For Loops

Which of these would you rather write

> > <code-in-title>Manually</code-in-title>
> > ```
> > print(2)
> > print(3)
> > print(5)
> > print(7)
> > print(11)
> > ```
> {: .code-in}
>
> > <code-out-title>With Loops</code-out-title>
> > ```python
> > for number in [2, 3, 5, 7, 11]:
> >     print(number)
> > ```
> {: .code-out}
{: .code-2col}

It may be less clear here, since you just need to do one operation (`print`) but if you had to do two operations, three, more?

## Structure

## A `for` loop is made up of a collection, a loop variable, and a body.

```python
for number in [2, 3, 5]:
    doubled = number * 2
    print(f"{number} doubled is {doubled}")
```

- `number` - this is the loop variable. It's a new variable, that's assigned to the values from the collection. It does not need to be defined before the loop.
- the collection, `[2, 3, 5]` is a `list` of numbers which we can tell from the square brackets used: `[`, `]`
- the loop body, where we double a number and the print out a message. The loop body is what gets executed for every iteration of the loop.

> > <code-in-title>The loop</code-in-title>
> > ```python
> > for number in [2, 3, 5]:
> >     doubled = number * 2
> >     print(f"{number} doubled is {doubled}")
> > ```
> {: .code-in}
>
> > <code-out-title>What's really happening internally</code-out-title>
> > ```python
> > # First iteration, number = 2
> > doubled = number * 2
> > print(f"{number} doubled is {doubled}")
> > # Second iteration, number = 3
> > doubled = number * 3
> > print(f"{number} doubled is {doubled}")
> > # Third iteration, number = 5
> > doubled = number * 5
> > print(f"{number} doubled is {doubled}")
> > ```
> {: .code-out}
{: .code-2col}

Writing loops saves us time and makes sure our code is accurate, that we don't accidentally introduce a typo somewhere in the loop body.

## Things You Can Loop Over

You can loop over characters in a string

```python
dna_string = 'ACTGGTCATCG'
for base in dna_string:
    print(base)
```

You can loop over lists:

```python
cast = ['Elphaba', 'Glinda', 'Fiyero', 'Nessarose']
for character in cast:
    print(character)
```

## Indentation

The first line of the `for` loop must end with a colon, and the body must be indented with *four spaces*. Many editors do this automatically for you and even convert <kbd>Tab</kbd>s into 4 spaces.

> <tip-title>Blocks in Python</tip-title>
>
> The colon at the end of the first line signals the start of a *block* of statements.
>
> ```
> for x in y:
>     print(x)
> ```
>
> or
>
> ```
> if x > 10:
>     print(x)
> ```
>
> or even further nesting is possible:
>
> ```
> for x in y:
>     if x > 10:
>         print(x)
> ```
>
{: .tip}

The indentation is in fact, quite necessary. Notice how this fails:

```python
#Fix me!
for number in [2, 3, 5]:
print(number)
```

And, likewise, this:

```python
patient1 = "z2910"
  patient2 = "y9583"
```

## Variable Naming

Loop variables can be called anything, `i`, `j`, and `k` are very commong defaults due to their long history of use in other programing languages.
As with all variables, loop variables are: Created on demand, and Meaningless; their names can be anything at all.

```python
for kitten in [2, 3, 5]:
    print(kitten)
```

But *meaningless* is bad for variable names, and whenever possible, we should strive to pick useful, accurate variable names that help use remember what's going on:

```
for sequence in sequences:
    print()
for patient in clinic_patients:
    print()
for nucleotide in dna_sequence:
    print()
```

## Range

You can use `range` to iterate over a sequence of numbers. This is a built in function (check `help(range)`!) so it's always available even if you don't `import` anything. The range produced is non-inclusive: `range(N)` is the numbers `0` to `N-1`, so the result will be exactly the length you requested.

```python
for number in range(10):
    print(number)
```

> <tip-title>Iterables can be weird</tip-title>
> In python `range` is a special type of iterable: none of the numbers are created until we need them.
>
> ```python
> print(range(5))
> print(range(-3, 8)[0:4])
> ```
>
> The easiest way to see what numbers are actually in there is to convert it to a `list`:
>
> ```python
> print(list(range(5)))
> print(list(range(-3, 8)))
> print(list(range(0, 10, 2)))
> ```
{: .tip}

## Accumulation

In programming you'll often want to accumulate some values: counting things (or "accumulating"). The pattern consists of creating a variable to store your result, running a loop over some data, and in that loop, adding to the variable for your result.

```python
# Sum the first 10 integers.
total = 0
for number in range(1, 11):
    total = total + (number)
print(f" final: {{ total }}")
```

But how did we get that result? We can add some "debugging" lines to the above code to figure out how we got to that result. Try adding the following line in the above loop

```
print(f'Currently {number}, our total is {total}')
```

You can add it before you update `total`, after it, or both! Compare the outputs to understand what's happening on each line.

> <tip-title>Controlling your loop!</tip-title>
>
> There are multiple ways to efficiently control your loop if you need it.
> these are the inbuilt python functions: continue & break
>
> when python encounters *continue* in your loop it will stop working and goes to the next iteration of the loop.
> ```
> for letter in 'Galaxy':
>     if letter == 'l':
>         continue
>     print(f'The letters are: {letter}')
>```
> with *break* python stops the loop and continues with the next part of the code like nothing happened
> ```
> for letter in 'Galaxy':
>     if letter == 'l':
>         break
>     print(f'The letters are: {letter}')
> print('Done')
> ```
{: .tip}

```python
# Test break and continue here
```

## Exercises

> <question-title>Tracing Execution</question-title>
>
> Create a table showing the numbers of the lines that are executed when this program runs,
> and the values of the variables after each line is executed.
>
> ```
> total = 0
> for char in "tin":
>     total = total + 1
> ```
> > <solution-title></solution-title>
> >
> > | Line | Variables            |
> > |------|----------------------|
> > | 1    | total = 0            |
> > | 2    | total = 0 char = 't' |
> > | 3    | total = 1 char = 't' |
> > | 2    | total = 1 char = 'i' |
> > | 3    | total = 2 char = 'i' |
> > | 2    | total = 2 char = 'n' |
> > | 3    | total = 3 char = 'n' |
> {: .solution}
{: .question}

```python
#Test your code here!
```

> <question-title>Reversing a String</question-title>
>
> Fill in the blanks in the program below so that it prints "stressed"
> (the reverse of the original character string "desserts").
>
> ```
> original = "stressed"
> result = ____
> for char in original:
>     result = ____
> print(result)
> ```
> > <solution-title></solution-title>
> > ```
> > original = "stressed"
> > result = ""
> > for char in original:
> >     result = char + result
> > print(result)
> > ```
> {: .solution}
{: .question}

```python
# Test your code here!
original = "stressed"
result = ____
for char in original:
    result = ____
print(result)
```


> <question-title>Practice Accumulating</question-title>
>
> Fill in the blanks in each of the programs below
> to produce the indicated result.
>
> ```
> # Total length of the strings in the list: ["red", "green", "blue"] => 12
> total = 0
> for word in ["red", "green", "blue"]:
>     ____ = ____ + len(word)
> print(total)
> ```
>
> > <solution-title></solution-title>
> > ```
> > total = 0
> > for word in ["red", "green", "blue"]:
> >     total = total + len(word)
> > print(total)
> > ```
> {: .solution}
>
> ```
> # List of word lengths: ["red", "green", "blue"] => [3, 5, 4]
> lengths = ____
> for word in ["red", "green", "blue"]:
>     lengths.____(____)
> print(lengths)
> ```
> > <solution-title></solution-title>
> > ```
> > lengths = []
> > for word in ["red", "green", "blue"]:
> >     lengths.append(len(word))
> > print(lengths)
> > ```
> {: .solution}
>
> ```
> # Concatenate all words: ["red", "green", "blue"] => "redgreenblue"
> words = ["red", "green", "blue"]
> result = ____
> for ____ in ____:
>     ____
> print(result)
> ```
> > <solution-title></solution-title>
> > ```
> > words = ["red", "green", "blue"]
> > result = ""
> > for word in words:
> >     result = result + word
> > print(result)
> > ```
> {: .solution}
>
> __Create an acronym:__ Starting from the list `["red", "green", "blue"]`, create the acronym `"RGB"` using
> a for loop.
>
> __Hint:__ You may need to use a string method to properly format the acronym.
> > <solution-title></solution-title>
> > ```
> > acronym = ""
> > for word in ["red", "green", "blue"]:
> >     acronym = acronym + word[0].upper()
> > print(acronym)
> > ```
> {: .solution}
{: .question}

```python
#Test your code here!
```

> ## Cumulative Sum
>
> Reorder and properly indent the lines of code below
> so that they print a list with the cumulative sum of data.
> The result should be `[1, 3, 5, 10]`.
>
> ```
> cumulative.append(total)
> for number in data:
> cumulative = []
> total += number
> total = 0
> print(cumulative)
> data = [1,2,2,5]
> ```
> > <solution-title></solution-title>
> > ```
> > total = 0
> > data = [1,2,2,5]
> > cumulative = []
> > for number in data:
> >     total += number
> >     cumulative.append(total)
> > print(cumulative)
> > ```
> {: .solution}
{: .question}

```python
# Test your code here!
```

> <question-title>A classic programmer test: Fizz Buzz</question-title>
> [FizzBuzz](https://en.wikipedia.org/wiki/Fizz_buzz) is a classic "test" question that is used in some job interviews to remove candidates who really do not understand programming. Your task is this:
>
> Write a for loop that loops over the numbers 1 to 50.
>
> - If the number is divisible by 3, write Fizz instead of the number
> - If the number is divisible by 5, write Buzz instead of the number
> - If the number is divisible by 3 and 5 both, write FizzBuzz instead of the number
> - Otherwise, write the number itself.
>
> > <solution-title></solution-title>
> > ```python
> > for i in range(1, 50):
> >     if i % 3 == 0 and i % 5 == 0:
> >         print("FizzBuzz")
> >     elif i % 3 == 0:
> >         print("Fizz")
> >     elif i % 5 == 0:
> >         print("Buzz")
> >     else:
> >         print(i)
> > ```
> {: .solution}
{: .question}

```python
# Do a FizzBuzz
```

> <question-title>Identifying Item Errors</question-title>
>
> 1. Read the code below and try to identify what the errors are
>    **without** running it.
> 2. Run the code, and read the error message. What type of error is it?
> 3. Fix the error.
>
> ```
> seasons = ['Spring', 'Summer', 'Fall', 'Winter']
> print(f'My favorite season is {seasons[4]}')
> ```
>
> > <solution-title></solution-title>
> > This list has 4 elements and the index to access the last element in the list is `3`.
> > ```
> > seasons = ['Spring', 'Summer', 'Fall', 'Winter']
> > print(f'My favorite season is {seasons[3]}')
> > ```
> {: .solution}
{: .question}

```python
# Fix me!
seasons = ['Spring', 'Summer', 'Fall', 'Winter']
print(f'My favorite season is {seasons[4]}')
```

> <question-title>Correct the errors</question-title>
>
> This code is completely missing indentation, it needs to be fixed. Can you make some guesses at how indented each line should be?
> ```
> data = [1, 3, 5, 9]
> acc = 0
> for i in data:
> if i < 4:
> acc = acc + i * 2
> else:
> acc = acc + i
> print(f'The value at {i} is {acc}')
> print(f'The answer is {acc}')
> ```
>
> > <solution-title></solution-title>
> >
> >
> > ```python
> > data = [1, 3, 5, 9]
> > acc = 0
> > # There is a : character at the end of this line, so you KNOW the next line
> > # must be indented.
> > for i in data:
> >     # Same here, another :
> >     if i < 4:
> >         acc = acc + i * 2
> >     # And again! Another :
> >     else:
> >         acc = acc + i
> > # But what about these lines?
> > print(f'The value at {i} is {acc}')
> > print(f'The answer is {acc}')
> > ```
> >
> > Here this code is actually ambiguous, we don't know how indented the two prints should be. This very synthetic example lacks good context, but there are three places it could be, with three different effects.
> >
> > There are two bits of knowledge we can use, however:
> > - the first print uses `i`, so it must be within the loop
> > - the second print cannot be indented more than the first print (Why? It would require a block like `for ... :` or `if .. :` to indent further.)
> >
> > The first option, no indentation, prints out the value once per loop, that seems good
> >
> > ```python
> > [...]
> >     else:
> >         acc = acc + i
> >     print(f'The value at {i} is {acc}')
> > ```
> >
> > The second, prints out the value only during the else case, not otherwise.
> >
> > ```python
> >     else:
> >         acc = acc + i
> >         print(f'The value at {i} is {acc}')
> > ```
> >
> > So that's probably wrong, and we should take the first option. That leaves two options for the final print, no indentation, or at the same level as our first print statement. We can guess that we probably want to print out the final result of the loop, and that it should not be indented.
> >
> > ```python
> > data = [1, 3, 5, 9]
> > acc = 0
> > for i in data:
> >     if i < 4:
> >         acc = acc + i * 2
> >     else:
> >         acc = acc + i
> >     print(f'The value at {i} is {acc}')
> > print(f'The answer is {acc}')
> > ```
> >
> {: .solution}
{: .question}

```python
# This code accidentally lost it's indentation! Can you fix it?
data = [1, 3, 5, 9]
acc = 0
for i in data:
if i < 4:
acc = acc + i * 2
else:
acc = acc + i
print(f'The value at {i} is {acc}')
print(f'The answer is {acc}')
```

> <question-title>Trimming a FASTQ string</question-title>
> Given a FASTQ string, and a list with quality scores, use `break` to print out just the good bit of DNA and it's quality score.
>
> ```
> # We've got a Read
> read = """
> @SEQ_ID
> GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
> +
> 55CCF>>>>>>CCCCCCC65!''*((((***+))%%%++)(%%%%).1***-+*''))**
> """.strip().split('\n')
>
> def quality_to_percent(q):
>     return 100 * (1 - (10 ** (q / -10)))
>
> sequence = read[1]
> quality_scores = [ord(x) - 33 for x in read[3]]
>
> for i in ... # TODO
> ```
>
> > <solution-title>Hint: Looping over two variables</solution-title>
> > There are two ways to do this, one you might be able to guess, and one that might be new:
> > 1. Loop over a `range()` using `len(sequence)`. Since `len(sequence) == len(quality_scores)`, when we access the Nth position of either, they match up.
> > 2. `zip(sequence, quality_scores)` will loop over both of these lists together. It produces a new list that looks like `[['G', 20], ['A', 20], ['T', 34]]`.
> {: .solution}
>
> > <solution-title></solution-title>
> > The naïve solution is quite easy and readable:
> > ```
> > for i in range(len(sequence)):
> >     if quality_scores[i] < 15:
> >         break
> >     print(f'Base {i} = {sequence[i]} with {quality_to_percent(quality_scores[i])}% accuracy')
> > ```
> > But we can make this a bit prettier using the `zip()` function:
> > ```
> > for base, score in zip(sequence, quality_scores):
> >     if score < 15:
> >         break
> >     print(f'Base = {base} with {quality_to_percent(score)}% accuracy')
> > ```
> > But note that we don't have the position in the list anymore, so we remove it from the print statement.
> {: .solution}
{: .question}

```python
# We've got a Read
read = """
@SEQ_ID
GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
+
55CCF>>>>>>CCCCCCC65!''*((((***+))%%%++)(%%%%).1***-+*''))**
""".strip().split('\n')

def quality_to_percent(q):
    return 100 * (1 - (10 ** (q / -10)))

# Extract the sequence
sequence = read[1]
# And the quality scores, and map those to the correct values.
quality_scores = [ord(x) - 33 for x in read[3]]

# Write something here
# That loops over BOTH the sequence and Quality Scores.
# And prints them out
# If the quality scores are `<15`, then break and quit printing.
for i in ...
```


topic: data-science
tutorial: gnmx-lecture2
title: Introduction to sequencing with Python (part one)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 1893085f9863a7feb9546dffbf060294
last_modified: 2026-05-15T22:28:11.069333
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture2/tutorial.html

content:
![Imgur image](https://i.imgur.com/1PCleoW.png)

# The problem

The difficulty with sequencing nucleic acids is nicely summarized by [Hutchinson 2007](http://dx.doi.org/10.1093/nar/gkm688):

1. The chemical properties of different DNA molecules were so similar that it appeared difficult to separate them.
2. The chain length of naturally occurring DNA molecules was much greater than for proteins and made complete sequencing seem unapproachable.
3. The 20 amino acid residues found in proteins have widely varying properties that had proven useful in the separation of peptides. The existence of only four bases in DNA therefore seemed to make sequencing a more difficult problem for DNA than for protein.
4. No base-specific DNAases were known. Protein sequencing had depended upon proteases that cleave adjacent to certain amino acids.

It is therefore not surprising that protein-sequencing was developed before DNA sequencing by [Sanger and Tuppy 1951](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1197535/). 

tRNA was the first complete nucleic acid sequenced (see pioneering work of [Robert Holley and colleagues](http://www.jstor.org/stable/1715055) and also [Holley's Nobel Lecture](https://www.nobelprize.org/nobel_prizes/medicine/laureates/1968/holley-lecture.pdf)). Conceptually, Holley's approach was similar to Sanger's protein sequencing: break molecule into small pieces with RNases, determine sequences of small fragments, and use overlaps between fragments to reconstruct (assemble) the final nucleotide sequence. 

The work on finding approaches to sequencing DNA molecules began in the late 60s and early 70s. One of the earliest contributions has been made by Ray Wu (Cornell) and Dave Kaiser (Stanford), who used _E. coli_ DNA polymerase to [incorporate radioactively labelled nucleotides into protruding ends of bacteriphage lambda](http://www.sciencedirect.com/science/article/pii/S0022283668800129?via%3Dihub). It took several more years for the development of more "high throughput" technologies by Sanger and Maxam/Gilbert. The Sanger technique has ultimately won over Maxam/Gilbert's protocol due to its relative simplicity (once dideoxynucleotides has become commercially available) and the fact that it required a smaller amount of starting material as the polymerase was used to generate fragments necessary for sequence determination. 

# Sanger/Coulson plus/minus method

><comment-title>Two Nobel prizes</comment-title>
>[Fred Sanger](https://www.nature.com/articles/505027a) is one of only [four people](https://en.wikipedia.org/wiki/Category:Nobel_laureates_with_multiple_Nobel_awards), who received two Nobel Prizes in their original form (for scientific, not societal, breakthroughs).
{: .comment}

This method builds on the idea of Wu and Kaiser (for *minus* part) and on the special property of DNA polymerase isolated from phage T4 (for *plus* part). The schematics of the method is given in the following figure (from [Sanger & Coulson: 1975](http://www.sciencedirect.com/science/article/pii/0022283675902132?via%3Dihub)):

![Plus/minus method](./images/sanger_plus_minus.png "Plus/minus method (from Sanger & Coulson 1975)")

In this method, a primer and DNA polymerase is used to synthesize DNA in the presence of P<sup>32</sup>-labeled nucleotides (only one of four is labeled). This generates P<sup>32</sup>-labeled copies of DNA being sequenced. These are then purified and (without denaturing) separated into two groups: *minus* and *plus*. Each group is further divided into four equal parts. 

In the case of *minus* polymerase a mix of nucleotides minus one is added to each of the four aliquotes: ACG (-T), ACT (-G), CGT (-A), AGT (-C). As a result in each case DNA strand is extended up to a missing nucleotide. 

In the case of plus only one nucleotide is added to each of the four aliquotes (+A, +C, +G, and +T) and T4 DNA polymerase is used. T4 DNA polymerase acts as an exonuclease that would degrade DNA
from 3'-end up to a nucleotide that is supplied in the reaction. 

The products of these are loaded into a denaturing polyacrylamide gel as eight tracks (four for minus and four for plus; from [Sanger & Coulson: 1975](http://www.sciencedirect.com/science/article/pii/0022283675902132?via%3Dihub)):

![Plus/minus gel](./images/sanger_plus_minus_gel.png "Plus/minus method gel radiograph (from Sanger & Coulson 1975)")

# Maxam/Gilbert chemical cleavage method

In this method DNA is terminally labeled with P<sup>32</sup>, separated into four equal aliquotes.  Two of these are treated with [Dimethyl sulfate (DMSO)](https://en.wikipedia.org/wiki/Dimethyl_sulfate) and the remaining two are treated with [hydrazine](https://en.wikipedia.org/wiki/Hydrazine). 

DMSO methylates G and A residues. Treatment of DMSO-incubated DNA with alkali at high temperature will break DNA chains at G and A with Gs being preferentially broken, while treatment of DMSO-incubated DNA with acid will preferentially break DNA at As. Likewise treating hydrazine-incubated DNA with [piperidine](https://en.wikipedia.org/wiki/Piperidine) breaks DNA at C and T, while DNA treated with hydrazine in the presence of NaCl preferentially breaks at Cs. The four reactions are then loaded on a gel generating the following picture (from [Maxam & Gilbert: 1977](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC392330/pdf/pnas00024-0174.pdf)):

![Maxam/Gilbert gel](./images/maxam_gilbert.jpeg "Radiograph of Maxam/Gilbert gel (from Maxam & Gilbert 1977)")

# Sanger dideoxy method

The original Sanger +/- method was not popular and had a number of technical limitations. In a new approach, Sanger took advantage of inhibitors that stop the extension of a DNA strand at particular nucleotides.
These inhibitors are dideoxy analogs of normal nucleotide triphosphates (from [Sanger et al. 1977](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC431765/pdf/pnas00043-0271.pdf)):

![Sanger ddN](./images/sanger_ddN.png "Sanger ddNTP gel (from Sanger et al. 1977)")

# Original approaches were laborious

In the original [Sanger paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC431765/pdf/pnas00043-0271.pdf) the authors sequenced bacteriophage phiX174 by using its own restriction fragments as primers. This was an ideal set up to show the proof of principle for the new method. This is because phiX174 DNA is homogeneous and can be isolated in large quantities. Now suppose that you would like to sequence a larger genome (say _E. coli_). Remember that the original version of Sanger method can only sequence fragments up to 200 nucleotides at a time. So to sequence the entire _E. coli_ genome (which by-the-way was not sequenced until [1997](http://science.sciencemag.org/content/277/5331/1453)) you would need to split the genome into multiple pieces and sequence each of them individually. This is hard because to produce a readable Sanger sequencing gel each sequence must be amplified to a suitable amount (around 1 nanogram) and be homogeneous (you cannot mix multiple DNA fragments in a single reaction as it will be impossible to interpret the gel). Molecular cloning enabled by the availability of commercially available restriction enzymes and cloning vectors simplified this process. Until the onset of next generation sequencing in 2005 the process for sequencing looked something like this:

* (**1**) - Generate a collection of fragments you want to sequence. It can be a collection of fragments from a genome that was mechanically sheared or just a single fragment generated by PCR.
* (**2**) - These fragment(s) are then cloned into a plasmid vector (we will talk about other types of vectors such as BACs later in the course).
* (**3**) - Vectors are transformed into bacterial cells and positive colonies (containing vectors with insert) are picked from an agar plate.
* (**4**) - Each colony now represents a unique piece of DNA. 
* (**5**) - An individual colony is used to seed a bacterial culture that is grown overnight.
* (**6**) - Plasmid DNA is isolated from this culture and now can be used for sequencing because it is (1) homogeneous and (2) we now have a sufficient amount.
* (**7**) - It is sequenced using universal primers. For example, the image below shows a map for pGEM-3Z plasmid (a pUC18 derivative). Its multiple cloning site is enlarged and sites for **T7** and **SP6** sequencing primers are shown.
* These are the **pads** I'm referring to in the lecture. These provide universal sites that can be used to sequence any insert in between. 


![pGEM3z](./images/pgem3z.png "Map of pGEM-3Z cloning vector (from Promega Inc.)")

Until the invention of NGS the above protocol was followed with some degree of automation. But you can see that it was quite laborious if the large number of fragments needed to be sequenced. This is because each of them needed to be subcloned and handled separately. This is in part why the Human Genome Project, which will be discussed in future lectures in detail, took so much time to complete. 

# Evolution of sequencing machines

The simplest possible sequencing machine is a [gel rig with polyacrylamide gel](https://en.wikipedia.org/wiki/Polyacrylamide_gel_electrophoresis). Sanger used it is his protocol obtaining the following results (from [Sanger et al. 1977](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC431765/pdf/pnas00043-0271.pdf)):

![Sanger gel](./images/sanger_gel.png "Typical polyacrylamide gel separating DNA fragments generate with Sanger's dideoxy method (from Sanger et al. 1977)")

Here for sequencing each fragment four separate reactions are performed (with ddA, ddT, ggC, and ddG) and four lanes on the gel are used. [One simplification of this process](http://www.jstor.org/stable/pdf/2879539.pdf) that
came in the 90s was to use fluorescently labeled dideoxy nucleotides. This is easier because everything can be performed in a single tube and uses a single lane on a gel
(from Applied Biosystems [support site](https://www3.appliedbiosystems.com/cms/groups/mcb_support/documents/generaldocuments/cms_041003.pdf)):

![PA versus chromotogram](./images/gel_versus_chromotogram.png "Comparison between a chromatogram and a polyacrylamide gel readouts (from Applied Biosystems)")

However, there is still substantial labor involved in pouring the gels, loading them, running machines, and cleaning everything post-run. A significant improvement was offered by the development of capillary electrophoresis allowing automation of liquid handling and sample loading. Although several manufacturers have been developing and selling such machines a _de facto_ standard in this area was (and still is) the Applied Biosystems (ABI) Genetics and DNA Analyzer systems. The highest throughput ABI system, 3730_xl_, had 96 capillaries and could automatically process 384 samples. 

# NGS!

384 samples may sound like a lot, but it is nothing if we are sequencing an entire genome. The beauty of NGS is that these technologies are not bound by sample handling logistics. They still require the preparation of libraries,
but once a library is made (which can be automated) it is processed more or less automatically to generate multiple copies of each fragment
(in the case of 454, Illumina, Ion Torrent, PacBio, Oxford Nanopore, Element, Complete Genomics etc...) and loaded onto the machine, where millions of individual fragments are sequenced simultaneously.
We will learn about these technologies later in this course.

# Reading

* 2001 [Overview of pyrosequencing methodology - Ronaghi](http://genome.cshlp.org/content/11/1/3)
* 2005 [Description of 454 process - Margulies et al.](http://www.nature.com/nature/journal/v437/n7057/pdf/nature03959.pdf)
* 2007 [History of pyrosequencing - Pål Nyrén](http://link.springer.com/protocol/10.1385/1-59745-377-3:1)
* 2007 [Errors in 454 data - Huse et al. ](http://genomebiology.com/content/pdf/gb-2007-8-7-r143.pdf)
* 2010 [Properties of 454 data - Balzer et al.](http://bioinformatics.oxfordjournals.org/content/26/18/i420.full.pdf+html)

## A few classical papers

In a series of now classical papers ([Paper 1](http://genome.cshlp.org/content/8/3/175.full), [Paper2](http://genome.cshlp.org/content/8/3/186.full)) Philip Green and co-workers have developed a quantitative framework for the analysis of data generated by automated DNA sequencers: 

![phred1](./images/phred1.png "The first paper")
![phred2](./images/phred2.png "The second paper")

In particular, they developed a standard metric for describing the reliability of base calls:

An important technical aspect of our work is the use of log-transformed error probabilities rather than untransformed ones, which facilitates working with error rates in the range of most importance (very close to 0).
Specifically, we define the quality value $$q$$ assigned to a base-call to be:

$$q = -10\times log_{10}(p)$$

where $$p$$ is the estimated error probability for that base-call. Thus a base-call having a probability of 1/1000 of being incorrect is assigned a quality value of 30. Note that high-quality values correspond to low error probabilities, and conversely.

We will be using the concept of "*quality score*" or "*phred-scaled quality score*" repeatedly in this course. 

## Myers - Green debate

Can we sequence a genome using the shotgun approach?

![Myers/Green debate](./images/myers_versus_green.png "Gene Myers versus Phil Green")

# Simulating Sanger sequencing with Python

[![xkcd1306](https://imgs.xkcd.com/comics/sigil_cycle.png)](https://xkcd.com/1306)

In this lesson we will cover some of the fundamental Python basics including variables, expressions, statements, and functions.

## Prep

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Within JupyterLab start a new Python3 notebook

## Preclass prep: Chapters [1](https://greenteapress.com/thinkpython2/html/thinkpython2002.html), [2](https://greenteapress.com/thinkpython2/html/thinkpython2003.html), [3](https://greenteapress.com/thinkpython2/html/thinkpython2004.html) from "Think Python"

## [Indentation](https://peps.python.org/pep-0008/#indentation) is everything!

><warning-title>Indentation or bust!</warning-title>
>Python is an indented language: code blocks are defined using indentation with [spaces](https://peps.python.org/pep-0008/#tabs-or-spaces)!
{: .warning}

In Python, indentation is used to indicate the scope of control structures such as `for` loops, `if` statements, and function and class definitions. The amount of indentation is not fixed, but it must be consistent within a block of code.
The recommended amount of indentation is 4 spaces, although some developers prefer to use 2 spaces. Indenting is important in Python because it is used to indicate the level of nesting and structure of the code,
which makes it easier to read and understand. Additionally, indentation is also used to indicate which lines of code are executed together as a block.

## The storyline

In this lecture, we will re-implement our Sanger sequencing simulator from the previous lecture and generate realistic gel images. 

## Generate a random sequence

First, we import a module called [`random`](https://docs.python.org/3/library/random.html) which contains a number of functions for generating and working with random numbers 

```python
import random
```
Next, we will write a simple loop that would generate a sequence of pre-set lengths:

```python
seq = ''
for _ in range(100):
    seq += random.choice('ATCG')
```

```python
seq
```

```
CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC
```

## Simulate one polymerase molecule

The code below iterates through each element of a sequence `seq` (assumed to be a string containing nucleotides) and it checks if the current nucleotide is equal to 'A'. If it is, it generates a random number between 0 and 1 using the `random.random()` function.

It then checks if the random number is greater than 0.5. If it is, the code does nothing and proceeds to the next iteration. If the random number is less than or equal to 0.5, the code adds the
lowercase version of the nucleotide ('a') to a string called `synthesized_strand` and then breaks out of the loop.

In every iteration of the loop, regardless of whether the nucleotide is 'A' or not, the code then adds the current nucleotide to the `synthesized_strand` string.

This means that when the current nucleotide is 'A', then the generated random number will decide whether the code will add the nucleotide 'A' or 'a' to the `synthesized_strand`, and it will break out of the loop
after adding the nucleotide to the `synthesized_strand`. To get a good idea of what is going on let's visualize the code execution in 

```python
synthesized_strand = ''

for nucleotide in seq:
    if nucleotide == 'A':
        d_or_dd = random.random()
        if d_or_dd > 0.5:
            None
        else:
            synthesized_strand += 'a'
            break
    synthesized_strand += nucleotide
```

This can be simplified by first removing `d_or_dd` variable:

```python
synthesized_strand = ''
for nucleotide in seq:
    if nucleotide == 'A':
        if random.random() > 0.5:
            None
        else:
            synthesized_strand += 'a'
            break
    synthesized_strand += nucleotide
print(synthesized_strand)
```

```
CTTGCGGCTATa
```

and removing an unnecessary group of `if ... else` statements:

```python
synthesized_strand = ''
for nucleotide in seq:
    if nucleotide == 'A' and random.random() > 0.5:
        synthesized_strand += 'a'
        break
    synthesized_strand += nucleotide
print(synthesized_strand)
```

```
CTTGCGGCTATAGGAATa
```

finally let's make `synthesized_strand += 'a'` a bit more generic:

```python
synthesized_strand = ''
for nucleotide in seq:
    if nucleotide == 'A' and random.random() > 0.5:
        synthesized_strand += nucleotide.lower()
        break
    synthesized_strand += nucleotide
print(synthesized_strand)
```

    CTTGCGGCTATAGGa


## Simulating multiple molecules

To simulate 10 polymerase molecules we simply wrap the code from above into a `for` loop:

```python
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'A' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    print(synthesized_strand)
```

    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTa
    CTTGCGGCTATa
    CTTGCGGCTATAGGa
    CTTGCGGCTa
    CTTGCGGCTa


One problem with this code is that does not actually save the newly synthesized strand: it simply prints it. To fix this we will create a [list](https://greenteapress.com/thinkpython2/html/thinkpython2011.html)
(or an array) called `new_strands` and initialize it by assigning an empty array to it:


```python
new_strands = []
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'A' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
```


```python
new_strands
```




    ['CTTGCGGCTa',
     'CTTGCGGCTa',
     'CTTGCGGCTATAGGAATAa',
     'CTTGCGGCTATAGGAATa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTa']



## Simulating multiple molecules and all nucleotides

And to repeat this for the remaining three nucleotides we will do the following crazy thing:


```python
new_strands = []
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'A' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
    
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'C' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
    
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'G' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
    
for _ in range(10):
    synthesized_strand = ''
    for nucleotide in seq:
        if nucleotide == 'T' and random.random() > 0.5:
            synthesized_strand += nucleotide.lower()
            break
        synthesized_strand += nucleotide
    new_strands.append(synthesized_strand)
```


```python
len(new_strands)
```

    40

Repeating the same code four times is just plain stupid so instead we will write a function called `polymerase`. Here we need to worry about the scope of variables.  The scope of a variable refers to the regions
of the code where the variable can be accessed or modified. Variables that are defined within a certain block of code (such as a function or a loop) are said to have a *local* scope, meaning that they can
only be accessed within that block of code. Variables that are defined outside of any block of code are said to have a *global* scope, meaning that they can be accessed from anywhere in the code.

In most programming languages, a variable defined within a function has a local scope, and it can only be accessed within that function. If a variable with the same name is defined outside the function,
it will have a global scope and can be accessed from anywhere in the code. However, if a variable with the same name is defined within the function, it will take precedence over the global variable and will be used within the function.

There are also some languages that have block scope, where a variable defined within a block (such as an if statement or a for loop) can only be accessed within that block and not outside of it.

In Python, variables defined in the main module have a global scope and can be accessed from any function or module. Variables defined within a function have local scope, and they can only be
accessed within that function. Variables defined within a loop or a block can be accessed only within the scope of the loop or block.


```python
def ddN(number_of_iterations, template, base, ddN_ratio):
    new_strands = []
    for _ in range(number_of_iterations):
        synthesized_strand = ''
        for nucleotide in template:
            if nucleotide == base and random.random() > ddN_ratio:
                synthesized_strand += nucleotide.lower()
                break
            synthesized_strand += nucleotide
        new_strands.append(synthesized_strand)
    return(new_strands)
```


```python
ddN(10,seq,'A',0.5)
```

    ['CTTGCGGCTATAGGa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGAATa',
     'CTTGCGGCTa',
     'CTTGCGGCTa',
     'CTTGCGGCTa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATa']



To execute this function on all four types of ddNTPs with need to wrap it in a `for` loop iterating over the four possibilities:


```python
for nt in 'ATCG':
    ddN(10,seq,nt,0.5)
```

## A bit about [lists](https://greenteapress.com/thinkpython2/html/thinkpython2011.html)

To store the sequences being generated in the previous loop we will create and initialize a list called `seq_run`:


```python
seq_run = []
for nt in 'ATCG':
    seq_run.append(ddN(10,seq,nt,0.5))
```

you will see that the seq run is a two-dimensional list:


```python
seq_run
```




    [['CTTGCGGCTa',
      'CTTGCGGCTATAGGa',
      'CTTGCGGCTa',
      'CTTGCGGCTATa',
      'CTTGCGGCTATAGGa',
      'CTTGCGGCTATa',
      'CTTGCGGCTa',
      'CTTGCGGCTATAGGAATAa',
      'CTTGCGGCTa',
      'CTTGCGGCTATa'],
     ['CTTGCGGCt', 'Ct', 'Ct', 'Ct', 'Ct', 'CTt', 'Ct', 'Ct', 'CTTGCGGCt', 'Ct'],
     ['CTTGc',
      'c',
      'c',
      'c',
      'c',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGc',
      'c',
      'CTTGCGGCTATAGGAATAAAAGGc',
      'c',
      'CTTGc'],
     ['CTTg',
      'CTTGCg',
      'CTTg',
      'CTTg',
      'CTTGCg',
      'CTTGCGGCTATAGGAATAAAAg',
      'CTTGCGGCTATAGGAATAAAAg',
      'CTTg',
      'CTTg',
      'CTTg']]



as you will read in your next home assignment list elements can be addressed by "index". The first element has the number 0:


```python
seq_run[0]
```




    ['CTTGCGGCTa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTa',
     'CTTGCGGCTATa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATa',
     'CTTGCGGCTa',
     'CTTGCGGCTATAGGAATAa',
     'CTTGCGGCTa',
     'CTTGCGGCTATa']



## A bit about [dictionaries](https://greenteapress.com/thinkpython2/html/thinkpython2012.html)

Another way to store these data is in a dictionary, which is a collection of `key:value` pairs where a key and value can be anything:


```python
seq_run = {}
for nt in 'ATCG':
    seq_run[nt] = ddN(10,seq,nt,0.90)
```


```python
seq_run
```




    {'A': ['CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTa',
      'CTTGCGGCTATAGGAATAa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTa',
      'CTTGCGGCTa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCa',
      'CTTGCGGCTATAGGa',
      'CTTGCGGCTATAGGAATAAAa',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAa',
      'CTTGCGGCTATAGGAa'],
     'T': ['CTTGCGGCTAt',
      'CTTGCGGCTATAGGAATAAAAGGCTTt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTAt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCAt',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCAt',
      'CTt'],
     'C': ['CTTGCGGCTATAGGAATAAAAGGc',
      'CTTGCGGCTATAGGAATAAAAGGc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGAc',
      'c',
      'CTTGCGGCTATAGGAATAAAAGGc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTc',
      'CTTGCGGc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGAc',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGc'],
     'G': ['CTTGCGGCTATAGg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTg',
      'CTTGCg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAAGGTCGTGGCACC',
      'CTTg',
      'CTTGCGGCTATAGGAATAAAAGg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCg',
      'CTTGCGGCTATAGGAATAAAAGGCTTTGCGg',
      'CTTGCGGCTATAGGAATAAAAg']}



dictionary elements can be retrieved using a key:


```python
seq_run['A']
```




    ['CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTa',
     'CTTGCGGCTATAGGAATAa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTa',
     'CTTGCGGCTa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCa',
     'CTTGCGGCTATAGGa',
     'CTTGCGGCTATAGGAATAAAa',
     'CTTGCGGCTATAGGAATAAAAGGCTTTGCGGGTAGTGACCGCGCCGCGTATGTAATTCATGGGTGTCGTCGCGCCCTCACAACTGCAa',
     'CTTGCGGCTATAGGAa']



## Drawing a sequencing gel

Now that we can simulate and store newly synthesized sequencing strands terminated with ddNTPs let us try to draw a realistic representation of the sequencing gel. For this, we will use
several components that will be discussed in much greater detail in the upcoming lectures. These components are:

- [`pandas`](https://pandas.pydata.org/) - a dataframe manipulation tool
- [`altair`](https://altair-viz.github.io/) - a statistical visualization library built on top of `vega-light` visualization grammar

These two libraries will be used in almost all lectures concerning Python in this class.

[Gel electophoresis](https://en.wikipedia.org/wiki/Gel_electrophoresis) separates molecules based on mass, shape, or charge. In the case of DNA all molecules are universally negatively charged and thus will always migrate
to (+) electrode. All our molecules are linear single-stranded pieces (our gel is *denaturing*) and so the only physical/chemical characteristic that distinguishes them is *length*.
Therefore the first thing we will do is to convert our sequences into their lengths. For this, we will initialize a new dictionary called `seq_lengths`:


```python
seq_lengths = {'base':[],'length':[]}
for key in seq_run.keys():
    for sequence in seq_run[key]:
        seq_lengths['base'].append(key)
        seq_lengths['length'].append(len(sequence))
```


```python
seq_lengths
```




    {'base': ['A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'A',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'T',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'C',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G',
      'G'],
     'length': [81,
      54,
      19,
      34,
      10,
      87,
      15,
      21,
      88,
      16,
      11,
      27,
      84,
      100,
      57,
      51,
      60,
      100,
      60,
      3,
      24,
      24,
      67,
      39,
      1,
      24,
      67,
      8,
      39,
      47,
      14,
      65,
      6,
      100,
      4,
      23,
      28,
      43,
      31,
      22]}



now let's import `pandas`:


```python
import pandas as pd
```

and inject `seq_lengths` into a pandas *dataframe*:


```python
sequences = pd.DataFrame(seq_lengths)
```

it looks pretty:


```python
sequences
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>base</th>
      <th>length</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>A</td>
      <td>81</td>
    </tr>
    <tr>
      <th>1</th>
      <td>A</td>
      <td>54</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A</td>
      <td>19</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A</td>
      <td>34</td>
    </tr>
    <tr>
      <th>4</th>
      <td>A</td>
      <td>10</td>
    </tr>
    <tr>
      <th>5</th>
      <td>A</td>
      <td>87</td>
    </tr>
    <tr>
      <th>6</th>
      <td>A</td>
      <td>15</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A</td>
      <td>21</td>
    </tr>
    <tr>
      <th>8</th>
      <td>A</td>
      <td>88</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A</td>
      <td>16</td>
    </tr>
    <tr>
      <th>10</th>
      <td>T</td>
      <td>11</td>
    </tr>
    <tr>
      <th>11</th>
      <td>T</td>
      <td>27</td>
    </tr>
    <tr>
      <th>12</th>
      <td>T</td>
      <td>84</td>
    </tr>
    <tr>
      <th>13</th>
      <td>T</td>
      <td>100</td>
    </tr>
    <tr>
      <th>14</th>
      <td>T</td>
      <td>57</td>
    </tr>
    <tr>
      <th>15</th>
      <td>T</td>
      <td>51</td>
    </tr>
    <tr>
      <th>16</th>
      <td>T</td>
      <td>60</td>
    </tr>
    <tr>
      <th>17</th>
      <td>T</td>
      <td>100</td>
    </tr>
    <tr>
      <th>18</th>
      <td>T</td>
      <td>60</td>
    </tr>
    <tr>
      <th>19</th>
      <td>T</td>
      <td>3</td>
    </tr>
    <tr>
      <th>20</th>
      <td>C</td>
      <td>24</td>
    </tr>
    <tr>
      <th>21</th>
      <td>C</td>
      <td>24</td>
    </tr>
    <tr>
      <th>22</th>
      <td>C</td>
      <td>67</td>
    </tr>
    <tr>
      <th>23</th>
      <td>C</td>
      <td>39</td>
    </tr>
    <tr>
      <th>24</th>
      <td>C</td>
      <td>1</td>
    </tr>
    <tr>
      <th>25</th>
      <td>C</td>
      <td>24</td>
    </tr>
    <tr>
      <th>26</th>
      <td>C</td>
      <td>67</td>
    </tr>
    <tr>
      <th>27</th>
      <td>C</td>
      <td>8</td>
    </tr>
    <tr>
      <th>28</th>
      <td>C</td>
      <td>39</td>
    </tr>
    <tr>
      <th>29</th>
      <td>C</td>
      <td>47</td>
    </tr>
    <tr>
      <th>30</th>
      <td>G</td>
      <td>14</td>
    </tr>
    <tr>
      <th>31</th>
      <td>G</td>
      <td>65</td>
    </tr>
    <tr>
      <th>32</th>
      <td>G</td>
      <td>6</td>
    </tr>
    <tr>
      <th>33</th>
      <td>G</td>
      <td>100</td>
    </tr>
    <tr>
      <th>34</th>
      <td>G</td>
      <td>4</td>
    </tr>
    <tr>
      <th>35</th>
      <td>G</td>
      <td>23</td>
    </tr>
    <tr>
      <th>36</th>
      <td>G</td>
      <td>28</td>
    </tr>
    <tr>
      <th>37</th>
      <td>G</td>
      <td>43</td>
    </tr>
    <tr>
      <th>38</th>
      <td>G</td>
      <td>31</td>
    </tr>
    <tr>
      <th>39</th>
      <td>G</td>
      <td>22</td>
    </tr>
  </tbody>
</table>
</div>

In our data, there is a number of DNA fragments that have identical length (just look at the dataframe above). We can condense these by grouping dataframe entries first by nucleotide (`['base']`) and then by length (`['length']`). For each group we will then compute `count` and put it into a new column named, ..., `count`:

```python
sequences_grouped_by_length = sequences.groupby(
    ['base','length']
).agg(
    count=pd.NamedAgg(
        column='length',
        aggfunc='count'
    )
).reset_index()
```

```python
sequences_grouped_by_length
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>base</th>
      <th>length</th>
      <th>count</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>A</td>
      <td>10</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>A</td>
      <td>15</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>A</td>
      <td>16</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>A</td>
      <td>19</td>
      <td>1</td>
    </tr>
    <tr>
      <th>4</th>
      <td>A</td>
      <td>21</td>
      <td>1</td>
    </tr>
    <tr>
      <th>5</th>
      <td>A</td>
      <td>34</td>
      <td>1</td>
    </tr>
    <tr>
      <th>6</th>
      <td>A</td>
      <td>54</td>
      <td>1</td>
    </tr>
    <tr>
      <th>7</th>
      <td>A</td>
      <td>81</td>
      <td>1</td>
    </tr>
    <tr>
      <th>8</th>
      <td>A</td>
      <td>87</td>
      <td>1</td>
    </tr>
    <tr>
      <th>9</th>
      <td>A</td>
      <td>88</td>
      <td>1</td>
    </tr>
    <tr>
      <th>10</th>
      <td>C</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>11</th>
      <td>C</td>
      <td>8</td>
      <td>1</td>
    </tr>
    <tr>
      <th>12</th>
      <td>C</td>
      <td>24</td>
      <td>3</td>
    </tr>
    <tr>
      <th>13</th>
      <td>C</td>
      <td>39</td>
      <td>2</td>
    </tr>
    <tr>
      <th>14</th>
      <td>C</td>
      <td>47</td>
      <td>1</td>
    </tr>
    <tr>
      <th>15</th>
      <td>C</td>
      <td>67</td>
      <td>2</td>
    </tr>
    <tr>
      <th>16</th>
      <td>G</td>
      <td>4</td>
      <td>1</td>
    </tr>
    <tr>
      <th>17</th>
      <td>G</td>
      <td>6</td>
      <td>1</td>
    </tr>
    <tr>
      <th>18</th>
      <td>G</td>
      <td>14</td>
      <td>1</td>
    </tr>
    <tr>
      <th>19</th>
      <td>G</td>
      <td>22</td>
      <td>1</td>
    </tr>
    <tr>
      <th>20</th>
      <td>G</td>
      <td>23</td>
      <td>1</td>
    </tr>
    <tr>
      <th>21</th>
      <td>G</td>
      <td>28</td>
      <td>1</td>
    </tr>
    <tr>
      <th>22</th>
      <td>G</td>
      <td>31</td>
      <td>1</td>
    </tr>
    <tr>
      <th>23</th>
      <td>G</td>
      <td>43</td>
      <td>1</td>
    </tr>
    <tr>
      <th>24</th>
      <td>G</td>
      <td>65</td>
      <td>1</td>
    </tr>
    <tr>
      <th>25</th>
      <td>G</td>
      <td>100</td>
      <td>1</td>
    </tr>
    <tr>
      <th>26</th>
      <td>T</td>
      <td>3</td>
      <td>1</td>
    </tr>
    <tr>
      <th>27</th>
      <td>T</td>
      <td>11</td>
      <td>1</td>
    </tr>
    <tr>
      <th>28</th>
      <td>T</td>
      <td>27</td>
      <td>1</td>
    </tr>
    <tr>
      <th>29</th>
      <td>T</td>
      <td>51</td>
      <td>1</td>
    </tr>
    <tr>
      <th>30</th>
      <td>T</td>
      <td>57</td>
      <td>1</td>
    </tr>
    <tr>
      <th>31</th>
      <td>T</td>
      <td>60</td>
      <td>2</td>
    </tr>
    <tr>
      <th>32</th>
      <td>T</td>
      <td>84</td>
      <td>1</td>
    </tr>
    <tr>
      <th>33</th>
      <td>T</td>
      <td>100</td>
      <td>2</td>
    </tr>
  </tbody>
</table>
</div>

The following chart is created using the `alt.Chart()` function and passing the data as an argument. The `mark_tick()` function is used to create a tick chart with a thickness of 4 pixels.

The chart is encoded with two main axis:

- y-axis which represents the length of the data and is encoded by the `'length'` field of the data.
- x-axis which represents the base of the data and is encoded by the `'base'` field of the data.
The chart also encodes a color, it encodes the `'count'` field of the data, sets the legend to `None`, and uses the `'greys'` scale from the Altair library.

Finally, the chart properties are set to a width of 100 pixels and a height of 800 pixels.

```python
import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q'),
    x = alt.X('base'),
    color=alt.Color('count:Q',legend=None,
                    scale=alt.Scale(scheme="greys"))
).properties(
    width=100,
    height=800)
```

![Gel rendering 1](./images/gel1.svg "A simulated gel image")

And here is a color version of the same graph using just one line of the gel:


```python
import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q'),
    color=alt.Color('base:N',#legend=None,
                    scale=alt.Scale(scheme="set1"))
).properties(
    width=20,
    height=800)
```

![Gel rendering 2](./images/gel2.svg "A simulated gel image using \"colored dyes\"")

## Putting everything together

```python
# Generate random sequences

seq = ''
for _ in range(300):
    seq += random.choice('ATCG')
```

```python
seq
```

```python
# Function simulating a single run of a single polymerase molecule

def ddN(number_of_iterations, template, base, ddN_ratio):
    new_strands = []
    for _ in range(number_of_iterations):
        synthesized_strand = ''
        for nucleotide in template:
            if nucleotide == base and random.random() > ddN_ratio:
                synthesized_strand += nucleotide.lower()
                break
            synthesized_strand += nucleotide
        new_strands.append(synthesized_strand)
    return(new_strands)
```

```python
# Generating simulated sequencing run

seq_run = {}
for nt in 'ATCG':
    seq_run[nt] = ddN(100000,seq,nt,0.95)
```

```python
# Computing lengths

seq_lengths = {'base':[],'length':[]}
for key in seq_run.keys():
    for sequence in seq_run[key]:
        seq_lengths['base'].append(key)
        seq_lengths['length'].append(len(sequence))
```

```python
# Converting dictionary into Pandas dataframe

sequences = pd.DataFrame(seq_lengths)
```

```python
# Grouping by nucleotide and length

sequences_grouped_by_length = sequences.groupby(
    ['base','length']
).agg(
    count=pd.NamedAgg(
        column='length',
        aggfunc='count'
    )
).reset_index()
```

```python
# Plotting (note the quadratic scale for realism)

import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q',scale=alt.Scale(type='sqrt')),
    x = alt.X('base'),
    color=alt.Color('count:Q',legend=None,
                    scale=alt.Scale(type='log',scheme="greys")),
    tooltip='count:Q'
).properties(
    width=100,
    height=800)
```

![Gel rendering 3](./images/gel3.svg "A simulated gel image using quadratic scale")

```python
# Plotting using color

import altair as alt
alt.Chart(sequences_grouped_by_length).mark_tick(thickness=4).encode(
    y = alt.Y('length:Q',scale=alt.Scale(type="sqrt")),
    color=alt.Color('base:N',#legend=None,
                    scale=alt.Scale(scheme="set1")),
    opacity=alt.Opacity('count:N',legend=None),
    tooltip='count:Q'
).properties(
    width=20,
    height=800)
```

![Gel rendering 4](./images/gel4.svg "A simulated gel image using quadratic scale with colors")


topic: data-science
tutorial: gnmx-lecture5
title: Introduction to sequencing with Python (part four)
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: e7fd37c2f11e59ba771fa561f37d41a6
last_modified: 2026-05-15T22:28:11.071251
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture5/tutorial.html

content:
# Reading and writing files in Python

First let's download a file we will be using to your notebooks:

```
!wget https://raw.githubusercontent.com/nekrut/BMMB554/master/2023/data/l9/mt_cds.fa
```

In Python, you can handle files using the built-in `open` function. The `open` function creates a file object, which you can use to read, write, or modify the file.

Here's an example of how to open a file for reading:


```python
f = open("mt_cds.fa", "r")
```

In this example, the open function takes two arguments: the name of the file, and the mode in which you want to open the file. The `r` mode indicates that you want to open the file for reading.

After you've opened the file, you can read its contents using the read method:

```python
contents = f.read()
print(contents)
```

You can also read the file line by line using the readline method:

```python
line = f.readline()
print(line)
```

When you're done reading the file, you should close it using the close method:

```python
f.close()
```

You can also use the `with` statement to automatically close the file when you're done:

```python
with open("mt_cds.fa", "r") as f:
    contents = f.read()
    print(contents)
```

You can also write to files using `write` method (note the `"w"` mode):

```python
f = open("sample.txt", "w")
f.write("This is a new line.")
f.close()
```

If you open an existing file in write mode, its contents will be overwritten. If you want to append to an existing file instead, you can use the `"a"` mode:

```python
f = open("sample.txt", "a")
f.write("This is another line.")
f.close()
```

When you're writing to a file, it's important to make sure you close the file when you're done. If you don't close the file, any changes you make may not be saved.

In addition to reading and writing text files, you can also use Python to handle binary files, such as images or audio files.

Let's download an image:

```
!wget https://imgs.xkcd.com/comics/file_extensions.png
```

Here's an example of how to read an image file:

```python
with open("file_extensions.png", "rb") as f:
    contents = f.read()
```

Note that when working with binary files, you must use the `"rb"` mode for reading and the `"wb"` mode for writing.

There are many more features and methods related to file handling in Python, but the basics covered here should be enough to get you started.

## [Fasta](https://en.wikipedia.org/wiki/FASTA_format)

Fasta is a file format that is commonly used to store biological sequences, such as DNA or protein sequences. In Python, you can read a Fasta file by opening the file, reading the lines one by one, and processing the data as needed.

Here's an example of how you might read a Fasta file in Python:

```python
sequences = {}
with open("mt_cds.fa", "r") as file:
  header = ""
  sequence = ""
  for line in file:
    line=line.rstrip()
    if line.startswith('>'):
      if header != "":
        sequences[header] = sequence
        sequence = ""
      header = line[1:]
    else:
      sequence += line
  if header != "":
    sequences[header] = sequence
```
The code above uses a `with` statement to open the file and read the lines one by one. If a line starts with a `">"`, it is assumed to be a header, and the current sequence is stored in the dictionary using the current header as the key. If the line does not start with a `">"`, it is assumed to be part of the current sequence.


## [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format)

Let's download a sample fastq file:

```python
!wget https://raw.githubusercontent.com/nekrut/BMMB554/master/2023/data/l9/reads.fq
```

Fastq is a file format that is commonly used to store high-throughput sequencing data. It consists of a series of records, each of which includes a header, a sequence, a quality score header, and a quality score string. In Python, you can read a Fastq file by opening the file, reading the lines four at a time, and processing the data as needed.

Here's an example of how you might read a Fastq file in Python:

```python
def read_fastq(file_path):
    records = []
    with open(file_path, "r") as f:
        while True:
            header = f.readline().strip()
            if header == "":
                break
            sequence = f.readline().strip()
            quality_header = f.readline().strip()
            quality = f.readline().strip()
            records.append((header, sequence, quality))
    return records
```
In this example, the `read_fastq` function takes a file path as an argument, and returns a list of records, where each record is a tuple of four strings: the header, the sequence, the quality score header, and the quality score string. The function uses a `while` loop to read the lines four at a time until the end of the file is reached.

You can use this function to read a Fastq file like this:

```python
records = read_fastq("reads.fq")
for header, sequence, quality_header, quality in records:
    print(header)
    print(sequence)
    print(quality_header)
    print(quality)
```
This will print the headers, sequences, quality score headers, and quality scores in the Fastq file. You can modify the read_fastq function to process the data in any way you need.


## [SAM](https://en.wikipedia.org/wiki/SAM_(file_format)) 

Let's download an example SAM file:

```
!wget https://raw.githubusercontent.com/nekrut/BMMB554/master/2023/data/l9/sam_example.sam
```

SAM (Sequence Alignment/Map) is a file format that is used to store the results of DNA sequencing alignments. In Python, you can read a SAM file by opening the file, reading the lines one by one, and processing the data as needed.

Here's an example of how you might read a SAM file in Python:

```python
def read_sam(file_path):
    records = []
    with open(file_path, "r") as f:
        for line in f:
            if line.startswith("@"):
                continue
            fields = line.strip().split("\t")
            records.append(fields)
    return records
```
In this example, the `read_sam` function takes a file path as an argument, and returns a list of records, where each record is a list of fields. The function uses a `with` statement to open the file and read the lines one by one. If a line starts with an `"@"`, it is assumed to be a header and is ignored. If the line does not start with an `"@"`, it is assumed to be a record, and the fields are extracted by splitting the line on tabs.

You can use this function to read a SAM file like this:

```python
records = read_sam("sam_example.sam")
for fields in records:
    print(fields)
```

This will print the fields in the SAM file. You can modify the read_sam function to process the data in any way you need. For example, you might want to extract specific fields, such as the reference name, the start position, and the cigar string.


topic: contributing
tutorial: design
title: Design and plan session, course, materials
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: ["learning-principles"]
tags_json: ["elixir", "train-the-trainers"]
content_hash: 6e3120afa30b6e26d9a29ef618f23b89
last_modified: 2026-05-27T09:01:47.921293
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/design/tutorial.html

content:
**Teaching and training**, core elements of academic life, can be enormously
rewarding but also quite challenging. Instructors are often required to perform
under various constraints, and frequently have to accommodate, engage and
motivate student cohorts with very different backgrounds and aptitudes in
limited time-frames. This can be daunting for experienced teachers, but
especially so for those who're relatively new to teaching and training.

> <comment-title>Definitions of key terms</comment-title>
>
> **Training**: instruction delivered via short courses designed to expand or
> build knowledge & practical skills in a given field, often conducted in the
> workplace or training centre to 'up-skill' members of a workforce
>
> **Teaching**: usually, instruction delivered over long time-scales via a
> series of courses (in schools, colleges, universities, etc.) designed to
> contribute to a formal programme that, if completed successfully, yields an
> accredited qualification in a given field (e.g., a degree)
>
{: .comment}

Formal education enterprises generally begin with **curriculum**
({% cite european2008terminology %}) design: this involves specifying

1. its purpose or **Teaching Goals (TGs)**;
2. its duration;
3. the **Knowledge, Skills and Abilities (KSAs)** intended to be achieved,
   expressed as a set of **Learning Outcomes (LOs)**;
4. how learners will demonstrate achievement of those LOs;
5. the materials, **Learning Experiences (LEs)** and assignments instructors
   will use to support learner progression towards the LOs;
6. the assessments for evaluating student learning and teaching effectiveness.

> <comment-title>Definitions of key terms</comment-title>
>
> **Curriculum**: the inventory of tasks involving the design, organisation &
> planning of an education or training enterprise, including specification of
> learning outcomes, content, materials & assessments, & arrangements for
> training teachers & trainers
>
> **Knowledge, Skills and Abilities (KSAs)**: list of special qualifications and
> personal attributes that learners should have after a training
> - **Knowledge**: the subjects, topics, and items of information that learners
>   should know
> - **Skills**: technical or manual proficiencies which are usually learned or
>   acquired through training.
> - **Abilities**: the present demonstrable capacity to apply several knowledge
>   and skills simultaneously in order to complete a task or perform an
>   observable behaviour.
>
> **Learning Outcomes (LOs)**: the KSAs that learners should be able to
> demonstrate after instruction, the tangible evidence that the teaching goals
> have been achieved; LOs are learner-centric
>
> **Learning Experience (LE)**: any setting or interaction in or via which
> learning takes place: e.g., a lecture, game, exercise, role-play, etc.
>
> **Teaching Goal (TG)**: the intentions of an instructor regarding the purpose
> of a curriculum/course/lesson/activity/set of materials; TGs are
> instructor-centric (also termed instructional objectives)
>
{: .comment}

Emphasis is placed on teaching and learning, and mechanisms for collecting
evidence that learners have changed over time
({% cite tractenberg2020guidelines %}). In this latter sense, the concept of
'curriculum' differentiates formal teaching from training, as formal programmes
usually afford time for learners both to be able to progress and to **demonstrate
their progression**.

By contrast, training courses are much shorter (measured in days or weeks,
rather than years); they hence necessarily focus on acquiring specific KSAs in
limited time-frames, generally without consideration of learner progression
beyond the course. However, the essential features of **effective curricula**
(i.e., those that achieve their stated LOs for the majority of learners) pertain
to instruction on any time-scale; they are thus also relevant for short courses,
and provide important considerations for those involved in, or embarking upon,
course design (whether face-to-face or online).

With this in mind, this tutorial outlines key steps of curriculum development –
and the role of Bloom's taxonomy ({% cite bloom1956taxonomy %}) – that can be
used to inform the design of effective courses.

> <comment-title>Sources</comment-title>
>
> This tutorial is significantly based on {% cite via2020course %}, one session
> of ELIXIR-GOBLET Train the Trainer curriculum and
> {% cite tractenberg2020guidelines %}
>
{: .comment}

> <hands-on-title>Select a topic for a 3 minute training - {% icon time %} 1 min - Silent reflection</hands-on-title>
>
> - Choose a topic for a 3 minutes training related to Galaxy
>
>   Examples:
>   - Introduction to Galaxy interface
>   - How to upload data to Galaxy
>   - Galaxy Training Network
>
> - Identify the target audience
> - Identify the prerequisites
{: .hands_on}

> <tip-title>Check the tutorial on principles of learning</tip-title>
>
> For more details about how learning works and values, check our
> [tutorial]({% link topics/contributing/tutorials/learning-principles/tutorial.md %})
>
{: .tip }

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Formal curriculum design

## Different types of curriculum

To set the scene for our considerations of course design, we examine some of the
foundations for effective curriculum development, drawing heavily on the
curriculum- and course- development guidelines developed by
{% cite tractenberg2020guidelines %}. Notable here is the fact that different
types of curriculum have been defined: *i.e.*, **intended**, **implemented**,
**attained** and **hidden curricula**. Recognising the existence of different
curriculum types (or, perhaps, different curricular outcomes) is important
because, while the intended curriculum is the starting point, it may not be the
curriculum actually attained: *i.e.*, what you aimed to teach and what students
actually learned may not be the same.

To improve outcomes, differences between the intended and attained curricula
need to be minimised. The only way to discover the attained curriculum is to
find out what learning actually occurred. This requires **actionable**
evaluation, to assess whether the teaching goals (TGs) and learning outcomes (LOs) were achieved, to identify
weaknesses in the implementation and to highlight improvements needed to
remediate them. These considerations are key to developing effective curricula
and courses.

> <comment-title>Definitions of key terms</comment-title>
>
> **Actionable**: supportive of a decision, or the taking of some action by a
> learner, instructor or institution
>
> **Attained curriculum**: what learners actually acquire & can demonstrate
> having followed the implemented curriculum ({% cite unesco-ibe-glossary %})
>
> **Hidden curriculum**: unintended curricular effects: unofficial norms,
> behaviours & values that are transferred (not necessarily consciously) by the
> school culture or ethos; this recognises that schooling happens in broad
> social & cultural environments that influence learning ({% cite glossary-education-reform %})
>
> **Implemented curriculum**: or taught curriculum, how the intended curriculum
> is delivered in practice: *i.e.*, the teaching & learning activities, & the
> interactions between learners & teachers, & among learners
>
> **Intended curriculum**: the formal specification of KSAs that students are
> expected to achieve & be able to demonstrate having followed the implemented
> curriculum
>
{: .comment}

> <hands-on-title>Define audience and teaching goal - {% icon time %} 10 min - Silent reflection</hands-on-title>
>
> 1. Take the topic you chose before
> 2. Define teaching objectives
>
>    Describe your goals and intentions as instructor:
>    - stakeholders
>    - potential professions
>    - desired performance and/or competences of course completers
>    - duration
>
{: .hands_on}

## Structuring curriculum design

Curriculum design benefits from being systematic: structured approaches help to
orchestrate and clarify what will be taught, **why it will be taught** and how;
they also afford opportunities to evaluate what does and doesn't work, and hence
what needs to change, ultimately leading to improvements in learning outcomes
({% cite fink2013creating %}, {% cite diamond2008designing %},
{% cite nilson2016teaching %}). Several different frameworks have been devised to
facilitate the design process, but each is motivated by the same underlying
philosophy: to help formulate programmes that promote **meaningful and enduring
learning**. If we're to understand whether we've really achieved this, we must

1. determine the purpose of the programme (what needs it addresses, why it's
   being developed, what learners will gain from it, why it's important),
2. define the intended LOs, and
3. develop **assessment** and evaluation mechanisms that will allow us to
   measure whether the programme successfully met its goals.

> <comment-title>Definitions of key terms</comment-title>
>
> **Assessment**: the evaluation or estimation of the nature, quality or ability
> of someone or something
>
{: .comment}

Not surprisingly, the same principles apply to course design. The process may
seem daunting, but for the sake of simplicity, we focus here on one model: *i.e.*,
that proposed by {% cite nicholls2002developing %}. Before discussing
this further, however, it's helpful to consider another very important tool used
in teaching and learning – Bloom's taxonomy ({% cite bloom1956taxonomy %}).

### Bloom's taxonomy of cognitive complexity

Learning taxonomies are useful tools that can help both to formulate and clarify
LOs, and to arrange them on a scale of increasing complexity. Bloom’s taxonomy
({% cite bloom1956taxonomy %}), probably the most easily understood and widely
used today, features a **six-level hierarchy of cognitive complexity**, ranging
from **Remember** (being able to recall facts and basic concepts) to **Evaluate**
being able to defend opinions or decisions), as illustrated in the following
figure:

![6 levels of Bloom's taxonomy (and their verbs) from bottom (lower-order thinking skills) to top (higher-order 'critical' thinking skills): Remember (recall or reiterate information) with memorize, recognise, identify, describe, reproduce, list, define, label, name, state, outline, order, arrange; Understand (demonstrate understanding of facts) with distinguish, classify, select, review, discuss, indicate, explain, estimate, locate, summarise; Apply (apply knowledge to real situations) with operate, manipulate, experiment, choose, modify, prepare, produce; Analyze (resolve ideas into simple parts, identify patterns) with calculate, examine, model, test, break down, infer, predict, solve; Synthesise (pull ideas into a coherent whole, create new ideas) with combine, formulate, illustrate, imagine, design, invent, compose; Evaluate (make & defend judgements, assess theories & outcomes) with critique, compare ideas, solve, recommend, rate](images/bloom_taxonomy.png "Bloom’s taxonomy of cognitive complexity, & a sample of associated verbs that can be used to create LOs. The verbs reflect a developmental trajectory, from lower- to higher-order thinking skills; they should be concretely observable & relate to how achievement of target LOs will be assessed. Source: {% cite via2020course %}")

As can be seen from the figure, each Bloom's level is accompanied by a set of
active verbs that express expected, measurable learner behaviours at that level:
*e.g.*, achieving the level **Understand** means to be able to classify, select or
explain a piece of information: here, _classify, select, explain_ are
observable, assessable behaviours that can be readily encapsulated in coherent
LOs.

Typical illustrations of the taxonomy, like that in the figure, depict
successive cognitive levels, suggesting that learners must achieve one level
before advancing to the next, implying a **developmental trajectory** from lower-
to higher-order cognitive skills. However, this structure should not be regarded
as completely rigid; indeed, {% cite anderson2001taxonomy %} published a revised
version in 2001 in which they placed **Synthesise** (the ability to create new or
original work) at the top of the hierarchy in place of **Evaluate**.
Notwithstanding the minutiae, it is perhaps more fruitful to regard the taxonomy
as a continuum or spectrum of cognitive levels, where each merges into the next,
providing a structured tool to help express measurable, assessable LOs, in which
the cognitive levels are made explicit.

Ultimately, we can use Bloom's taxonomy to **align instruction with learners' levels of complexity of thinking** (and experience). In practice, you can use the Bloom’s levels of cognitive complexity (and the [accompanied actionable verbs in the previous link](https://tips.uark.edu/blooms-taxonomy-verb-chart/)) to:
* write Learning Outcomes
* design instruction and learning experiences i.e. how you are to teach/train, which infrastructure and how to present the materials etc
* assess learning (taking place during curricula or course and after)

The cognitive aspects embodied in LOs are important. Teaching should promote
more complex behaviours than just recall or recognition (unless remembering is
the intended LO), and push learners to achieve greater cognitive complexity
({% cite nilson2016teaching %}, {% cite ambrose2010learning %},
{% cite weinstein2018understanding %}, {% cite roediger2006power %},
{% cite jensen2014teaching %}). This can be done by embedding development in
learning activities and materials, and ensuring that LOs reflect the lowest to
the highest levels of cognitive complexity realistically achievable on
completing those activities or having engaged with those materials (realistic
aims are key, especially for short courses: *e.g.*, expecting learners with no
prior subject knowledge to achieve the level **Evaluate**, say in a 1-day course,
will guarantee failure and frustration for learners and instructors alike).

# Nicholl's five phases of curriculum design

The backdrop for our considerations of course design is Nicholls' paradigm for
curriculum development, illustrated in:

![(in a clockwise direction) Phase 1: Learning Outcomes (Los) --> Simple, measurable, achievable, realistic, time-bound? --> Phase 2: Learning Experiences (LEs) --> LEs aligned with LOs? --> Phase 3: Content --> Support LEs? Promote LOs? --> Phase 4: Assessment --> Aligned with LEs? Support progress towards LOs? --> Phase 5: Evaluation --> LOs achievable by most learners? Content, LEs & assessments support LOs?](images/nicholl_paradigme.png "Nicholls’ phases of curriculum design & their dependencies. For each phase, key considerations are shown (diamonds). Where these are not satisfied, that or previous phases should be revisited (red arrows), otherwise it is safe to move to the next phase(s) (green arrows). When all considerations are satisfied, the curriculum or course can be characterised, with concrete evidence, as successful (star). By simplified we mean that here you only see the arrows going from one box to the following in line. However, usually we need to revise and go back one or several boxes if we have failed to specify or articulated Learning outcomes from the beginning. This is what we mean by Learning Outcomes drives the whole process. Source: {% cite via2020course %}")

Its five-phase structure has been briefly summarised by
{% cite tractenberg2020guidelines %}, as follows:

1.  Select LOs;
2.  Select or develop LEs that will help learners achieve the LOs;
3.  Select or develop content relevant to LOs;
4.  Develop assessments to ensure learners progress toward LOs;
5.  Evaluate the effectiveness of LEs for leading learners to LOs.

As can be seen from [Figure 2](#figure-2), the model's phases are interdependent; all are
ultimately dependent on the first – defining LOs. Moreover, the phases are
iterative: this means that LOs influence later decisions, but later decisions
may also reflect backwards, thereby providing opportunities to check for
alignment of each phase to the target LOs (in other words, to ensure that
successive phases are mutually consistent with, and supportive of, the LOs).
Thus, the role of LOs is pivotal: they must have specific characteristics to
function, and support each of the other phases as they do.

[Figure 2](#figure-2) illuminates an important feature of the model: that **LOs are the
starting point, and drive all decision-making**. This is just as true for courses
as it is for programmes ({% cite diamond2008designing %}). Missing from the model,
however, is the dependence of LOs on a hierarchy of cognitive complexity that
establishes a developmental trajectory, like that seen in Bloom's taxonomy. We
reflect on this crucial point, and its relevance for course design, in the
discussion of each of the five phases below.

## Define intended Learning Outcomes (LOs)

Just as for curricula, Phase 1 of course design begins with stating the LOs. To
help formulate LOs, it's important to take a step back and think about what you
aim to achieve (i.e., what are your TGs and the KSAs you intend to be achieved?),
how you propose to get there, and how you’ll know you succeeded.
{% cite messick1994interplay %} encapsulated this process in the form of three
succinct questions:

1. What KSAs are the targets of instruction (and assessment)?
2. What learner actions/behaviours will reveal these KSAs?
3. What tasks will elicit these specific actions or behaviours?

These questions were originally posed in 1994 in the context of assessment.
Their focus on KSAs – the LOs – thus guides not only the creation of relevant
tasks (to reveal the target KSAs) but also the rational development of
**appropriate** assessments: i.e., they provide a framework for, and clarify,
**what to assess**. The questions can thus support all phases of course
development, starting with the selection of intended KSAs stated in a set of
LOs.

The "Learning outcomes" (LOs) are essentially statements that tell what the learners will be to do at the end of the session. For this reason, every well-designed training should start with LOs.

The first step of good instructional design consists is identifying the set of Knowledge, Skills, Abilities (KSAs) that learners are expected to acquire by the end of instruction.

### Teaching objectives and Learning Outcomes

**Objectives** are statements of what you are setting out to teach, although expressed as if the students were going to learn it. Despite Aims are usually intended to be more general and encompassing than Objectives, they are essential the same.

**Outcomes** (more accurately "desired outcomes") are statements of what you might (in principle) assess. You may not end up assessing all of them, but they are statements of what a student will know or be able to do, if she or he has learned everything in the course or session.

### Learning objectives and learning outcomes

> <comment-title>Sources</comment-title>
>
> The following section is adapted from the webpage [http://provost.rpi.edu/learning-assessment/learning-outcomes/objectives-vs-outcomes](http://provost.rpi.edu/learning-assessment/learning-outcomes/objectives-vs-outcomes) and the [Operational definitions:](https://irds.stanford.edu/sites/g/files/sbiybj10071/f/clo.pdf) document from the [Rensselaer Polytechnic Institute Office of the Provost](http://provost.rpi.edu/institutional-research).
{: .comment}

The terms Learning Objectives and Learning Outcomes are often used interchangeably by the community. Even worse, they are often abbreviated by the acronym (LO), thus introducing even more ambiguity. In our opinion there is no need. Objectives (aims or goals) and Outcomes (results) may overlap, but are not genuinely the same.

Furthermore, American educational psychologists tend to use "Learning Objectives" to mean "Learning Outcomes" and "Teaching aims, goals or objectives" to mean "Learning Objectives".

In the attempt of removing ambiguities we will use the term **Teaching Objectives** to mean:

* Objectives describe the goals and intentions of the instructor.
* Objectives, often termed the input in the course, state the purpose and goals of the course.
* Objectives focus on content and skills important within the classroom or programme.
* Objectives may describe what the instructors will do.
* Objectives can often be numerous, specific, and detailed.

And the term **Learning Outcomes** to mean:

* Student Learning Outcomes catalog the overarching "products" of the course and are the evidence that the goals or objectives were achieved.
* Learning Outcomes are statements that describe or list measurable and essential mastered content-knowledge—reflecting skills, competencies, and knowledge that students have achieved and can demonstrate upon successfully completing a course.
* Outcomes express higher-level thinking skills that integrate course content and activities and can be observed as a behavior, skill, or discrete useable knowledge upon completing the
course.
* Outcomes are exactly what assessments are intended to show – specifically what the student will be able to do upon completing the course.
* An assessable outcome can be displayed or observed and evaluated against criteria.
* Outcomes are clear and measurable criteria for guiding the teaching, learning, and assessment process in the course.
* Statements expressing which KSAs learners will be able to demonstrate upon completion of a learning experience or a sequence of learning experiences
* What learners will be able to do for the end of instruction and the teacher / instructor can in principle evaluate

Going beyond the definition however, a learning outcome needs to be also practical, i.e. there is a need to assign observable actions in order to let teacher assess them.

#### How to write learning outcomes using OBSERVABLE verbs

In order to write learning outcomes, you can use the following scheme:

1. Think about what learners will be able to do by the end of instruction
2. Use the sentences:

    *By the end of the lesson (session/course/instruction) the learner will be able to.........*

    OR

    *The successful learner will be able to......*

3. Replace dots with a verb that you can assess (name, explain, solve, distinguish, etc.).
4. Avoid verbs that are open to many interpretations: appreciate, have faith in, know, learn, understand, believe
5. Keep in mind that learning outcomes can be as specific as you want. You may even think about a different learning outcome for each task you assign to learners.

Identify observable and non observable actions (verbs)

> <hands-on-title>Identify observable and non observable actions (verbs) - {% icon time %} 3 min</hands-on-title>
>
> Here is a list of verbs:
> - Compare
> - Classify
> - Distinguish
> - Explain
> - Believe
> - Give examples of
> - Define
> - Learn
> - Describe
> - List
> - Recognize
> - Know
> - Connect
> - Measure
> - Identify
> - Have faith in
>
> * Write in the shared notes which ones describe observable actions
> * Write in the shared notes which ones are instead open to multiple interpretation
>
{: .hands_on}

<!--{% include _includes/quiz.html id="verbs.yaml" %}-->

> <tip-title>Observable and non observable actions (verbs)</tip-title>
>
> In classifying verbs, it may help to focus on how you would assess the corresponding action / behaviour.
> Example: "identify"
> - Q: What would I do to assess whether a learner is able to identify something?
> - A: Ask them to do this challenge.
{: .tip}


<!--
> <comment-title>What are Learning Outcomes (LOs)?</comment-title>
>
> - Statements expressing which KSAs learners will be able to demonstrate upon
>   completion of a learning experience or a sequence of learning experiences
> - What learners will be able to do for the end of a lesson and the teacher can
>   in principle evaluate
>
{: .comment}
-->



> <question-title>Evaluate LOs</question-title>
>
> 1. Select a tutorial or a course
>
>    Several possibilities:
>    1. Take a course you currently run, plan to run or have run in the
>       past
>    2. Take your favorite tutorial from the [GTN Material]({{ site.baseurl }})
>    3. Take the [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %})
>       introducing to data import, that we created for this lesson
>
> 2. Check the content
> 3. Are its intended LOs stated?
>
>    > <solution-title>for any GTN tutorials, including the dummy one</solution-title>
>    >
>    > LOs are always stated on the top for the tutorial, in the Overview box
>    >
>    {: .solution}
>
{: .question}

Writing coherent LOs is challenging: they must contain appropriate (Bloom's)
verbs ([Figure 1](#figure-1)) that express measurable, observable and assessable actions,
accurately describing what successful learners will be able to do – **and at what
level of cognitive complexity** – after instruction.

Various characteristics of, and principles for articulating, LOs have been
published (e.g. {% cite stanford-institutional-research-decision-support %} or
{% cite national2016higher %}): some of these are listed briefly in the box
below.

> <comment-title>Learning outcomes should</comment-title>
>
> - **be specific & well defined**: LOs should concisely state the specific KSAs
>   that learners should develop as a result of instruction;
> - **be realistic**: LOs must be attainable given the context and resources
>   available for instruction, and consistent with learners’ abilities,
>   developmental levels, prerequisite KSAs, and the time needed vs. time
>   available to achieve them;
> - **rely on active verbs, phrased in the future tense**: LOs should be stated
>   in terms of what successful learners will be able to do as a result of
>   instruction;
> - **focus on learning products, not the learning process**: LOs should not
>   state what instructors will do during instruction, but what learners will be
>   able to do as result of instruction;
> - **be simple, not compound**: LOs shouldn’t include compound statements that
>   join two or more KSAs into one statement;
> - **be appropriate in number**: LOs should be deliverable and assessable
>   within the time available for instruction;
> - **support assessment that generates actionable evidence**: here, actionable
>   means supportive of a decision, or taking some action by a learner or
>   instructor.
>
{: .comment}

<!-- NILOA five principles for articulating LOs -->

> <hands-on-title>Learning outcomes - {% icon time %} 2 min - Silent working</hands-on-title>
>
> 1. Take the small lesson you selected at the beginning
> 2. Try to jot a few LOs down
>
{: .hands_on}

Given their detail and complexity, and the importance of aligning the
instructional inputs you devise with the outcomes for learners you intend, it
can be hard to know where to start. This possibly explains why it may feel
easier to begin developing a course by selecting its content rather than first
trying to understand its impact on student learning. Nevertheless, ensuring that
target LOs meet, or are consistent with, the characteristics outlined in the box
below helps to promote better alignment of instructional inputs and learner
outcomes.

<!--{% include _includes/quiz.html id="evaluate_learning_outcomes.yaml" %}-->

> <question-title>Evaluate LOs</question-title>
>
> "By the end of the course learners will know the Pythagora's theorem
> 
> What do you think about this LO?
>
> > <solution-title>Evaluate Learning outcomes - Solution</solution-title>
> >
> > "By the end of the course learners will know the Pythagora's theorem"
> >
> >  This LO is not good:
> >
> > - How to assess whether learners know the Pythagora's theorem?
> > - What do we mean by "knowing" the Pythagorean theorem?
> >   - They are able state it?
> >   - They are able explain it?
> >   - They are able apply it?
> >   - They are able demonstrate it?
> >   - They are able to use it in solving problem?
> >
> > It would make more sense to ask: "What will learners be able to do to show they know the Pythagorean theorem?"
> >
> {: .solution}
{: .question}

In short, when defining LOs, the key question to ask is, are they SMART?

- **S**pecific / **S**imple
- **M**easurable
- **A**chievable / **A**ttainable
- **R**ealistic / **R**elevant
- **T**ime-bound / **T**ransferable

If they do not satisfy this test, they should be revised; only when they meet
these criteria is it safe to progress to Phase 2, as shown in [Figure 2](#figure-2).
SMART LOs will be the roadmap as we plan out instructional strategies, write the
content and **create assessments**.

> <hands-on-title>Create SMART LOs - {% icon time %} 10 min - Silent working</hands-on-title>
>
> 1. Take the LOs you created
> 2. Consider if your LOs are SMART
> 3. Try to revise any of them that do not meet the SMART criteria as
>    follows:
>
>    1. Select an active verb that can (in principle) be observed & assessed
>    2. Complete the sentence
>
>       *"At the end of this course, learners will be able to..."*
>
>       If it helps, review the verbs of the Bloom's taxonomy listed in [Figure 1](#figure-1).
>
>       It's important to:
>
>       - Focus here on what learners will be able to do at the end of
>         instruction, *e.g.* *will they be able to...*
>         - *... describe its content?*,
>         - *... explain a concept?*,
>         - *... implement an algorithm?,
>         - *... solve a problem?*,
>         - *... evaluate results?*
>
>       - Avoid verbs open to multiple interpretation
>       - Use a verb that describes an observable action
>
> 3. Determine how well you have structured your LOs
>
>    1. Visit the [Intended Learning Outcome Advisor](https://web.cs.manchester.ac.uk/iloadvisor)
>    2. Paste each LOs into the input box
>    3. Press the **SUBMIT** button
>
>    How well did you do?
>
> 4. Consider revising your LOs if the Advisor identified any issues
>
{: .hands_on}

<!-- ELIXIR TtT

> <hands-on-title>Define audience, goal and outcomes (10 min)</hands-on-title>
>
> 5. Identify the learning experiences
> 6. Start thinking about the content - but don’t spend time in preparing this
>    yet
>
{: .hands_on}

-->

Ultimately, **LOs provide the necessary structure and context for decision-making
by instructors (and learners), hence their primary role in course design**.

## Select LEs that will lead to the LOs

Phase 2 involves identifying the most appropriate LEs to lead learners to the
intended LOs. It is important to appreciate that different LEs can lead learners
to demonstrate different Bloom's-level accomplishments: *e.g.*,
- lectures differ from problem-sets – solving problems helps learners to work
  with, and manipulate, information rather than passively listening to it;
- similarly, lab exercises differ from writing computer programs – writing
  original code affords learners the opportunity to create something new rather
  than simply following instructions.

Some example LEs are listed in Table 1, together with the Bloom's level and the
kinds of TG (Teaching Goals) and LO that each may support.

| Learning experience | Highest Bloom's levels supported | Example TG(s) _"This LE will allow me to..."_                                                                                                                       | Example LO(s) _"Learners will be able to..."_                                                                                 |
| ------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Lecture, webinar    | Remember, Comprehend             | Inspire learners, ignite learners' enthusiasm, clarify/explain a concept, provide an overview, give context, summarise content                                    | _list_ the key points of the lecture/webinar, _summarise_ take home message(s)                                              |
| Exercise, practical | Apply, Analyse                   | Help learners digest course materials, solve typical problems, apply knowledge, show how to do things with appropriate guidance, give an idea of how a tool works | _follow_ a set of instructions or protocol, _calculate_ a set of results or outcomes from a given protocol                  |
| Flipped class       | Apply, Analyse                   | Teach learners how to formulate questions, help learners to memorise new information & concepts, or analyse & understand course materials                         | _summarise_ the content material, \*ask\* appropriate questions                                                             |
| Peer instruction    | Synthesise, Evaluate             | Prepare learners to defend an argument, give learners opportunities to explain things, thereby helping to develop critical thinking & awareness                   | _explain_ how they solved an exercise, _evaluate_ others' choices/decisions, _diagnose_ errors in the exercise-solving task |
| Group discussion    | Synthesise, Evaluate             | Give learners opportunities to practice questioning, develop new ideas & critical thinking                                                                        | _communicate_ their own ideas, _defend_ their own opinions                                                                  |
| Group work          | Synthesise, Evaluate             | Promote collaborative work & peer instruction, provide opportunities for giving/receiving feedback, & digesting course materials                                  | _provide_ feedback on their peers' work, _share_ ideas, _explain_ the advantages of team-work                               |
| Problem-solving     | Synthesise, Evaluate             | Promote learner abilities to identify & evaluate solutions, develop new ideas, make decisions, evaluate decision effectiveness, troubleshoot                      | _diagnose_ faulty reasoning or an underperforming result, _correct_ errors                                                  |

> <comment-title>Definitions of key terms</comment-title>
>
> **Exercise**: an activity designed to help learners to mentally put into
> practice learned skills & knowledge
>
> **Flipped class**: a learner-centred approach in which students are introduced
> to new topics prior to class; class time is then used to explore those topics
> in greater depth via interactive activities
>
> **Group discussion**: an in-class, learner-centred approach in which students
> discuss ideas, solve problems &/or answer questions, guided by the instructor
>
> **Group work**: a learner-centred approach in which students are organised
> into groups (& perhaps assigned specific roles) & are given tasks to perform
> collaboratively
>
> **Lecture**: a didactic approach in which oral presentation is used to
> describe & explain concepts & to impart facts
>
> **Peer-instruction**: an interactive, in-class, learner-centred approach in
> which groups of two or more students briefly discuss a question or assignment
> given by the instructor
>
> **Practical**: an activity to put into practice learned skills & knowledge,
> generally in a lab setting
>
> **Problem-solving**: a learner-centred approach in which students are required
> to systematically investigate a problem by building or determining the best
> strategy to solve it (using what is known to discover what is not known)
>
> **Webinar**: a lecture delivered online
>
{: .comment}

Having defined SMART LOs in Phase 1, Phase 2 thus hinges on choosing the most
appropriate LEs to best lead learners towards them: if LOs include, for example,
being able to write a computer program, then the LEs must allow learners to
apply the knowledge they have acquired and to demonstrate that they’ve written a
piece of functional code: i.e., LEs and LOs must be aligned (if they are not,
this can lead directly to a gap between instructional inputs and intended
outcomes, which is one reason why course evaluation to detect such misalignments
is so crucial). If LEs don’t satisfy this criterion, alternative LEs should be
found, or the LOs should be revisited and revised before progressing to Phase 3
(as shown in Figure 2).

Sometimes, it may be necessary to use specific LEs: e.g., if a course is
lecture-based, it may not be possible to choose alternatives; or a particular
teaching scenario may not allow for an ideal LE. Even if you’re not in a
position to select the most appropriate LEs, the LOs must still be consistent
with the given LEs; and if those LEs won’t help learners to achieve the LOs,
then the LOs should be revised – i.e., LOs are still the most important feature
to consider.**The key is to determine exactly what specific LEs can contribute
to learning or how they will help move learners towards the LOs.**

> <question-title>Evaluate LEs in GTN tutorials</question-title>
>
> - Think of the course or tutorial you selected before (*e.g.*, the
>   [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %}))
>
> - Are any of the LEs listed in Table 1 used in the course or tutorial you
>   selected?
>
>   > <solution-title>for GTN tutorial, including the dummy one</solution-title>
>   >
>   > Most of the GTN tutorials are using Practical and Exercises as LEs.
>   >
>   {: .solution}
>
> - What kinds of achievement or learning are they most likely to promote?
>   Is this consistent with the stated LOs?
>
>   > <solution-title>for GTN tutorial, including the dummy one</solution-title>
>   >
>   > They are most likely to show how to do things and give an idea of how a tool
>   > work.
>   >
>   {: .solution}
>
> - Does the course or tutorial have any LO that can't be achieved through the
>   LEs that the course currently uses?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   > "Determine the best strategy to get data into Galaxy" may not be achieved
>   > with only practical
>   >
>   {: .solution}
>
> > <solution-title>LEs in GTN tutorials</solution-title>
> >
> > Most of the GTN tutorials are using Practical and Exercises as LEs.
> >
> > They are most likely to show how to do things and give an idea of how a tool works.
> >
> > In the [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %}),
> > the LO "Determine the best strategy to get data into Galaxy" may not be achieved
> > with only practical
> >
> {: .solution}
{: .question} 

<!--{% include _includes/quiz.html id="LEs_GTN_tutorials.yaml" %}-->

## Select and develop content relevant to the LOs

### Select content

With LOs and LEs aligned, Phase 3 involves finding the most appropriate
**content** to support learners to achieve the intended LOs. Regardless of where
content is drawn from, what matters is how it supports the LOs.

> <comment-title>Definitions of key terms</comment-title>
>
> **Content**: a specific subject or topic item (e.g., DNA, RNA, proteins, a
> biochemical pathway, R programming) that is the target of learning
>
{: .comment}

To this end, content selection should be judicious: it shouldn't try to be all-encompassing
({% cite nilson2016teaching %}), but should consider the target Bloom’s levels in
the LOs, and the preparation of learners. Once content that's considered to be
the core of a course has been identified, additional **auxiliary** materials can
also be selected, and offered, say, as 'further reading'. Using LOs to drive
content selection in this way thereby both provides focus (avoiding the
temptation to squeeze as much as possible into a course to ensure coverage
({% cite mckeachie2013mckeachie %}, {% cite lujan2006too %})), and increases the
likelihood of accomplishing those LOs.

Sometimes, specific content may be deemed essential ({% cite de2017curriculum %}).
Nevertheless, this must still support the LEs and promote achievement of the
LOs; if it doesn't, then additional content and/or LEs should be considered that
will, or the LOs themselves should be revised, to prevent misalignment of
instructional inputs and learning outcomes. Overall, the role of **LOs here is to
help focus on relevant content, and avoid material that's either
non-essential/too broad or too narrow**.

If content is, or seems, fixed, it can be difficult to make the shift from
content- to LO-driven decisions. In this case, Messick and Bloom's can be used to make
adjustments so that what it means to learn that content or what it looks like to have learned that
content can be made concrete and observable. If such fixed content does not lead to observable
behaviors that are, or support the achievement of, LOs, then additional content should be
considered that will; otherwise, it may be necessary to discuss the feasibility of modifying the
content or revising the LOs with those who are able to make changes, to avoid misalignment of
intended and actual curricula

<!-- >> <question-title>Evaluate content</question-title>
>
> - Think of the course or tutorial you selected before (*e.g.*, the
>   [dummy tutorial]({% link topics/introduction/tutorials/dummy/tutorial.md %}))
>
> - Review the LOs and identify which parts of the course content support the
>   LOs
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Map out the relationships between the LOs, the LEs via which they will be
>   delivered & those specific items of content (e.g., item A supports LO 1, &
>   will be delivered using a lecture)
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Is there any piece of content that doesn't support any LO(s)?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Is there at least one piece of content for each LO?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
> - Is there at least one LE for each piece of content?
>
>   > <solution-title>for the dummy tutorial</solution-title>
>   >
>   >
>   {: .solution}
>
{: .question}-->

Defining content can be complicated, specially to identify how it can cover the
different LOs. Trainers may need to tools to help them defining content,
*e.g.* concept maps.

### Concept maps

Concept maps are rough and visual scheme on how the different concepts are linked.
They begin with a main idea (or concept) and then branch out to show how
that main idea can be broken down into specific topics.

For example, the concepts around the nucleus could be represented using the
following concept map:

![Example of concept map to explain the nucleus. It start on the top with "Nucleus", two arrows leave from this. On the left "Nucleus -- has function -->" links to "has function --> performing chemical activities" and "has function --> cellular reproduction". On the right, "Nucleus -- consist of --> chromosomes", which splits into: on the left "chromosomes -- has function --> inheritance"; on the right "chromosomes -- composed of -->" with 2 arrows to the left "-- composed of --> DNA" which then split into "DNA -- composed of --> genes" and "DNA --> has function --> carrying of information", to the right "-- composed of --> proteins"](images/nucleus_concept_map.jpg "Example of concept map to explain the nucleus. 9 nodes and 6 edges, so 15 concepts in total")

***Why concept maps?*** Concept maps are graphical tools for organizing and
representing knowledge. They include concepts and the relationships between them.

***How to use concept maps?*** Concept maps can be used by trainers to help building
content, by breaking training in small pieces (7+/2 concepts) and by helping
trainers and helpers to connect things together. A good concept map usually
starts with a focused question, a context, and is built with iteration and
feedback.

For example, to answer the questions *"What are the FAIR principles?"*
1. we start by listing the keywords that should be discussed: Research data,
Research models, Findable, Accessible, Interoperable, Re-usable
2. we link them together in a concept map:

  ![It starts on the top with a box "What are FAIR principles?". An arrow goes from this box to a box below "Set of guiding principles". From the 2nd box, an arrow goes down with "to make" written and then splits toward 2 boxes: "Research data" on the left, "Research models" on the right. A line connects these 2 boxes with "and" written on it. An arrow starts from research data and splits into 4 arrows pointing to boxes (from left to right): "Findable", "Accessible", "Interoperable", "Re-usable"](images/fair_concept_map.png "Example of concept map to explain the FAIR principles")

Concept maps try to objective but they depend on their creators. Everyone has
their own concept map, like the way to go from point A to point B from 2 persons
living in the same city.

In curriculum/lesson/session planning, concept maps present key concepts in a
highly concise manner. They can take into account the existing knowledge of
learners and new concepts This helps in the teaching plan to measure how much
can be covered, and the hierarchical organization suggests a sequence to cover
material.

> <hands-on-title>Draw a concept map - {% icon time %} 15 min - Silent working</hands-on-title>
>
> - Draw a concept map of your topic of interest
> - Start with a question
> - Include 7 (+/- 2) concepts
> - Include relationships and cross-links between these concepts
> - Arrange it in a hierarchical structure with the key concepts on top
> - Count the items (concepts and links)
{: .hands_on}

> <hands-on-title>Give / receive feedback on concept maps - {% icon time %} 10 min - Group of 2 persons</hands-on-title>
>
> - Pair with someone
> - Check the concept name of your partner
> - Write to your partner
>   - One thing you are confuse/not sure about the map
>   - One thing you like/it is clear about the map
>
{: .hands_on}

> <tip-title>Further reading</tip-title>
>
> For more details about concept maps, we recommend you {% cite novak2008theory %}
> ([PDF](http://cmap.ihmc.us/publications/researchpapers/TheoryUnderlyingConceptMaps.pdf))
>
{: .tip }

### Content

- **Content collection**: collecting appropriate content to the needs and
  capabilities of your target audience
- **Content reduction**: one of the biggest challenges in designing training
  courses is the reduction of content to the training format. key points!

> <hands-on-title>Producing the content and the training material - {% icon time %} 15 min</hands-on-title>
>
> - Prepare the content for your 3 min training
>
>    The structure should be something like:
>
>    - 0:20 sec - Introduction
>    - 2:20 sec - Topic
>    - 0:20 sec - Conclusion
>
> - Use your concept map and adapt as needed
{: .hands_on}

> <details-title>From a 2-min presentation to a lesson/session: a plan</details-title>
>
> Template to help developing plan:
>
> > | Time | Activity | Description | Goal |
> > | ---- | -------- | ----------- | ---- |
> > |      |          |             |      |
> > |      |          |             |      |
> > |      |          |             |      |
> {: .matrix}
>
> For example: for a 1h15 session
>
> > Time | Activity | Description | Goal
> > --- | --- | --- | ---
> > 9:00 - 9:15 | Warm-up | Learners summaries the key points of each session from the previous day and answer questions from the audience. The instructor describes the plan of the day in detail  | Retrieval from memory, repetition, get prepared for new topics, expose learners
> > 9:15 - 9:25 | Lecture  | Python functions | Learning to write a function, about function input and output, and how to call a the function
> > 9:20 - 10:00 | Practical activity | Two exercises to be solved in pairs on a single computer. After solving the 1st exercise, the "driver" andr"navigator" will swap. 2 learners (1 / exercise) will display their solutions to the audience. Questions and discussion | Learners will be able to write and call a function calculating the distance between 2 points in the 3D  space and a function taking the base and height of a triangle as input and returning its area
> > 10:00 - 10:15 | Wrap-up | Group test on functions (match input and output with specific functions: fill gaps in pieces of code). Game: repetition using ball throwing | Assess learning. Do we need to work more on functions? Repeat meaning and usage of all Python objects introduced so far
> {: .matrix}
{: .details}

## Develop assessments to ensure progression

In any course, learners benefit from having opportunities to show that they're
progressing. Generally, this is done using various types of assessment,
including tests, feedback surveys, and so on. Nicholls' model includes two
different types of assessment, to evaluate:

1. **learning**, to detect changes in learner performance during instruction, to
   identify their strengths, and diagnose their weaknesses – i.e., **formative
   assessment** (or feedback). This can inform decisions about how to modify
   instruction to better promote learning; it can also inform learners about
   changes they may need to make to improve their learning;
2. **instructional outcomes**, to verify whether learners achieved the stated LOs
   after instruction – i.e., **summative assessment**. This can help to inform
   decisions both about learners (e.g., ranking their performances) and about
   the course (*i.e.*, whether any of its LOs, LEs, etc. need to be redesigned).

> <comment-title>Definitions of key terms</comment-title>
>
> **Formative assessment**: formal or informal assessments (or feedback) made
> during learning so that instruction or practice can be better targeted for
> lear


topic: contributing
tutorial: github-interface-contribution
title: Contributing with GitHub via its interface
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 7ef36a1fe946f2196badb9c08282a272
last_modified: 2026-05-15T22:28:10.825132
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/github-interface-contribution/tutorial.html

content:
All the training material which you find on [{{ site.url }}{{ site.baseurl }}/]({{ site.baseurl }}/) is stored on a [GitHub](https://github.com) repository ([{{ site.github_repository }}]({{ site.github_repository }})), a code hosting platform for version control and collaboration. GitHub interface is quite intuitive and simplifies the contributions from anyone.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to use GitHub interface to contribute to the training material:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# GitHub

The GitHub repository for the training material is: [{{ site.github_repository }}]({{ site.github_repository }}).

> <hands-on-title>Familiarization with GitHub</hands-on-title>
>
> 1. Go on the GitHub repository: [{{ site.github_repository }}]({{ site.github_repository }})
>
>    ![GitHub interface](../../images/github_interface.png "Interface of the GitHub repository of the training material")
>
> 2. Click on `CONTRIBUTORS.yaml` file
>
>    ![Click on CONTRIBUTOR](../../images/github_contributor.png)
>
> 3. View the file
>
>    You should see something like:
>
>    ![Click on CONTRIBUTOR](../../images/github_contributor_interface.png "CONTRIBUTOR file")
>
{: .hands_on}

This `CONTRIBUTORS.yaml` file is where we collect the information (name, email, etc) about the different contributors to display them on our [Hall of Fame]({% link hall-of-fame.md %}). You will add your information there. But first you need to sign in to GitHub to be able to change this file.

> <hands-on-title>Sign in to GitHub</hands-on-title>
>
> 1. Create a GitHub account (if you do not have one already)
>
>    ![GitHub signup](../../images/github_sign_up.png)
>
> 2. Sign in (once you have a GitHub account)
>
{: .hands_on}

# Edit a file

You can now modify the `CONTRIBUTORS.yaml` file to add your information there

> <hands-on-title>Edit a file</hands-on-title>
>
> 1. Open the `CONTRIBUTORS.yaml` file on GitHub
> 2. Click on {% icon hands_on %} icon (top right of the file)
>
>    ![GitHub edit](../../images/github_edit.png)
>
>    A new page will open:
>
>    ![GitHub contributor edit page](../../images/github_contributor_edit.png "CONTRIBUTOR file in edit mode")
>
> 4. Modify the `CONTRIBUTORS.yaml` by adding yourself
>
>    You should use your GitHub username and add it followed by `:` (the `:` is important) at the correct position given the alphabetical order.
>
> 5. Scroll down to the bottom of the file
> 6. Fill the **Propose file change** form
>
>    It can also be named **Commit changes** for the ones with write accesses to the repository
>
>    1. Fill the box "Update CONTRIBUTORS.yaml" with "Add < GitHub username > as contributor" (replace < GitHub username > by your GitHub username)
>
>       > <comment-title>Commit messages</comment-title>
>       > This a commit message: a description explaining why a particular change was made. Theses messages capture the history of the changes, so other contributors can understand what have been done and why
>       {: .comment}
>
>    2. Leave "Add an optional extended description..." empty
>
>    ![Propose file change form](../../images/github_propose_change_form.png)
>
> 7. Click on **Propose file change**
>
>    ![Submit propose file change form](../../images/github_propose_change_form_submit.png)
>
{: .hands_on}

Without realizing it, GitHub let you create your first branch (named here `patch-1`) and add a changement on this branch.

> <comment-title>Branching</comment-title>
> Branching is the way to work on different versions of a repository at one time. By default your repository has one branch named `main` which is considered to be the definitive branch. When you create a branch off the main branch, you're making a copy, or snapshot, of `main` as it was at that point in time.
>
> By changing a file in this branch, it will diverge from the `main` branch. It will contain data that is only on this new branch.
{: .comment}

# Open a Pull Request

Then the addition of your information in the `CONTRIBUTORS.yaml` file is currently only on your branch `patch-1`. Not on the `main` branch and so not only on the [Hall of Fame]({% link hall-of-fame.md %}). You can't add or push directly to the `main` branch, so you need to create what we call a pull request.

The GitHub interface guides you through this process: after clicking on **Propose file change**, a new page opens:

![Pull request form](../../images/github_pr_interface.png "Pull request form")

> <hands-on-title>Edit a file</hands-on-title>
>
> 1. Open and read the [CONTRIBUTING.md]({{ site.github_repository }}/blob/main/CONTRIBUTING.md) file
> 2. Come back to the pull request
> 3. Fill in the pull request description
>
>    ![Pull request description](../../images/github_pr_form.png)
>
>    1. Add a title for the Pull Request
>    2. Add a message explaining the changes you made
>
>       This message is a good way to introduce yourself and to explain the message you made. Be kind and descriptive. It helps the reviewers to understand why you did your changes and if it should be intergrated into the `main` branch (and then website).
>
>       > <comment-title>Pull request messages</comment-title>
>       > GitHub uses [Markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet), a simple Markup language, to render the Pull request messages. You can then add bold test, lists, images, etc.
>       {: .comment}
>
> 4. Scroll down and check the changes you made
>
>    - In green with `+`: what you added
>    - In red with `-`: what you deleted
>
> 5. Click on **Create pull request** or switch to **Create draft pull request** from the dropdown menu
>
>    ![Create pull request dropdown in the GitHub interface is shown, the draft pull request button option is highlighted](../../images/PR_create_menu.png)
>
>    >  <details-title>"Draft pull requests"</details-title>
>    >  [Creating a pull request as a draft](https://github.blog/2019-02-14-introducing-draft-pull-requests/#tag-your-work-in-progress) serves as an indication that you are still working on the content.
>    >  Reviewers may comment on the current state and give general feedback, but they will know that they are not looking at the final version of your contribution.
>    >
>    >  In the Galaxy Training Material repository we have also disabled the computationally most expensive automated tests on draft pull requests, and we encourage you to use the draft stage as a small contribution to sustainable computing.
>    {: .details}
{: .hands_on}

You have created your first pull request!

Your pull request will be reviewed. There are two possible outcomes:

1. Your pull request is accepted. Congratulations! Your changes will be merged into the main branch of the original repository. The website will be re-built and you will be in the [Hall of Fame]({% link hall-of-fame.md %})
2. Your pull request needs modifications: the reviewers will ask for some changes, possibly because the automatic tests are failing.

# Update a Pull Request

One of the reviewers of your pull request asked you to add your name after your GitHub username in the `CONTRIBUTORS.yaml` file.

> <hands-on-title>Update a Pull Request</hands-on-title>
>
> 1. Go to the list of pull request tab on GitHub
> 2. Click on your pull request
>
>     You can see here the comments the reviewers left you
>
> 3. Click on **Files changed** tab and see the changes you made
>
>    ![Pull request files changed tab](../../images/github_pr_file_changed.png)
>
> 4. Click on {% icon hands_on %} icon
> 5. Add your name below your GitHub username
>
>    It should look like:
>
>    ```
>    bebatut:
>         name: Bérénice Batut
>    ```
>
> 5. Navigate to the bottom of the file
> 6. Fill the **Commit changes** form, similarly to the **Propose file change** form before
> 7. Make sure the **Commit directly to the `patch-1` branch** is selected
>
>    ![Commit directly to the `patch-1` branch](../../images/github_pr_commit.png)
>
> 8. Click on **Commit changes**
>
>    ![Submit propose file change form](../../images/github_pr_commit_changes.png)
>
>    The pull request should be automatically updated
>
> 9. Check that the new changes are added to the pull request on GitHub
>
{: .hands_on}


# Close the Pull Request

Great! You now know how to make pull request on GitHub, and how to make changes after a review.
Reviewers can now approve and merge your pull request.

Because this was just a practice pull request, let's close it again.


> <hands-on-title>Close the Pull Request</hands-on-title>
>
> Once you have run through all these steps, please close the pull request again.
>
> 1. Go to the [list of pull request tab on GitHub](https://github.com/galaxyproject/training-material/pulls)
> 2. Click on your pull request
> 3. Scroll to the bottom of the page
> 3. Click on *"Close pull request"* button
>
> Whenever you add your first real contribution, you can add yourself to the `CONTRIBUTORS.yaml` file in that PR.
>
{: .hands_on}


# Conclusion


With this tutorial, you learn how to use GitHub to change a file, create a pull request and then contribute to the training material. What you have learned here can be applied to any file.

> <details-title>More about GitHub</details-title>
> Via the GitHub interface, you can also go further: create file, branch directly, etc.
> To learn that, we recommend you to read the [GitHub guide](https://guides.github.com/activities/hello-world/)
{: .details}


topic: contributing
tutorial: create-new-tutorial-tours
title: Creating Interactive Galaxy Tours
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 59e4326faf738b141c88d70a9db322c9
last_modified: 2026-05-27T09:01:47.918510
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-tours/tutorial.html

content:
Galaxy is a great solution to train the bioinformatics concepts:

- numerous bioinformatics tools are available (almost 5,000 in the ToolShed)
- it can be used by people without any computer science skills
- it trains to use technology, outlining available resources and efforts that have made them accessible to researchers
- it is scalable

In 2016, the Galaxy Training Network decide to set up a new infrastructure for delivering easily Galaxy related training material. The idea was to develop something open and online based on a community effort, as most of the time in Galaxy.

We take inspiration from [Software Carpentry](https://software-carpentry.org). We collected everything on a GitHub repository: [https://github.com/galaxyproject/training-material ](https://github.com/galaxyproject/training-material). We decided a structure based on tutorials with hands-on, fitting both for online self-training but also for workshops, grouped in topics. Each tutorial follows the same structure and comes with a technical support to be able to run.

In this tutorial, you will understand how to design and develop a new tutorial fitting in this training material repository. As doing helps to understand, we will develop a small tutorial to explain BLAST with the full infrastructure to be able to run this tutorial anywhere.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# A Galaxy Interactive Tour

A Galaxy Interactive Tour is a way to go through an entire analysis, step by step inside Galaxy in an interactive and explorative way. It is a great pedagogic way to run the tutorial directly inside Galaxy.

![Demonstration of an Interactive Tour](../../../dev/images/galaxy_tour_demo.gif "Demonstration of an Interactive Tour")

A tour is a YAML file like:

```
id: galaxy_ui
name: Galaxy UI
description: A gentle introduction to the Galaxy User Interface
title_default: "Welcome to Galaxy"

steps:
    - title: "Welcome to Galaxy"
      content: "This short tour will guide you through Galaxy's user interface.<br>
                You can navigate with your arrow keys and leave the tour at any time point
                with 'Escape' or the 'End tour' button."
      backdrop: true

    - title: "Upload your data"
      element: ".upload-button"
      intro: "Galaxy supports many ways to get in your data.<br>
              Use this button to upload your data."
      position: "right"
      postclick:
        - ".upload-button"
```

- at the top some metadata related to the Tour:
    - `id`: ID of the tour
    - `name`: name of the tour
    - `description`: a short description of the tour
    - `title_default`: a title
- several steps corresponding to the different boxes

    Each step is beginning with a dash `-` and with possible arguments

    Argument | Description
    ---  | ---
    `title`  | Header of each step-container
    `content` | Text that is shown to the user
    `element` | [JQuery Selector](https://api.jquery.com/category/selectors/) of the element you want to describe / click
    `placement` | Placement of the text box relative to the selected element
    `preclick` or `postclick` | Elements that receive a click() event before (`preclick`) or after (`postclick`) the step is shown
    `textinsert` | Text to insert if element is a text box (e.g. tool search or upload)
    `backdrop` | `true/false`:  Show a dark backdrop behind the popover and its element, highlighting the current step

    [Full reference of the properties](https://bootstraptour.com/api/)

The YAML file of a tour can be integrated in a Galaxy instance by placing the YAML file in the `config/plugins/tours` directory of the Galaxy code and restarting the Galaxy instance

# Creating a Galaxy Interactive Tour

[A Web browser plugin](https://github.com/TailorDev/galaxy-tourbuilder) is available to help the creation and the test (on the fly) of an interactive tour.

<blockquote class="imgur-embed-pub" lang="en" data-id="a/0YVvz"><a href="https://imgur.com/a/0YVvz">Galaxy Tour Builder by TailorDev</a></blockquote><script async src="https://s.imgur.com/min/embed.js" charset="utf-8"></script>

> <hands-on-title>Install and start the plugin</hands-on-title>
>
> 1. Install the plugin using the app store of your web-browser:
>     - [Chrome Web Store](https://chrome.google.com/webstore/detail/galaxy-tour-builder/mdfbapknmcpnbmggahhaegehbbbmhmgg)
>     - [Mozilla Add-ons (Firefox)](https://addons.mozilla.org/en-US/firefox/addon/galaxy-tour-builder/)
>     - [Opera add-ons](https://addons.opera.com/en/extensions/details/galaxy-tour-builder/)
>
> 2. Load the webpage of any Galaxy instance
> 3. Start the plugin by clicking on the icon with Galaxy icon close to the address bar
{: .hands_on}

We can now create easily a Galaxy Interactive Tour and test it on the fly.

> <hands-on-title>Create a Galaxy Interactive Tour</hands-on-title>
>
> 1. Create a Galaxy Interactive Tour for "BLAST" tutorial
> 2. Test it with the plugin
> 3. Copy the YAML content and add it to a file
> 2. Add the file to the `tours` directory of the tutorial
> 3. Test it on a local Galaxy instance
{: .hands_on}

# Conclusion


topic: contributing
tutorial: github-contribution
title: Contributing to the Galaxy Training Network with GitHub
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 45ae850ac715d990d18761ed3382d617
last_modified: 2026-05-27T09:01:47.921543
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/github-contribution/tutorial.html

content:
Most of the GTN content is written in [GitHub Flavored Markdown](https://guides.github.com/features/mastering-markdown/) with some metadata (or variables) found in [YAML](http://yaml.org/) files. Everything is stored on a [GitHub](https://github.com) repository: [{{ site.github_repository }}]({{ site.github_repository }}).

The process of development of new content is open and transparent, using git and following the [GitHub flow](https://guides.github.com/introduction/flow/), which we will walk you through:

![Open source development process](../../images/oss_development_process.png "Open source development process")

1. Create a fork
1. Clone your fork of this repository to create a local copy on your computer
1. Create a new branch in your local copy for each significant change
2. Commit the changes in that branch
1. Push that branch to your fork on GitHub
1. Submit a pull request from that branch to the [original repository]({{ site.github_repository }})
1. If you receive feedback, make changes in your local clone and push them to your branch on GitHub: the pull request will update automatically
1. Pull requests will be merged by the training team members after at least one other person has reviewed the Pull request and approved it.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to contribute to the GitHub repository:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Create a GitHub account, if you have not already

> <hands-on-title>Create a GitHub account</hands-on-title>
>
> 1. Go to the [GitHub website](https://github.com)
> 2. Create an account & follow the instructions. If anything is unclear, you can check [their documentation on account creation](https://docs.github.com/en/get-started/start-your-journey/creating-an-account-on-github).
{: .hands_on}

# Create a fork of this repository on GitHub

{% include _includes/cyoa-choices.html option1="Desktop" option2="Command-line" default="Desktop"
       text="You can engage with GitHub via either an application on your computer, called GitHub Desktop, or through using the Command-line. If you're not used to using the Command-line, or you just like a user interface, then use the GitHub Desktop." %}

<div class="Desktop" markdown="1">

> <hands-on-title>Download GitHub Desktop</hands-on-title>
>
> 1. Go to the [GitHub Desktop website](https://desktop.github.com/download/)
> 2. Select the **Download for (your kind of computer)**
> 3. Follow any instructions
{: .hands_on}

</div>

A fork is a copy of a repository. [Forking](https://help.github.com/articles/fork-a-repo/) a repository allows you to freely experiment with changes without affecting the original project:

![Explanation of the forking](../../images/PR_schema_01.png "Creation of a fork"){: width="900px"}

Forking a repository is a simple two-step process:

> <hands-on-title>Fork the repository</hands-on-title>
>
> 1. Go on the GitHub repository: [{{ site.github_repository }}]({{ site.github_repository }})
> 2. Click on **Fork** (top-right corner of the page)
>
>    ![Fork](../../images/PR_fork.jpg)
>
> > <comment-title></comment-title>
> > When you click the **Fork** button GitHub will show you a list with your user account and any groups where you can create a fork of this repository (starting with `@`). If you see yourself only below "You have existing forks of this repository:", it means you already have a fork and you have nothing to do.
> {: .comment}
>
{: .hands_on}

# Clone the GitHub repository on your computer

To modify the content of the repository, you need a copy of it on your computer. This step of importing a git repository is called "cloning":

![Explanation of the cloning](../../images/PR_schema_02.png "Cloning a repository"){: width="900px"}

<div class="Desktop" markdown="1">

> <hands-on-title>Clone the GitHub repository</hands-on-title>
>
> 1. Get the URL of your fork. You can find this on the main page of your fork under the green button:
>    1. Click on **Code** (right)
>
>       ![Get fork URL](../../images/clone_code_button.png)
>
>    2. Select **Open with GitHub Desktop**
>
>       Your computer might ask if it's ok to open the website Select *Allow* if so.
>     
>       ![Clone a Repository](../../images/clone_a_repo.png)
>
>    2. If you are happy with the location of the folder, select **Clone**
>       This may take some time the first time (as the GTN repository is quite large!), but eventually the folder will be in your computer.
{: .hands_on}

</div>

<div class="Command-line" markdown="1">

> <hands-on-title>Clone the GitHub repository</hands-on-title>
>
> 1. Get the URL of your fork. You can find this on the main page of your fork under the green button:
>    1. Click on **Code** (right)
>
>       ![Get fork URL](../../images/clone_code_button.png)
>
>    2. Copy the URL
>
>       It should be something like `https://github.com/< Your GitHub Username >/training-material.git`
>
> 1. Open a terminal
> 2. Navigate with `cd` to the folder in which you will clone the repository
> 3. Clone the repository with the command:
>
>    ```
>    $ git clone https://github.com/< Your GitHub Username >/training-material.git
>    ```
>
> 4. Navigate to the repository
>
>    ```
>    $ cd training-material
>    ```
{: .hands_on}

> <comment-title></comment-title>
> If you already have a local copy of the GitHub repository, you need to update it before doing any changes. To learn how to do that, please follow the [last section](#stay-up-to-date).
{: .comment}

</div>

# Create a new branch

You have now your repository locally and you want to modify it. For this example tutorial, you will add yourself as contributor of the project to appear on the [Hall of Fame]({% link hall-of-fame.md %}).

In GitHub flow, there is a concept: one new feature or change = one branch.

When you're working on a project, you're going to have a bunch of different features or ideas in progress at any given time – some of which are ready to go, and others which are not. Branching exists to help you manage this workflow. You should develop different features on different branches to help keep the changes you make simple and easy to review.

![Divergence of a branch compared to main](../../images/PR_branches_01.png "When you create a branch in your project, you're creating an environment where you can try out new ideas. Changes you make on a branch don't affect the main branch")

Here for this tutorial, you will create a branch called "my_new_branch" in which you will modify the `CONTRIBUTORS.yaml` file, the file used to generate the [Hall of Fame]({% link hall-of-fame.md %}).

<div class="Desktop" markdown="1">

> <hands-on-title>Create a branch</hands-on-title>
>
> 1. Go to *Current Branch*
>
>       ![Click on Current Branch](../../images/desktop-select_current_branch.png)
>
> 2. Either select an existing branch by searching, or (as in this case) type a name for a new branch:
>    ```
>    my_new_branch
>    ```
>       ![Select a branch option](../../images/desktop-select_a_branch.png)
>       ![Type in new branch name](../../images/desktop-new_branch.png)
>
> 3. Select **Create New Branch**
>
>       ![Create branch confirmation](../../images/desktop-create_branch.png)
>
> 4. Select **Create Branch**
{: .hands_on}


</div>

<div class="Command-line" markdown="1">

> <hands-on-title>Create a branch</hands-on-title>
>
> 1. List the existing branch
>
>    ```
>    $ git branch
>      * main
>    ```
>
>    The branch on which you are is shown with the `*`
>
> 2. Create a new branch
>
>    ```
>    $ git checkout -b my_new_branch
>    Switched to a new branch 'my_new_branch'
>    ```
>
> 3. List the existing branch to check that the branch has been created and you are now on it
{: .hands_on}

</div>

This branch is added to your local copy:

![Creation of a branch on the local copy of the repository](../../images/PR_schema_03.png "Creation of a branch on the local copy of the repository")

# Make your changes on this branch

You have created your first branch! Now you want to make the change in the `CONTRIBUTING.yaml` file. By changing a file in this branch, it will diverge from the `main` branch. It will contain data that is only on this new branch:

![Divergence of the branch compared to main](../../images/PR_branches_02.png "The changes on your branch will not be on the main branch")

<div class="Desktop" markdown="1">
> <hands-on-title>Make changes in a branch</hands-on-title>
>
> 1. Select **Open the repository in your external editor** if you have one. Otherwise, you will need to find a text editor. You can consider using [Visual Studio Code](https://code.visualstudio.com), for example. The text editors allow you to navigate through folders and type in code with automatically detected formatting, which is very helpful!
>
>       ![Open in editor](../../images/desktop-open_in_editor.png)
>
> 2. Scroll down to the `CONTRIBUTORS.yaml` file and follow the text in that file to add yourself.
>
>       ![Find the contributors file](../../images/desktop-contributors_file.png)
>
> 3. Save the file
>
> 4. Go back to your GitHub Desktop, which should now show that file change. Add a useful *commit message* and a *description* saying what you did.
>
>       ![Find the contributors file](../../images/desktop-commit.png)
>
> 5. Select **Commit to my_new_branch**
>
{: .hands_on}
</div>


<div class="Command-line" markdown="1">

> <hands-on-title>Make changes in a branch</hands-on-title>
>
> 1. Open with your favorite text editor the `CONTRIBUTORS.yaml` file that is on your computer
> 2. Add yourself in the `CONTRIBUTORS.yaml` file
>
>    You should use your GitHub username and add it followed by `:` at the correct position given the alphabetical order
>
> 2. Save the file
> 3. Check the changes you made
>
>    ```
>    $ git status
>    On branch my_new_branch
>    Changes not staged for commit:
>      (use "git add/rm <file>..." to update what will be committed)
>      (use "git checkout -- <file>..." to discard changes in working directory)
>
>        modified:   CONTRIBUTORS.yaml
>
>    no changes added to commit (use "git add" and/or "git commit -a")
>    ```
>
> 4. Add the file and commit the changes
>
>    ```
>    $ git add CONTRIBUTORS.yaml
>    $ git commit -m "Add ..."
>    ```
>
> 5. Check that there are no more changes to commit with `git status`
>
{: .hands_on}

</div>


# Push your branch on your GitHub repository

The changes you made on your branch are only on the local copy of the repository. To propagate them online, you need to push them on your fork on GitHub:

<div class="Desktop" markdown="1">
> <hands-on-title>Push the changes</hands-on-title>
>
> 1. Select **Publish branch**
>       ![Publish branch](../../images/desktop-publish_branch.png)
>
{: .hands_on}

</div>


<div class="Command-line" markdown="1">

> <hands-on-title>Push the changes</hands-on-title>
>
> 1. Push the changes to the GitHub repository
>
>    ```
>    $ git push origin my_new_branch
>    ```
>
>    > <details-title>"push origin"</details-title>
>    > When you `git push`ed, you specified `origin`. Git repositories can know that forks exist in multiple places. When you clone one, it creates a "remote" (a remote repository) which it names `origin`, set to the URL that you used when you cloned. By having multiple remotes, you can manage more complex workflows.
>    {: .details}
>
{: .hands_on}

</div>

To check this has worked, you can go to your GitHub account.

> <hands-on-title>Push the changes</hands-on-title>
> 1. Go to your GitHub repository on a web browser
> 2. Change to the "my_new_branch" branch:
>    1. Click on **Branch: main** (left)
>
>       ![Selecting branch on GitHub](../../images/PR_branch_github.png)
>
>    2. Select the branch "my_new_branch"
>
> 4. Check that your name is in the `CONTRIBUTORS.yaml` file
>
{: .hands_on}

![Pushing changes to the fork from the local copy](../../images/PR_schema_04.png "Pushing changes from the local copy to the fork on GitHub")

# Open a pull request

You pushed your changes to GitHub, but currently they are only on your fork. You want to have these changes in the main GitHub repository in order to appear on our [Hall of Fame]({% link hall-of-fame.md %}) online. You can't add or push directly the main GitHub repository, so you need to create what we call a pull request:

![Pull request](../../images/PR_schema_05.png "Pull Requests provide a way to notify project maintainers about the changes you'd like them to consider")

<div class="Desktop" markdown="1">
> <comment-title></comment-title>
> You will see that you can also make pull requests directly from GitHub Desktop. That also works great!
{: .comment}

</div>

> <hands-on-title>Create a pull request</hands-on-title>
>
> 2. Go to your GitHub repository
> 1. Click on **Compare & pull request**
>
>    !["Opening a pull request"](../../images/PR_button.png)
>
> 3. Check that the selected branch are correct: **main** on the left and your branch name on the right
>
>    ![Branches in PR](../../images/PR_branch_check.png)
>
> 2. Fill in the pull request description
>
>    ![PR description](../../images/PR_description.png)
>
>    1. Add a title for the Pull Request
>    2. Add a message explaining the changes you made (Be kind <i class="fa fa-smile-o" aria-hidden="true"></i>)
>    3. Click on **Create pull request** or switch to **Create draft pull request** from the dropdown menu
>
>       ![Create pull request dropdown in the GitHub interface is shown, the draft pull request button option is highlighted](../../images/PR_create_menu.png)
>
>       >  <details-title>"Draft pull requests"</details-title>
>       >  [Creating a pull request as a draft](https://github.blog/2019-02-14-introducing-draft-pull-requests/#tag-your-work-in-progress) serves as an indication that you are still working on the content.
>       >  Reviewers may comment on the current state and give general feedback, but they will know that they are not looking at the final version of your contribution.
>       >
>       >  In the Galaxy Training Material repository we have also disabled the computationally most expensive automated tests on draft pull requests, and we encourage you to use the draft stage as a small contribution to sustainable computing.
>       {: .details}
>
> 3. Go to **Pull requests** to check if it is there
{: .hands_on}

Once the pull is open, it will be reviewed. There are two possible outcomes:

1. Your pull request is accepted. Congratulations! Your changes will be merged into the main branch of the original repository. The website will be re-built and you will be in the [Hall of Fame]({% link hall-of-fame.md %})
2. Your pull request needs modifications: the reviewers will ask for some changes, possibly because the automatic tests are failing.

# Make any requested changes

<div class="Desktop" markdown="1">

Three things can happen:

1. A reviewer commits changes to your pull request (they fix something themselves)
2. A reviewer suggests changes to your pull request (they think something will be helpful, but want you to check and decide)
3. A reviewer posts comments, asking you to make some sort of change.

On larger pull requests, you can end up with all three! Here's how to deal this.

## 1: A reviewer commits their own changes to your pull request

These changes currently only exist in your GitHub repository online, not on your local branch. In case you have to make any further changes, it's important you *fetch* those changes onto your local branch. This is how you stay up to date.

> <hands-on-title>Fetch origin</hands-on-title>
> 1. On your GitHub Desktop, select **Fetch origin**
>       ![Fetch origin](../../images/desktop-fetch_origin.png)
>
{: .hands_on}

You can then make further changes locally, keeping your local and online branch synched.

## 2: A reviewer suggests changes

> <hands-on-title>Commit suggested changes</hands-on-title>
>
> 1. On your Pull Request link where you see the suggested changes, commit them by clicking on the "Commit suggestion" button (or engage further, or edit them) on GitHub
>
>       ![Fetch origin](../../images/desktop-commit_suggestion.png)
>
> 2. Now that your branch is updated, you need to **Fetch origin**
>
>       ![Fetch origin](../../images/desktop-fetch_origin.png)
>
{: .hands_on}

You can then make further changes locally, keeping your local and online branch synched (are you noticing a pattern?).

## 3: A reviewer posts comments to be addressed

</div>

For example, one of the reviewers of your pull request may have asked you to add your name after your GitHub username in the `CONTRIBUTORS.yaml` file, if you didn't already.

<div class="Desktop" markdown="1">

> <hands-on-title>Make further changes</hands-on-title>
>
> 1. Just like when you made your initial branch and edited the file in your text editor, go to GitHub Desktop | **Current Branch** (make sure it is the correct one!) | **Open the repository in your external editor**
>
> 2. Navigate to the `CONTRIBUTORS.yaml` file and make any suggested changes
>
> 3. Save the file
>
> 4. Commit the changes to your branch (as before)
>
> 5. Select **Push origin** to push these changes from your local folder to your online GitHub branch (which is linked to your pull request)
>
>       ![Push origin](../../images/desktop-push_origin.png)
>
>    The pull request should be automatically updated
>
> 6. Check that the new changes are added to the pull request on GitHub
>
{: .hands_on}

</div>


<div class="Command-line" markdown="1">



> <hands-on-title>Make further changes</hands-on-title>
>
> 1. Make any requested changes in the `CONTRIBUTORS.yaml` file
>
>    It should look like
>
>    ```yaml
>    bebatut:
>         name: Bérénice Batut
>    ```
>
> 2. Check the changes that you made
>
>    ```
>    $ git status
>    On branch my_new_branch
>    Changes not staged for commit:
>      (use "git add/rm <file>..." to update what will be committed)
>      (use "git checkout -- <file>..." to discard changes in working directory)
>
>        modified:   CONTRIBUTORS.yaml
>
>    no changes added to commit (use "git add" and/or "git commit -a")
>    ```
>
> 3. Add the file and commit the changes
>
>    ```
>    $ git add CONTRIBUTORS.yaml
>    $ git commit -m "Add ..."
>    ```
>
> 4. Check that there are no more changes to commit with `git status`
> 5. Push the new changes to GitHub
>
>    ```
>    $ git push origin my_new_branch
>    ```
>
>    The pull request should be automatically updated
>
> 6. Check that the new changes are added to the pull request on GitHub
>
{: .hands_on}

</div>

# Check the automatic tests

When a pull request is opened, some automated tests are automatically launched on [Travis](http://travis-ci.com/) to be sure that the changes do not break the website, the URL are valid, etc.

On the bottom of your pull request, you can see the status of the tests:

- Yellow (with circle)

    ![Running](../../images/PR_test_yellow.png)

    The tests are still running

- Red (with cross)

    ![Failed tests](../../images/PR_test_red.png)

    When it is red, you can investigate why by clicking on **Details**. You will be redirected on [Travis](http://travis-ci.com/) where you can see the logs of the tests. Get in touch with us on [Gitter]({{ site.gitter_url }}) if you need help to understand the issue.

- Green (with tick)

    ![Passed tests](../../images/PR_test_green.png)

    The tests passed. Good job!

    Even it is green, we recommend to check the result of the tests, as some of tests are allowed to fail (to avoid too much noise).




<div class="Command-line" markdown="1">

# Stay up to date

You now want to work on a new tutorial or make some other new changes. However since you get a local copy, some changes have happened to the original GitHub repository. You need then to update your local copy of the repository before changing anything.

> <hands-on-title>Update the local copy</hands-on-title>
>
> 1. Move to the `main` branch
>
>    ```
>    $ git checkout main
>    ```
>
> 2. Add a reference to the original GitHub repository
>
>    ```
>    $ git remote add upstream https://github.com/galaxyproject/training-material.git
>    ```
>
>    > <comment-title>Error "remote upstream already exists"</comment-title>
>    > If you have done step 2 before and try to `remote add` again, git will tell you that a "remote upstream already exists". In this case you can safely continue to step 4.
>    {: .comment}
>
> 4. Update the local copy of the repository by "pulling" in the content of the original GitHub repository
>
>    ```
>    $ git pull upstream main
>    ```
>
{: .hands_on}


You can now restart the GitHub flow to propose new changes: start by [creating a new branch](#create-a-new-branch).
</div>


# Finishing the pull request

Great! You now know how to make pull request on GitHub, and how to make changes after a review. Reviewers can now approve and merge your pull request, however, there's no point adding you as a contributor if you don't plan to contribute! So you have to make a choice - do you plan to contribute now, later, or never?

{% include _includes/cyoa-choices.html disambiguation="whencontribute"  option1="Contribute-now" option2="Contribute-later" option3="Contribute-never" default="Contribute-now"
       text="Your choice impacts how you will continue with your pull request." %}

<div class="Contribute-never" markdown="1">

If you do not intend to contribute to Galaxy, please close the pull request as per below.

> <hands-on-title>Close the Pull Request</hands-on-title>
>
> 1. Go to the [list of pull requests tab on the GitHub website](https://github.com/galaxyproject/training-material/pulls)
> 2. Click on your pull request
> 3. Scroll to the bottom of the page
> 3. Click on "Close pull request" button
>
{: .hands_on}

</div>

<div class="Contribute-later" markdown="1">

If you plan to contribute to Galaxy, but not anytime soon, we ask that you close your Pull Request. When you come back to contribute to Galaxy, you can add any changes to the branch that you are working on, and open a fresh pull request following the same instructions above!

> <hands-on-title>Close the Pull Request</hands-on-title>
>
> 1. Go to the [list of pull requests tab on the GitHub website](https://github.com/galaxyproject/training-material/pulls)
> 2. Click on your pull request
> 3. Scroll to the bottom of the page
> 3. Click on "Close pull request" button
>
{: .hands_on}

</div>

<div class="Contribute-now" markdown="1">

Thank you so much for being willing to contribute! The easiest way to start contributing is to look through training materials to find typos, poor figure legends or alt text - basically, a text-based issue that you can correct. If you're feeling bold, you can also look at some of these [ideas for ways to contribute]({% link faqs/gtn/contributors_getting_started.md %}).

1. Find a typo/error
2. Fix it locally, just the way you edited the `CONTRIBUTORS.yaml` file above.
3. Commit this change to your branch (the same one with yourself added to the `CONTRIBUTORS.yaml` file)
4. Push this change from your local machine to the Github site
5. Check the change is reflected in your Pull Request

The steps are the same as your initial contribution to the `CONTRIBUTORS.yaml` file. You will then, as before, engage further with reviewers to finalise your Pull Request, and eventually (if all goes well!) your Pull Request will be approved and merged!

{% icon congratulations %} Congrats, you'll have made your first contribution to the Galaxy Training Network!

</div>

# Conclusion


With this tutorial, you have learned some basics `git` commands and principles:

![Summary of the links between GitHub, fork and local repository](../../images/PR_global_schema.png "Summary of the links between GitHub, fork and local repository")

You also learned the GitHub flow and its cycle:

1. Create a new branch in your local copy
2. Commit the changes in that branch
1. Push that branch to your fork on GitHub
1. Submit a pull request from that branch to the main repository
1. Wait for feedbacks and make requested changes
1. Update your local copy
1. Restart the cycle

You can now contribute and help us to improve our tutorials!

> <details-title>Git in Depth</details-title>
> This tutorial was a quick introduction to explain the basics of contributing to the training material. We recommend that everyone follow a more detailed git tutorials:
> - [Software Carpentry tutorial](http://swcarpentry.github.io/git-novice/) which explains a bit more in detail some git commands and the concept of remote, local and stagging
> - [Learn Git Branching](https://learngitbranching.js.org/) by GitHub to test the different git commands
>
> You should also download, print and keep always with you the [Git Cheat Sheet](https://services.github.com/on-demand/downloads/github-git-cheat-sheet.pdf)
{: .details}


topic: contributing
tutorial: meta-analysis-plot
title: Single Cell Publication - Data Plotting
difficulty: advanced
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["R"]
content_hash: 42239cf6a85f0c8a5e8e2839660ac310
last_modified: 2026-05-15T22:28:10.850811
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/meta-analysis-plot/tutorial.html

content:
We'll use some ggplot and tidyverse code to plot the data we collected in part 1

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


We'll use tidyverse (which includes things like magrittr (`%>%`) and ggplot2) to load our data. Reshape2 provides the cast/melt functions which can be used to reshape specific datasets into formats that are easier to plot.

```R
library(tidyverse)
library(reshape2)
# This path will probably need to be changed depending on where you downloaded that dataset.
data = read_tsv("sc.tsv")
```

## Data Cleaning

Let's start by cleaning our data a bit. We've got a couple problems with it:

- there might be some null mergedAt dates, we'll want to remove those
- we probably want to ignore a couple of classes of datasets that were added/removed (images, and the explicit 'ignore' type.)

```R
clean = data %>% 
	select(num, class, additions, deletions, mergedAt) %>% 
	filter(!is.na(mergedAt)) %>% 
	group_by(num, class, mergedAt, month=floor_date(mergedAt, 'month'), quarter=floor_date(mergedAt, 'quarter')) %>% 
	summarise(additions=sum(additions), deletions=-sum(deletions)) %>%
	filter(class != "ignore") %>%
	filter(class != "image") %>%
	arrange(mergedAt) %>%
	as_tibble()
```

We'll setup a 'theme' for our plot that mainly consists of using the black and white theme which is quite elegant and readable, and then making some font sizes a wee bit larger:

```R
theme = theme_bw() + theme(
  axis.text=element_text(size=14), 
  plot.title=element_text(size=18), 
  axis.title=element_text(size=16),
  legend.title=element_text(size=14),
  legend.text=element_text(size=14))
```

We want all of our plots to look the same which is why we use this trick, it helps us keep a consistent aesthetic without having to re-type the configuration every single time.

Let's get to plotting!

## Plot: Lines added/removed by date/time period

We have a dataset that looks like:

```R
clean %>% head(10)
```

We'll want to plot the changes versus the time (either quarter or month), and
then maybe plot them differently based on the class of the addition/removal. So that translates into an aesthetics statement like `aes(x=time, y=additions, fill=class)`

In ggplot2 you can plot data some different ways, if you provide the datasets upfront, e.g. `data %>% ggplot()` you'll generally do something like `data %>% ggplot(aes(x=a, y=b)) + geom_something()` and `geom_something` will take the data from the ggplot call. However if you want to plot multiple series, you can also provide the data directly to the `geom_*` functions like so:

```R
ggplot() +
  geom_col(data=clean, aes(x=quarter, y=additions, fill=class)) +
  geom_col(data=clean, aes(x=quarter, y=deletions, fill=class)) + scale_fill_brewer(palette = "Paired") +
  geom_point(data=clean, aes(x=mergedAt, y=0), shape=3, alpha=0.3, color="black") +
  theme +
  xlab("Quarter") + ylab("Lines added or removed") + guides(fill=guide_legend(title="Category")) +
  ggtitle("Lines added or removed by file type across GTN Single Cell Learning Materials")
ggsave("sc-lines-by-quarter.png", width=12, height=5)

ggplot() +
  geom_col(data=clean, aes(x=month, y=additions, fill=class)) + 
  geom_col(data=clean, aes(x=month, y=deletions, fill=class)) + scale_fill_brewer(palette = "Paired") +
  theme +
  geom_point(data=clean, aes(x=mergedAt, y=0), shape=3, alpha=0.3, color="black") +
  xlab("Month") + ylab("Lines added or removed") + guides(fill=guide_legend(title="Category")) +
  ggtitle("Lines added or removed by file type across GTN Single Cell Learning Materials") 
ggsave("sc-lines-by-month.png", width=12, height=5)
```

![very inconsistent lines added by quarter some quarters have more and some have less but it is mostly workflow and then tutorial lines being added or removed.](./images/sc-lines-by-quarter.png)

In this case we plotted the data for additions and deletions separately, and we additionally added points based on the actual date of the PRs to visualise their density.


Let's produce @gtn:nomadscientist's favourite running sum plots in addition. We'll start by reshaping our data. Currently we have data that looks like:

time | variable | value
-- | --- | ---
today | measure1 | 1
today | measure2 | 10
yesterday | measure1 | 30
yesterday | measure2 | 5

And we'll re-shape this to look like this, which will make it easier to calculate changes over the course of a specific series:

time | measure1 | measure2
--- | --- | ---
today | 1 | 10
yesterday | 30 | 5


We'll use the `dcast` function to do that:

```
clean %>% select(month, class, additions)  %>% dcast(month ~ class, value.var="additions") %>% head()
```

That doesn't look quite right so, let's change how the data is aggregated:

```R
# cumulative
clean %>% select(month, class, additions)  %>% dcast(month ~ class, value.var="additions", fun.aggregate = sum)
```

Let's do it for real now:

```R
cumulative = clean %>% select(month, class, additions)  %>% 
  dcast(month ~ class, value.var="additions", fun.aggregate = sum) %>%
  mutate(across(bibliography:workflows, cumsum)) %>% 
  reshape2::melt(id.var="month")

cumulative %>% ggplot(aes(x=month, y=value, color=variable)) + geom_line() + 
  theme_bw() + theme +
  xlab("Month") + ylab("Lines added") + guides(color=guide_legend(title="Category")) +
  ggtitle("Cumulative lines added by category, across GTN Single Cell materials")
ggsave("sc-lines-cumulative.png", width=12, height=8)
```

![workflows and tutorials are the highest with on the order of 25k lines added, it used to be quite jagged with code being thrown over the wall but now it's rather smooth with consistent constant updates.](./images/sc-lines-cumulative.png)

## Contributions over time

Let's again start with some cleaning, namely removing all rows with 0 records, and removing future records (at the time of writing.)

```R
roles = read_tsv("sc-roles.tsv")
w = roles %>% 
  filter(count != 0) %>% 
  filter(!grepl("2025", date)) %>%
  filter(!grepl("2024-12-01", date))

w %>% 
  ggplot(aes(x=date, y=count, color=area)) + 
  theme +
  xlab("Date") + ylab("Unique Contributors") + guides(color=guide_legend(title="Contribution Area")) +
  ggtitle("Contributors over time to GTN Single Cell Materials") +
  geom_line()
ggsave("sc-contribs.png", width=12, height=6)
```

![plot of several lines increasing over time in a very discrete way, authorship remains highest](./images/sc-contribs.png)

## New X over Time

Let's plot all of the new single cell things added over time, all the new FAQs, Tutorials, Slides, Etc:

```R
added_by_time = read_tsv("single-cell-over-time.tsv")
added_by_time %>% dcast(date ~ `type`) %>% 
  arrange(date) %>% 
  mutate(across(event:workflow, cumsum)) %>% 
  melt(id.var="date") %>% 
  as_tibble() %>% arrange(date) %>% 
  ggplot(aes(x=date, y=value, color=variable)) + geom_line() + 
  theme_bw() + theme +
  xlab("Date") + ylab("New Single Cell Items") + guides(color=guide_legend(title="Contribution Type")) +
  ggtitle("New Single Cell events, FAQs, news, slides, tutorials, videos and workflows in the GTN")
ggsave("sc-files-cumulative.png", width=6, height=6)
```

We may also want to know how many changes there have been since the start date of this study, e.g. October 1st, 2020:

```R
since_oct = added_by_time %>% dcast(date ~ `type`) %>% 
  arrange(date) %>% 
  mutate(across(event:workflow, cumsum)) %>% 
  filter(date > as.Date("2020-10-01"))

# Pull out the first/last date as our start ane dnwend
start_date = (since_oct %>% head(n=1))$date
end_date = (since_oct %>% tail(n=1))$date

# Table of our changes.
since_oct %>% 
  filter(date == start_date | date == end_date) %>%  # Just those rows
  mutate(date = case_when(date == start_date ~ 'start', date == end_date ~ 'end')) %>% # Relabel the start/end as literal string start and end
  pivot_longer(-date) %>% pivot_wider(names_from=date, values_from=value) %>% # Transpose the data
  mutate(increase=end - start) %>% # And calculate our increase
  select(name, increase) %>% arrange(-increase)
```

## Pageviews

GTN uses the Galaxy Europe Plausible server for collecting metrics (you can change your preferences in [your GTN privacy preferences]({% link user/privacy.md %})).
We can download the data from the server and plot it, filtering by our preferred start/end dates and filters (namely that page includes `/topics/single-cell/`).
Unfortunately the data is downloaded as a zip file which we'll then need to extract data from:

```R
system("wget 'https://plausible.galaxyproject.eu/training.galaxyproject.org/export?period=custom&date=2024-11-07&from=2022-08-01&to=2024-11-07&filters=%7B%22page%22%3A%22~%2Ftopics%2Fsingle-cell%2F%22%7D&with_imported=true&interval=date' -O sc-stats.zip")
```

We can use the unzip function to read a single file directly from the zip:

```R
views = read_csv(unzip("sc-stats.zip", "visitors.csv"))
```

With that we're ready to plot. We're going to use a new feature for our plot,
`annotate`. Annotation allows you to draw arbitrary features atop your plot, in
this case we're going to draw rectangles to indicate outages and events that
might have affected our data.

```R
y = 3300
xoff = 3
views %>% group_by(date=floor_date(date, 'week')) %>% 
  summarise(date, visitors=sum(visitors), pageviews=sum(pageviews)) %>% 
  filter(visitors != 0 | pageviews != 0) %>%
  melt(id.var="date")  %>% 
  ggplot(aes(x=date, y=value, color=variable)) + geom_line() + 
  annotate("rect", xmin = as.Date("2023-10-01"), xmax = as.Date("2023-10-27"), ymin = 0, ymax = y,  alpha = .2) + # Outage
  annotate("rect", xmin = as.Date("2023-05-22") - xoff, xmax = as.Date("2023-05-26") - xoff, ymin = 0, ymax = y,  alpha = .2) + # Smorg3
  annotate("rect", xmin = as.Date("2024-10-07") - xoff, xmax = as.Date("2024-10-11") - xoff, ymin = 0, ymax = y,  alpha = .2) + # GTA
  annotate("rect", xmin = as.Date("2024-09-16") - xoff, xmax = as.Date("2024-09-20") - xoff, ymin = 0, ymax = y,  alpha = .2) + # Bootcamp
  theme_bw() + theme +
  scale_y_continuous(expand = c(0, 0), limits = c(0, NA)) +
  xlab("Date") + ylab("Count") + guides(color=guide_legend(title="Metric")) +
  ggtitle("GTN Single Cell Visits")
ggsave("sc-pageviews.png", width=14, height=4)
```

for our lovely pageview plot!

![pageviews over time, it's relatively stable except for the large annotated outage, and a large spike around the bootcamp date](./images/sc-pageviews.png)

> <tip-title>Do you want to re-do this analysis? For your community?</tip-title>
> [Discuss this with us](https://github.com/galaxyproject/training-material/issues/new?title=GTN%20Metadata%20reuse) and we can perhaps generalise this analysis, to reduce the amount of data processing you need to do, and make it more accessible for everyone!
{: .tip}


topic: contributing
tutorial: python
title: Teaching Python
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 249c7b9de061bf9e0bfe872bf8549758
last_modified: 2026-05-15T22:28:10.853071
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/python/tutorial.html

content:
Improving student learning and out comes should always be a goal of teaching. Here we present several strategies to improve student experiences during courses by focusing on how they approach specific problems, and giving them real world applicable solutions to those problems.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to run a local instance of the GTN website:
>
> 1. TOC
> {:toc}
>
{: .agenda}

"Live coding", as espoused by the Carpentries, is a fantastic strategy to communicate material to students and ensure they get a hands-on experience simultaneously.
Showing what happens live on the screen is received well by students, if
they can manage to watch what we type and try to type it themselves
simultaneously. We know at least that our examples give the correct result,
*but* students never see anything other than correct, working code, and never
have to formulate an internal model for how to write code. They end up
copying and pasting and not understanding *why*.

Predicting code behaviour without running it is a key component of work as a
programmer, and a lot of the time we spend debugging relies on us emulating
the computer in our head. Without a solid mental model of code behaviour
one cannot predict how it will function in one situation, much less other or
non-standard situations. Planning for code to handle both good and bad
inputs requires some creativity and mentally planning around expected values
at various points throughout the execution.

This situation leaves students unprepared for incorrect or buggy code,
either (un)intentionally included in homework assignments, or, generated by
themselves, if they cannot identify where code will fail without executing
it.

Augmenting lessons with:

- Pair programming
- Tracing - Stepping through the internal state
- Faded examples
- Compounding examples
- Debugging intentionally broken examples

Will give students enough tools to respond dynamically to failure
states with informed experience to resolve issues they encounter as programmers.

The student's mental model of the code underlies everything they do as a programmer, from conception to implementation to debugging to their self efficacy:

> This study shows that a well-developed and accurate mental model directly
> affects course performance and also increases self efficacy, the other key
> element in course performance. Given this double impact, helping students
> develop good mental models should remain a goal in introductory programming
> courses.
{: .quote cite="{% cite_url Ramalingam_2004 %}"}

This is a foundational skill to be able to *think* through a program, step by step, and understand how the code executes and which variables exist when, and what their values should be. This mental modelling allows students to predict the behaviour of a system, and when it diverges from their prediction, recognise any potential bugs.

## Course Management Strategies

### Pair Programming

Complementary to the other strategies, [Pair Programming](https://en.wikipedia.org/wiki/Pair_programming) or "pairing" provides a reinforcement activity where they utilise similar skills. As one person writes and executes code, the other person 'drives' the experience, telling them what to write ({% cite Williams %}, {% cite Williams_2001 %}). It has become a common learning model in introductory courses due to its benefits to students ({% cite mendes2005investigating %}, {% cite mendes2006replicated %}, {% cite Hannay_2007 %}). Specifically this technique has also been shown to be beneficial for women in computer science and gives them better chances for success in future programming endeavours ({% cite werner2004pair %}). Adopting this technique is promising, provided you adhere to principles outlined by {% cite Mentz_2008 %}.

These can often be implemented as breakout rooms wherein students are assigned a handful of problems to complete. After the breakout rooms end, you can have students summarize solutions, pick on individual ones for their ideas, etc.

## Problem Strategies

### Tracing Code Execution

> > <code-in-title>Code</code-in-title>
> > ```python
> > # Initialise our accumulator
> > x = 1 + 1
> > # Loop over our input data
> > for i in range(10): # 0..9
> >     # In-loop temporary variable
> >     tmp = x * 2 + i
> >     # Update our accumulator
> >     x = tmp + 1
> > # Output our result
> > print(f'The final value is {x}')
> > ```
> {: .code-in}
>
> > <code-out-title>Trace</code-out-title>
> >
> > Line | `i` | `x` | `tmp`
> > ---- | --- | --- | -----
> > 2    | n/a | 2   |  n/a
> > 4    | 0   | 2   |  n/a
> > 6    | 0   | 2   | 4
> > 8    | 0   | 5   | 4
> > 4    | 1   | 5   |  n/a
> > 6    | 1   | 5   | 11
> > 8    | 1   | 12  | 11
> > 4    | 2   | 12  | n/a
> > 6    | 2   | 12  | 26
> > 8    | 2   | 27  | 26
> >
> {: .code-out}
{: .code-2col}

While there is no bug in the above, when there *is* a bug present, having students produce a table like that significantly improves their understanding of code flow and execution {% cite Hertz_2013 %}. "Tracing" is a valuable and easy to complete exercise, and the results can even be checked automatically leading to good scalability of the exercise across larger classes.

Here students can also use a Debugger like [pudb](https://github.com/inducer/pudb) which can follow the execution of a bit of code, and show exactly how it's working.

<script id="asciicast" src="https://asciinema.org/a/530572.js" data-theme="monokai" async></script>

Here teaching liberal use of the `print()` command, as opposed to more complicated tools like the above, can give students the tools they need to solve problems.
This was generated by [hexylena/auto-python](https://github.com/hexylena/auto-python/) which can be reused or contributed to if new examples are needed.

### Faded Examples

When teaching programming one must constantly be cognisant of the student's cognitive load. It is a complicated task that demands a lot of students, requiring types of explicit logic analysis that they may not have engaged in before. Both learning based on problem-solving and worked examples may cause high cognitive loads for different audiences, and exploring alternatives is important ({% cite Retnowati_2017 %}). Faded examples such as what is seen below are exactly such an alternative, starting with a fully worked example and removing successive components until we reach a problem description requiring a full solution. This leads to fewer unproductive learning events ({% cite Renkl_2004 %}).

```python
### Write a function that multiplies two numbers
def multiply(a, b):
    c = a * b
    return c
```

The initial problem shows the entire solution to students

```python
### Write a function that adds two numbers
def add(___):
    ____
    return c
```

Increased fading, here we call out blanks students should fill in specifically with syntactically incorrect underscores.

```python
### Write a function that subtracts two numbers
```
Final fading, the entire problem is gone except for the description of what they need to do.

Faded examples however, do come at a higher cost of implementation than worked out examples ({% cite Zamary_2018 %}). They require writing the correct worked out example and then determining which components to remove, which presents an additional cost during course updates that if examples are changed they need to be double checked to ensure they are still valid, whereas worked examples can be checked more automatically.

### Compounding Problems

Compounding problems are a good strategy for homework problems, as you can ask multiple things of students and provide a gentle ramp up to increased complexity. Start by designing a small but complex problem like "write a fastq trimmer", where here they need to implement several different subtasks:

- file processing
- several utility functions
- multiple filter stages
- a single main function which combines all of the above

If done correctly, the students have freedom to move around individual functions that aren't dependent on each other, making sure they're correct, before building them up into a final function.

There are two alternative ways to further design the problem:

- Provide it broken down, precise small functions they should implement.
- Describe the problem and let students determine the optimal way to break it down into small, manageable components.

Which option is prefgerrable depends strongly on how advanced your students are. See the example homework [FastQ_trimmer.html](homework/FastQ_trimmer.html) and it's [associated ipynb file](homework/FastQ_trimmer.ipynb).

### Debugging

Debugging is the act of identify and resolving "bugs" or defects within code, a term popularly attributed to my personal hero Admiral Grace Hopper:

> While she was working on a Mark II computer at Harvard University, her associates discovered a moth stuck in a relay and thereby impeding operation, whereupon she remarked that they were "debugging" the system
{: .quote cite="{% cite_url enwiki:1069955193 %}"}

Debugging also functions as a reinforcement method we can use once students have an ok mental model of code execution, a necessary pre-requisite for this activity, which can be further developed through debugging ({% cite Ramalingam_2004 %}) alongside their self-efficacy ({% cite Michaeli_2019 %}). Debugging activities can take many forms but most commonly the task is to correct incorrect code, an activity that works best if they are primed with a number of methods of debugging ({% cite Murphy_2008 %}) such as the "Wolf Fence" ({% cite Gauss_1982 %}), commenting out code, or breakpoints.

```python
### Fix me!
for number in range(10):
    # use a if the number is a multiple of 3, otherwise use b
    if Number \% 3 == 0:
        message = message + a
    else:
        message = message + "b"
print(message)
```

The above debugging exercise featuring code with numerous issues from type confusion, variable typos, and failure to initialise a variable.
Students can run this example iteratively to figure out where it fails and attempt to fix it.

Use of more complex debugging tools is not always indicated, as the cognitive complexity may be too much for students. 

## Comparisons to K-12 methodologies

In K-12 teaching ({% cite Sentance_2017 %}), this intervention is used to good
results. Their model, PRIMM ({% cite Sentance_2017 %}), starts with a good
mental model which is required to predict, tracing during investigation, and
debugging to modify code, all building towards students making things
themselves.

[Here are some examples](https://primmportal.com/) of how to implement the PRIMM methodology in exercises.


topic: contributing
tutorial: create-new-topic
title: Including a new topic
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: aafe68848924f2601184f8e6a129cea8
last_modified: 2026-05-27T09:01:47.917953
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-topic/tutorial.html

content:
Each training material is related to a topic. All training materials (slides, tutorials, ...) related to a topic are found in a dedicated directory (*e.g.* `transcriptomics` directory contains the material related to exome sequencing analysis).

## Directory structure

Each topic has the following structure:

```
├── README.md
├── metadata.yaml
├── images
├── docker
│   ├── Dockerfile
├── slides
│   ├── index.html
├── tutorials
│   ├── tutorial1
│   │   ├── tutorial.md
│   │   ├── slides.html
│   │   ├── data-library.yaml
│   │   ├── workflows
│   │   │   ├── index.md
│   │   │   ├── workflow.ga
│   │   ├── tours
│   │   │   ├── tour.yaml
```


### `images` directory

The `images` directory collects all images/pictures needed for the training materials related to the topic, *i.e* pictures for the slides or the tutorials.

Images shared between several topics are in the `shared/images` directory at the root.

All images for the slides must be in `images` directory. The images must be in good quality. The sources (`svg` or other) of the images must also be added to the `images` directory. We encourage you to use [yEd](https://www.yworks.com/products/yed) to easily generate diagrams and [Inkscape](https://inkscape.org/en/) for any other images.


### `tutorials` directory

This directory collects the materials related to the topic, one material per subdirectory, though it may consist of slides, tutorials, or both.

The tutorials are hands-on built for workshop and self-training, with description of the whole infrastructure needed to run the tutorial on any Galaxy instance (tools, data library, etc).

The templates for the tutorials are different from the other pages to help users to focus on the content of the tutorial. To improve the output of the tutorial, several metadata are mandatory for every tutorials, such as the requirements or the objectives of the tutorials. Boxes are also used to highlight some key points as the hands-on or the tips.

The content of each tutorial is generated with [Jekyll](https://jekyllrb.com/) from a Markdown file and some metadata (e.g. the requirements, the Zenodo link, the questions) defined inside the metadata of the related topic.

> <comment-title>Contributing</comment-title>
> Want to contribute to a tutorial? Check out [our training materials about that]({% link topics/contributing/index.md %}).
{: .comment}

Sometimes, an hands-on tutorial is not the most appropriate format for a tutorial and slides are better.

### `docker` directory


For each topic, a flavored Docker image must integrate the tools needed for
the tutorials. The corresponding image must be based on official Galaxy Docker
images.

The `docker` image will also integrate the Galaxy tours available for each topics and the workflows.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

There are two kinds of topics we will cover: topics with their own materials, and synthetic topics that are based solely on a tag. For large collections of materials around a single central theme a topic with materials is a good choice. However, sometimes you will want to highlight materials spread across numerous or disparate topics like e.g. Cancer or SARS-CoV-2 tutorials where you might have individual tutorials on assembly or transcriptomics, but you'd like a single page listing all of them.

# Creating a new topic with it's own materials

## Defining the topic

When we structured the repository, we decided to use as topics the categories that are used in the [ToolShed](https://toolshed.g2.bx.psu.edu/). The ToolShed assigns a category to each tool. Therefore, to decide where to put your tutorial, have a look at which ToolShed's category the main tools in your tutorial belong. For example, this tutorial will rely on the NCBI Blast+ tool.

> <hands-on-title>Defining the topic for the tutorial</hands-on-title>
>
> 1. Search for NCBI Blast+ on the [ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. Check in which category it has been placed
>
>    > <question-title></question-title>
>    >
>    > In which topic will you put the new tutorial?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > If we search for [NCBI Blast+ in the ToolShed](https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/7538e2bfcd41), it is placed in 2 categories (bottom): "Next Gen Mappers", and "Sequence Analysis".
>    > > We decided to put it in "Sequence analysis" because this is the most general one for this tutorial.
>    > {: .solution}
>    {: .question}
{: .hands_on}

In this tutorial, we want to add a new topic called about "my-favorite-topic".

## Creating the skeleton for the topic

Once the topic name has been chosen, we can create it.

> <hands-on-title>Create all the required files and folders structures automatically</hands-on-title>
>
> 1. Open a terminal
> 2. Run (by adapting the information between the quotes)
>
>    ```
>    $ planemo training_init \
>             --topic_name "my-favorite-topic" \
>             --topic_title "Test" \
>             --topic_target "use" \
>             --topic_summary "Summary of the topic"
>    ```
>
> 3. Check that a new directory has been generated in `topics`
> 4. Check that a YAML file with your topic name has been generated in `metadata` folder
> 5. Make sure that Jekyll is running
>
>    > <comment-title>Jekyll</comment-title>
>    > Want to learn how to start Jekyll? [Check out our tutorial to serve the website locally]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
>    {: .comment}
>
> 6. Check if the topic has been correctly added at [http://localhost:4000/training-material/](http://localhost:4000/training-material/)
>
{: .hands_on}

## Adapt the metadata for your topic

Several metadata are defined in `metadata.yaml` file in your topic folder to :

{% assign kid_key = "Topic Schema" %}
{% assign kid_val = site.data['schema-topic'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

> <hands-on-title>Update the new topic to the website</hands-on-title>
>
> 1. Open the `metadata.yaml` file in your topic folder
> 2. Fill the correct metadata of the topic
> 3. Make sure that Jekyll is running
>
>    > <comment-title>Jekyll</comment-title>
>    > Want to learn how to start Jekyll? [Check out our tutorial to serve the website locally]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
>    {: .comment}
>
> 4. Check how it changes the local website
>
{: .hands_on}

### Subtopics

For large topics, we can define subtopics and create multiple tutorial lists, within a topic's `metadata.yaml`

```yaml
name: transcriptomics
type: use
title: Transcriptomics
summary: Training material for all kinds of transcriptomics analysis.
docker_image: "quay.io/galaxy/transcriptomics-training"

[...]

subtopics:
- id: introduction
  title: "Introduction"
  description: "Start here if you are new to RNA-Seq analysis in Galaxy"
- id: end-to-end
  title: "End-to-End Analysis"
  description: "These tutorials take you from raw sequencing reads to pathway analysis"
- id: visualisation
  title: "Visualisation"
  description: "Tutorials covering data visualisation"
  crosstopic_tutorials:  ## optional
    - topic: microbiome
      tutorial: beer-data-analysis
      priority: 2  ## optional
```

Each subtopic has:
- an ID, used as a reference in the tutorial
- a short descriptive title
- a longer description discussion what is contained in that subtopic

Each subtopic **optionally** may define:
- tutorials from other topics to show, via the `crosstopic_tutorials` metadata

Tutorials can be assigned to subtopics by adding e.g. `subtopic: singlecell` to the tutorial metadata. An example of this subtopic division can be found in the [admin section]({{site.baseurl}}/topics/admin/ )


# Creating a tag based topic

For tag based topics, first ensure that all of the relevant tutorials share a single tag, across all materials that should be included in this view.

## Defining the topic

Compare with other topic level metadata files in the `metadata/` directory of the training material. Then create a file of your own naming the topic and providing editorial board members and so on. We will use the Covid-19 synthetic topic as an example:

```yaml
---
name: "covid19"
type: "use"
title: "SARS-CoV-2"
summary: "Tutorials covering analysis of SARS-CoV-2 (COVID 19)"
tag_based: true

editorial_board:
  - wm75

gitter: galaxyproject/sars-cov-2
```

As you can see it is very short, and there are only a handful of important points:

- `tag_based` must be set to `true`
- `name` must be equal to the tag name, that's being used across tutorials.
- `subtopics` must not be set.

With this done, all materials tagged `covid19` will be aggregated and available under this synthetic topic. They are organised by the "parent" topic, so e.g. assembly tutorials are collected together and transcriptomics tutorials are also in a section together, similar to how other tutorials define subtopics.

## More manual tag based topics

If you wish to more manually curate the tutorials listed in your tag-based topic, you can provide subtopics. These must have IDs that match tags that are expected to be found within your tag-based topic.

```yaml
---
name: gmod
type: use
topic_type: technology
title: GMOD
edam_ontology: ["topic_0362"]

tag_based: true

subtopics:
  - id: jbrowse1
    title: "JBrowse 1"
    description: Learn to use JBrowse v1 here.
  - id: apollo2
    title: Apollo v2
    description: Apollo is a collaborative, real time genome annotation editor.
```

Here anything with a tag `jbrowse1` will appear under the JBrowse section, same for `apollo2` and the Apollo section. Tutorials may appear multiple times, but this is still useful as it gives you a way to provide additional organisation within your topic.


# Conclusion

We just created a new topic. We can now fill it by [creating new tutorials]({% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %})


topic: contributing
tutorial: create-new-tutorial-quiz
title: Adding Quizzes to your Tutorial
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 001f8a6acdd0d536bafae961602dd304
last_modified: 2026-05-15T22:28:10.808221
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-quiz/tutorial.html

content:
Interactive quizzes can be used either alone, or with a classroom of students, to check student's knowledge.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## How it Works

We developed a small Kahoot-like interface, where a teacher can initiate a quiz, students can join the teacher's session (using PeerJS), and do a collaborative quiz.

## Quiz Format

Each quiz starts a lot like the tutorials!

```yaml
title: SQL Advanced Recap
contributors:
- hexylena
```

Within the questions section we have a list of question

```yaml
questions:
- title: How do you find the number of records in a query result?
  answers:
    - count(name)
    - count(*)
    - sum(id)
    - max(id)
  correct:
    - count(name)
    - count(*)
  timeout: 20
  type: choose-many
```

There are a few different types of questions:

- `choose-1`, one correct answer
- `choose-many`, potentially multiple correct answers
- `poll`, there's no right answer, just ask the students how they're feeling!


You can include images in the main area, if you need some context for a question

```yaml
- title: Which of these joins is NOT valid
  image: /training-material/topics/data-science/images/carpentries-sql/sql-join-structure.svg
  answers:
    - Select * From P as P1 Join P as P2 on P1.id = P2.id
    - SELECT * From P Join Q Join V on P.id = Q.person and Q.taken = V.id
    - SELECT * From S join Q on S.name=V.dated and V.site = Q.quant
    - SELECT * From S Join V on S.name = V.site
  correct: SELECT * From S join Q on S.name=V.dated and V.site = Q.quant
  timeout: 60
  type: choose-1
```

Poll's let you check in with students, how they're feeling, or take their opinion on what they think.

```yaml
- title: How are you feeling?
  answers:
    - Great
    - Horrid
  timeout: 20
  type: poll
  live: true
```

The `live` key there allows polls to show the results "live" and let students change answers while the time runs.

## Folder Structure

You place quizzes in a `quiz` subdirectory of your tutorial

```console
.
├── quiz
│   └── a.yaml
└── tutorial.md
```

## Inserting quizzes throughout a tutorial

If you want to show a quiz at a specific place within a tutorial

```markdown
{% raw %}
{% include _includes/quiz.html id="a.yaml" %}
{% endraw %}
```

{% include _includes/quiz.html id="a.yaml" %}

And then it's available!

## Default Location

By default all quizzes are found at the bottom of the tutorial.


topic: contributing
tutorial: create-new-tutorial
title: Creating a new tutorial
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 060bfe2908e33ae383ed099b8a77854e
last_modified: 2026-05-27T09:01:47.918637
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial/tutorial.html

content:
Galaxy is a great solution to train bioinformatics concepts:

- numerous bioinformatics tools are available (over 8,000 in the ToolShed)
- it can be used by people without any computer science skills
- it trains to use technology, outlining available resources and efforts that have made them accessible to researchers
- it is scalable

In 2016, the Galaxy Training Network decided to set up a new infrastructure for easily delivering Galaxy-related training material. The idea was to develop something open, online, based on a community effort, and on top of the Galaxy platform.

We took inspiration from [Software Carpentry](https://software-carpentry.org) and collected everything on a GitHub repository: [https://github.com/galaxyproject/training-material](https://github.com/galaxyproject/training-material).
We decided on a structure focusing on tutorials with hands-on activities; fitting both for online self-training but also for workshops. Each tutorial follows the same structure and comes with a virtualised instance, allowing you to run the training anywhere you have resources available.

Here you will learn how to create a new tutorial by developing a small tutorial explaining how to retrieve climate data from Copernicus (using **Copernicus Climate Data Store** tool).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title></comment-title>
> This tutorial explains the different steps to create a tutorial for the Galaxy Training Material.
> It may require some knowledge that you may not have or do not have the time to learn. If this is the case, you can create a skeleton of a tutorial with whatever existing materials you have, using your prefered text editor, and then share it with us by opening [issue on GitHub]({{ site.github_repository }}/issues/new), writing us on [Gitter]({{ site.gitter_url }}), or sending us an [email](mailto:{{ site.email }}).
{: .comment}

# Define the topic

The first step we need to do is to identify in which topic to place our new tutorial. This can be tricky. When we structured the GTN, we decided that each training material should be related to a topic.

We decided to use the categories from the [ToolShed](https://toolshed.g2.bx.psu.edu/) as our initial list of topics. Since every tool uploaded to the ToolShed must be in at least one category, you can look at the main tools in your tutorial and see which categories they are placed in within the ToolShed. This can provide a guide for where you might put your new tutorial. For example, if your tutorial will rely on the **Copernicus** tool:

> <hands-on-title>Defining the topic for the tutorial</hands-on-title>
>
> 1. Search for **Copernicus** on the [ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. Check in which category it has been placed
>
> > <solution-title></solution-title>
> > There are a couple of steps to reaching the answer:
> > 1. Search for `Copernicus`
> > 2. Press the <kbd>Enter</kbd> key to search
> > 3. Click on the result named `c3s`
> > 4. At the bottom of this page, there is a box labelled "Categories"
> >
> > It is placed in one category: "Climate Analysis"
> {: .solution}
>
> 3. Look if the categories fit to existing topics in the Galaxy Training Material
{: .hands_on}

> <comment-title>No fitting topic for the tools in your tutorial?</comment-title>
> If the categories in ToolShed do not fit to any existing topics, we recommend to use your better judgment to identify in which topic your tutorial should go. You can also ask us on [Gitter]({{ site.gitter_url }}) or raise an issue on [GitHub]({{ site.github_repository }}) explaining the aim of the tutorial. We will be happy to help you there.
{: .comment}

> <comment-title>Creating a new topic</comment-title>
> Want to create a new topic? [Check out our tutorial to create a new topic]({% link topics/contributing/tutorials/create-new-topic/tutorial.md %})
{: .comment}

# Store a tutorial

All training materials (slides, tutorials, ...) related to a topic are found in a dedicated directory (*e.g.* `climate` directory contains the material related to Climate analyses). Each topic has the following structure:

```
├── README.md
├── metadata.yaml
├── images
├── docker
│   ├── Dockerfile
├── slides
│   ├── index.html
├── tutorials
│   ├── tutorial1
│   │   ├── tutorial.md
│   │   ├── slides.html
│   │   ├── data-library.yaml
│   │   ├── workflows
│   │   │   ├── workflow.ga
│   │   ├── tours
│   │   │   ├── tour.yaml
```

Once the topic has been chosen, we can create the tutorial. An ideal tutorial in the Galaxy Training Network contains:
- a tutorial file `tutorial.md` written in Markdown with hands-on
- an optional slides file `slides.html` in Markdown with slides to support the tutorial
- a directory `workflows` with workflows extracted from the tutorial
- a YAML file `data-library.yaml`  with the links to the input data needed for the tutorial

The most important file is the `tutorial.md`, where the content of the tutorial is. The other files are not mandatory. They are there to support the tutorial and make it robust and usable across many environments. But they can help a lot in the development of a new tutorial.
The `data-library.yaml` is used by Galaxy servers to create a *Shared Data Library* containing your tutorial data, allowing you quicker access during training courses. If you have defined a `zenodo_link` in your tutorial metadata, then the `data-library.yaml` file will be automatically added by one of the GTN GitHub Actions if it is missing, so there is no need to add it yourself. 

# Create the workflow

Our tutorials try to follow the "learn by doing" approach; they combine both theoretical and practical sections done on Galaxy.

Most tutorials explain step-by-step a data analysis by running the tools. The steps taken in the tutorial can be represented as a workflow.

Before writing the tutorial, it is a good practice to get a workflow with the different steps that will be run during the tutorial. The workflow does not have to be the final one, but at least the major steps should be included. It helps to get a direction for the tutorial, but also to generate a skeleton of the tutorial, as we will see later.

> <comment-title>Use tools that are available on the ToolShed</comment-title>
> We recommend using tools in your workflows, especially for training, that are available on the Galaxy ToolShed.
{: .comment}


> <hands-on-title>Prepare the workflow for the training</hands-on-title>
>
> 1. Go to your favorite Galaxy server
> 2. Create a workflow with the different steps (tools) of your tutorial either from scratch or from an existing history
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
>    {% snippet faqs/galaxy/workflows_extract_from_history.md %}
>
>    For the Copernicus tutorial:
>    1. Add an **Input Dataset**
>    2. Add **Copernicus Climate Data Store**
>    3. Link the **Input Dataset** to **Copernicus Climate Data Store** on `API Request filename`
>    4. Rename **Input Dataset** to `API Request file`
>    5. Save the workflow
>
> 3. Add the topic name as a Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor
>
>    {% snippet faqs/galaxy/workflows_annotate.md %}
>
>    For the Copernicus tutorial:
>    - **Annotation**: `Retrieve climate data from Copernicus`
>    - **Tag**: `climate`
>
> 4. Make the workflow accessible (publishing is not necessary)
>
>    {% snippet faqs/galaxy/workflows_publish.md %}
>
{: .hands_on}


# Get a toy dataset

To run the different steps, the tutorial needs some data. The selected data must be informative enough to illustrate the meaning of using a tool or a given technique, but not too big to require long waiting times for processing during a workshop. Upload and download of files into and out of Galaxy is usually quick, but the time taken for a tool to run can be long. Tool run times of no more than 10-15 mins are recommended. Typically, the selected data should be the informative subset of a full real-life dataset.

Below we describe two examples of how toy datasets were generated for tutorials:

- **Example 1**: creating a toy dataset from scratch
  - Take one 16S sequence (for example found in the test case of a Galaxy tool):
  - Generate a reference database
      - Blast it on the NR database on [NCBI Blast](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastn&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome)
      - Extracting one similar sequence found with Blast
      - Search and extract 2 other sequences of the same species using the [NCBI Nucleotide database](https://www.ncbi.nlm.nih.gov/nuccore)

- **Example 2**: creating a toy dataset from an existing larger one
  - When the experiment takes a FASTQ as input and a few reads are sufficient:
    - Use **seqtk_sample** {% icon tool %} to extract randomly reads from your input fastq.
  - However, when it requires a lot of reads to be meaningful, you can use the following strategy (used for the ATAC-seq tutorial using [this workflow](./workflows/Galaxy-Workflow-MakeAFakeInput.ga)):
    - Run the workflow until the mapping step on the full dataset (or big enough to have good results).
    - Select IDs of reads which map on the smallest chromosome (for example chr22 for human data).
    - In order to keep in the toy dataset enough diversity, you can also take randomly 1% of the reads IDs.
    - Concatenate the two lists and remove the duplicated IDs.
    - Use **seqtk_subseq** {% icon tool %} to sample your original FASTQ with the list of IDs.

We would then develop the tutorial and test it on this toy dataset. Once we were ready to share it, we would upload the datasets to [Zenodo](https://zenodo.org/) to store them on long-term and obtain a dedicated DOI in the [Galaxy training network community](https://zenodo.org/communities/galaxy-training/?page=1&size=20).

> <hands-on-title>Upload the dataset to Zenodo</hands-on-title>
>
> 1. Go to [Zenodo](https://zenodo.org/)
> 2. Log in using your GitHub credentials
>
>    You may need to authorize Zenodo to access your GitHub account (only to read your information)
>
> 3. Click on **Upload** (top panel)
> 4. Start a new upload
> 5. Upload the files corresponding to your datasets
>
>     > <comment-title>No possible changes in the files after publication</comment-title>
>     > File addition, removal or modification is not allowed after you have published your upload.
>     > So be careful when you start your upload that all the files you need are ready.
>     >
>     > The metadata can be changed after publication.
>     {: .comment}
>
> 6. Search for and Select *Galaxy Training Network* in **Communities**
> 7. Select *Dataset* in **Upload type**
> 8. Use the title of your tutorial and mention also Galaxy Training Material
> 9. Add all the persons who contributed to the tutorial as authors
> 10. Add a short description of the tutorial and a link to the training material website
> 11. Keep *Open Access* as **Access right** and *Creative Commons Attribution 4.0* as **License**
> 12. Fill out any remaining information
> 13. Click on **Save**
> 14. Click on **Publish**
> 15. Copy the DOI link in the new page
{: .hands_on}

# Create the skeleton of the tutorial

Once we have the workflows and toy dataset (optionally already on Zenodo), we can generate the tutorial folder, including the `tutorial.md`.

Writing the tutorial while adding the different tools and their parameters, and following good formatting, can be quite cumbersome. To help in that process, we developed a training development kit available with [Planemo](https://planemo.readthedocs.io/en/latest/index.html). The tool has several commands. But the main one can be used to create the skeleton of a tutorial (folder, well-located, with good structure). It can also take a workflow as input and add it to the `workflows` folder. But more interestingly, the tool can extract from the workflow and add in the `tutorial.md` file the different steps with which tools to run and which parameters to select. If a Zenodo URL is provided, it also creates the `data-library.yaml` file.

This tool can be used via the command-line but also via a [webserver](https://ptdk.apps.galaxyproject.eu/). The webserver can take a public workflow that is available on [usegalaxy.org](https://usegalaxy.org), [usegalaxy.eu](https://usegalaxy.eu) or [usegalaxy.org.au](https://usegalaxy.org.au).

> <hands-on-title>Create the skeleton of a tutorial using the webserver</hands-on-title>
>
> 1. Make the workflow public
>
>    {% snippet faqs/galaxy/workflows_publish.md %}
>
> 2. Copy the workflow ID that can be found in the URL of the current page (after `?id=`)
> 3. Open the [PTDK webserver](https://ptdk.apps.galaxyproject.eu/)
> 4. Fill in the information
>    - Tutorial name (the name will be the name of the folder of the tutorial)
>    - Tutorial title
>    - Galaxy instance with the public workflow
>    - Id of the workflow
>    - (Not mandatory) Zenodo URL with the input data
>
>    For the Copernicus tutorial:
>    - **Tutorial name** (the name will be the name of the folder of the tutorial): `climate-data-retrieval-copernicus`
>    - **Tutorial title**: `Retrieve climate data from Copernicus`
>    - **Galaxy instance with the public workflow**: `usegalaxy.eu`
>    - **Id of the workflow**: `ac5b66c42681e7a8`
>
> 5. Click on **Submit**
> 6. Download the generated archive
>
>    This archive contains the tutorial skeleton, including:
>    - tutorial content, `tutorial.md` file, filled with all steps from the workflow and their parameters
>    - its workflow (`workflow` folder)
>    - a `data_library.yaml` file if Zenodo link was provided
>
> 7. Add the new material to Galaxy Training Material by unzipping the downloaded archive in the tutorials folder of the topic for the new tutorial
>
>    > <comment-title>Using the GitHub interface</comment-title>
>    >
>    > Prefer to use the GitHub interface?
>    >
>    > 1. Unzip the downloaded archive
>    > 2. Edit the content of the `tutorial.md` (as explained below)
>    > 3. Go to the GitHub repository of the Training Material
>    > 4. Fork the GitHub repository
>    > 5. Click on `topics`
>    > 6. Select the topic for the new tutorial
>    > 7. Go to `tutorials`
>    > 8. Click on **Create new file**
>    > 9. Type `name/tutorial.md`, replacing "name" by the name of your tutorial (not the title)
>    > 10. Copy the content of the downloaded and edited `tutorial.md` file there
>    > 11. Fill the **Commit new file** form
>    > 12. Create a new branch using the name of the tutorial
>    > 13. Click on **Propose new file**
>    > 14. Open a Pull Request, as explained in our tutorial
>    > 15. Add the workflow file (in `workflow` folder) by updating the Pull Request
>    {: .comment}
>
{: .hands_on}

If the workflow is not available on one of the previously listed Galaxy servers, we recommend that you run the tool via the command line:

> <hands-on-title>Create the skeleton of a tutorial via the command line</hands-on-title>
>
> 1. Get the workflow ID
>
>    The ID can be found on the URL when running, editing or sharing the workflow (after `?id=`)
>
> 2. Get your API key on the Galaxy instance
>
>    {% snippet faqs/galaxy/preferences_admin_api_key.md %}
>
> 3. (If not done yet) Get the Galaxy Training Material repository locally and move into it
>
>     1. (If not done yet) Clone the training material GitHub repository: `git clone https://github.com/galaxyproject/training-material.git`
>     2. Navigate to the `training-material/` folder with `cd`
>
> 4. Get planemo
>
>     - Option 1: Using Conda:
>       1. Set up the conda environment
>
>          It will install some needed tools (ruby, nodejs, etc) in a protected environment, without interfering with the existing tools or versions.
>
>          1. Install conda (if not already installed): `make install-conda`
>          2. (You may need to exit the terminal and re-open for conda to be recognised. Navigate back to the same place.)
>          3. Create the `galaxy_training_material` conda environment: `make create-env`
>
>       2. Activate the conda environment with `conda activate galaxy_training_material`
>
>     -  Option 2: Using pip by running `pip install planemo`
>
> 6. Generate the skeleton of your tutorial (by adapting the information between the quotes)
>
>      ```
>      $ planemo training_init \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --tutorial_title "Title of the tutorial" \
>             --galaxy_url "URL to Galaxy instance in which you created the workflow" \
>             --galaxy_api_key "Your API key on the Galaxy instance" \
>             --workflow_id "ID of the workflow on the Galaxy instance" \
>             --zenodo_link "URL to the Zenodo record"
>      ```
>
> 4. Check that a new directory (with your tutorial name) has been generated in the topic folder
{: .hands_on}

# Write the tutorial

Now that you have the structure in place, you can then fill the tutorial, per se.

> <hands-on-title>Write the tutorial</hands-on-title>
>
> 1. Open the `tutorial.md` file with your favorite text editor
> 2. Fill out the tutorial by following the [dedicated tutorial]({% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md %})
>
>    1. Add metadata at the top of the tutorial
>    2. Add a proper introduction
>    3. Organize the tutorial into different sections
>    4. Introduce the different concepts and the different steps of the analysis
>    5. Check the hands-on boxes
>    6. Comment on the outputs of the different steps
>    7. Add some question/solution boxes for assessment
>    8. Add a conclusion
>
> 3. (Optional) Build the website locally and check that the tutorial is there by following the [Jekyll tutorial]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
{: .hands_on}

# Keep track of the changes

The material is stored in a [GitHub repository]({{ site.github_repository }}), a code hosting platform for version control and collaboration. So to develop training material, we are following the [GitHub flow](https://guides.github.com/introduction/flow/), which is based on fork, branches, and pull requests.

This can be done online via the GitHub interface or locally on your computer via the command line.

> <comment-title>Learning how to contribute</comment-title>
> Want to learn how to contribute? Check our tutorials:
> - [Contributing with GitHub via its interface]({% link topics/contributing/tutorials/github-interface-contribution/tutorial.md %})
> - [Contributing with GitHub via command-line]({% link topics/contributing/tutorials/github-command-line-contribution/tutorial.md %})
{: .comment}

# Add slides (optional)

Sometimes, you may want to have slides to support a tutorial and introduce it during a workshop. Sometimes, a set of slides is better than a tutorial to cover a specific topic.

> <hands-on-title>Add slides</hands-on-title>
>
> 1. Create a slide deck in `slides.html` following the [Slide tutorial]({% link topics/contributing/tutorials/create-new-tutorial-slides/slides.html %})
{: .hands_on}

# Conclusion


To develop a new tutorial:

1. Determine the topic
2. Create a workflow
3. Find a good toy dataset and upload it to Zenodo
4. Create the skeleton for the tutorial
5. Add the skeleton to the training material
6. Write the tutorial
7. Keep track of the changes
8. Add slides (optional)
9. Submit as a Pull Request to GitHub

For the next time, you can make it quicker.

> <hands-on-title>Generation of a tutorial</hands-on-title>
>
> 1. Determine the topic
> 2. Create your workflow on a running Galaxy instance
> 3. Add the topic name as a Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor
> 4. Create a Zenodo record with the input data
> 5. Generate the skeleton of your tutorial
>
>    - Option 1: from the [PTDK webserver](https://ptdk.apps.galaxyproject.eu/) and get the skeleton in the training material
>    - Option 2: from a workflow located on a Galaxy
>
>      ```
>      $ planemo training_init \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --tutorial_title "Title of the tutorial" \
>             --galaxy_url "URL to Galaxy instance in which you created the workflow" \
>             --galaxy_api_key "Your API key on the Galaxy instance" \
>             --workflow_id "ID of the workflow on the Galaxy instance" \
>             --zenodo_link "URL to the Zenodo record"
>      ```
>
> 6. Fill the remaining metadata in the `tutorial.md`
> 7. Fill the content of the `tutorial.md`
> 8. Check it by serving the website locally
>
>    > <comment-title>Serving the website locally</comment-title>
>    > Want to learn how to see the change on the website locally? [Check out our dedicated tutorial]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
>    {: .comment}
>
> 9. Submit it to GitHub
{: .hands_on}


topic: contributing
tutorial: generating-pdf
title: Generating PDF artefacts of the website
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: []
tags_json: []
content_hash: b15be4a691bd45c9d26ca62ec45bec8e
last_modified: 2026-05-15T22:28:10.822979
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/generating-pdf/tutorial.html

content:
The website with the training material can be run locally. Sometimes, it is also interesting to freeze the tutorials or to get PDFs of the tutorials.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to run a local instance of the GTN website:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Generate PDFs artifact

To generate the PDFs, a command `make pdf` is given. This command:

- Launches a detached Jekyll server to serve the website
- Generates the PDFs of the tutorials by calling Chrome via command line
- Generates the PDFs of the slide decks by calling decktape

> <hands-on-title>Checking the website generation locally</hands-on-title>
>
> 1. Install a browser
> 2. Generate the PDFs: `make pdf`
> 3. Check the generated PDFs in `_pdf` folder
{: .hands_on}


topic: contributing
tutorial: slides-with-video
title: Adding auto-generated video to your slides
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 4885eb013706d585a4450553dc371e45
last_modified: 2026-05-15T22:28:10.862530
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/slides-with-video/tutorial.html

content:
# Video Lectures


Based on the work by Delphine Larivière and James Taylor with their [COVID-19 Lectures](https://github.com/galaxyproject/video-lectures/) we have implemented a similar feature in the Galaxy Training Network.

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## How it Works

We wrote a short script which does the following:

*Locally and in production*:

- Extracts a 'script' from the slides. We extract every presenter comment in the slidedeck, and turn this into a text file.
- Every line of this text file is then narrated by [Amazon Polly](https://aws.amazon.com/polly/) (if you have money) or [MozillaTTS](https://github.com/synesthesiam/docker-mozillatts) (free).
- The slide deck is converted to a PDF, and then each slide is extracted as a PNG.
- Captions are extracted from the audio components.
- The narration is stitched together into an mp3
- The images are stitched together into an mp4 file
- The video, audio, and captions are muxed together into a final mp4 file

*In production*

- We use Amazon Polly, paid for by the Galaxyproject
- The result is uploaded to an S3 bucket

# Enabling Video

We have attempted to simplify this process as much as possible, but making good slides which work well is up to you.

## Writing Good Captions

Every slide must have some narration in the presenter notes. It does not make sense for students to see a slide without commentary. For each slide, you'll need to write presenter notes in full, but short sentences.

### Sentence Structure

Use simple and uncomplex sentences whenever possible. Break up ideas into easy to digest bits. Students will be listening to this spoken and possibly reading the captions.

*2021-05-01* There used to be a limit of ~120 characters per sentence, but this is no longer an issue. We now break up sentences which are too long in the captions and show them over multiple timepoints. So if you need to write a really long sentence, you can, but we still advise to simplify sentences where possible.

### Captions per Slide

Every slide must have some speaker notes in this system, **NO exceptions**.

### Punctuation

Sentences should end with punctuation like `.` or `?` or even `!` if you're feeling excited.

### Abbreviations

These are generally fine as-is. (e.g. `e.g.`/`i.e.` is fine as-is, `RNA` is fine, etc.) Make sure abbreviations are all caps though.

> **Good**
> This role deploys CVMFS.
{: .code-in}

### "Weird" Names

In the captions you will want to teach the GTN how to pronounce these words by editing `bin/ari-map.yml` to provide your definition.

E.g.

Word       | Pronunciation
---------- | ---
SQLAlchemy | SQL alchemy
FastQC     | fast QC
nginx      | engine X
gxadmin    | GX admin
/etc       | / E T C

The same applies to the many terms we read differently from how they are written, e.g. 'src' vs 'source'. Most of us would pronounce it like the latter, even though it isn't spelt that way. Our speaking robot doesn't know what we mean, so we need to spell it out properly.

So we write the definition in the [`bin/ari-map.yml`](https://github.com/galaxyproject/training-material/blob/master/bin/ari-map.yml) file.

### Other Considerations

(*Written 2020-12-16, things may have changed since.*)

Be sure to check the pronunciation of the slides. There are known issues with [heteronyms](https://en.wikipedia.org/wiki/Heteronym_(linguistics)), words spelt the same but having different pronunciation and meaning. Consider "read" for a classic example, or ["analyses"](https://en.wiktionary.org/wiki/analyses#English) for one that comes up often in the GTN. "She analyses data" and "Multiple analyses" are pronounced quite differently based on their usage in sentences. See the [wiktionary](https://en.wiktionary.org/wiki/analyses#English) page for more information, or the [list of English heteronyms](https://en.wiktionary.org/wiki/Category:English_heteronyms) you might want to be aware of.

This becomes an issue for AWS Polly and Mozilla's TTS which both don't have sufficient context sometimes to choose between the two pronunciations. You'll find that "many analyses" is pronounced correctly while "multiple analyses" isn't.

Oftentimes the services don't understand part of speech, so by adding adjectives to analyses, you confuse the engine in to thinking it should be the third person singular pronunciation. This is probably because it only has one or two words of context ahead of the word to be pronounced.

## Enable the Video

Lastly, we need to tell the GTN framework we would like videos to be generated.

> <hands-on-title>Enable video</hands-on-title>
>
> 1. Edit the `slides.html` for your tutorial
> 2. Add `video: true` to the top
{: .hands_on}

That's it! With this, videos can be automatically generated.


# Voices

There are multiple voices available, see the following list:

Name    | Region    | Neural | Audio Clip
---     | ------    | ----   | ----
Amy     | en-GB     | True   | <audio controls src="{% link assets/audio/amy.mp3 %}"></audio>
Aria    | en-NZ     | True   | <audio controls src="{% link assets/audio/aria.mp3 %}"></audio>
Brian   | en-GB     | True   | <audio controls src="{% link assets/audio/brian.mp3 %}"></audio>
Emma    | en-GB     | True   | <audio controls src="{% link assets/audio/emma.mp3 %}"></audio>
Joanna  | en-US     | True   | <audio controls src="{% link assets/audio/joanna.mp3 %}"></audio>
Joey    | en-US     | True   | <audio controls src="{% link assets/audio/joey.mp3 %}"></audio>
Kendra  | en-US     | True   | <audio controls src="{% link assets/audio/kendra.mp3 %}"></audio>
Matthew | en-US     | True   | <audio controls src="{% link assets/audio/matthew.mp3 %}"></audio>
Nicole  | en-AU     | False  | <audio controls src="{% link assets/audio/nicole.mp3 %}"></audio>
Olivia  | en-AU     | True   | <audio controls src="{% link assets/audio/olivia.mp3 %}"></audio>
Raveena | en-IN     | False  | <audio controls src="{% link assets/audio/raveena.mp3 %}"></audio>
Salli   | en-US     | True   | <audio controls src="{% link assets/audio/salli.mp3 %}"></audio>
Ayanda  | en-ZA     | True   | <audio controls src="{% link assets/audio/ayanda.mp3 %}"></audio>
Geraint | en-GB-WLS | True   | <audio controls src="{% link assets/audio/geraint.mp3 %}"></audio>

By default a random voice is chosen every time the video is rebuilt (only
whenever a change is made to that slide deck.) We do this to ensure a good
diversity of genders and nationalities in the audio samples.

However, if you have a preferred voice, you can set that permanently for that
video, add the following metadata to the top of your slide deck:

```
voice:
  id: Lupe
  lang: es-US
  neural: true
```

The above voice example is specific to Spanish language content, hence not being represented in the first list.

# How it works: In Detail

1. We take our markdown slides, e.g. [`topics/introduction/tutorials/galaxy-intro-short/slides.html`](https://github.com/galaxyproject/training-material/blob/main/topics/introduction/tutorials/galaxy-intro-short/slides.html)
2. In order for them to be processed, slides must have an annotation saying `video: true` in the header metadata, and then 'speaker notes' (everything after the ??? before the ---)
3. This is turned into our 'plain text slides' which just renders the markdown a bit more nicely ([example]({{site.baseurl}}/topics/introduction/tutorials/galaxy-intro-short/slides-plain.html))
4. Then we run ari.sh which does the following:

	- `make video` is run which runs [`bin/ari-make.sh`](https://github.com/galaxyproject/training-material/blob/main/bin/ari-make.sh)
	- This builds PDFs for any slides which have changed
	- And runs `./bin/ari.sh` with the PDF, the original Slides, and where the mp4 should be saved.
		- In [`./bin/ari.sh`](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh)
		- It [extracts metadata](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L38) from the tutorial (title, authors, etc.)
		- It [builds a 'script'](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L51), a json document with blocks for every line of the speaker notes that were in the slides.
		- [Those get converted into mp3 files](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L55) by AWS Polly (or MozillaTTS), one per slide.
		- The PDFs get turned into [a series of PNG images](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L60)
		- We take the timings of the mp3 files together with the json 'script' to [write out webvtt / srt subtitles](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L69) which get embedded into the video, and supplied next to it.
		- [editly is used](https://github.com/mifi/editly) to knit together the PNGs + mp3s with appropriate delay


All of this is run on cron by [`.github/workflows/video.yml`](https://github.com/galaxyproject/training-material/blob/main/.github/workflows/video.yml) which handles building all of these videos and then later uploading them to s3.

Many of the scripts internally are prefixed with `ari`, we named our internal version after [github.com/jhudsl/ari/](https://github.com/jhudsl/ari/) which inspired it, but we wanted a version that would be more closely tied to the GTN and integrate with our infrastructure nicely, so we ended up writing our own.

# Conclusion


topic: contributing
tutorial: updating_tutorial
title: Updating tool versions in a tutorial
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 8fc57b06bd7f92f8c9c844fdc2c14db6
last_modified: 2026-05-15T22:28:10.862916
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/updating_tutorial/tutorial.html

content:
Here, we provide a clear set of instructions for updating a GTN tutorial that uses the Galaxy interface for performing analysis. The GTN, and Galaxy, have a number of features to make it as reproducible and shareable as possible, so navigating what needs to be done for an overall update may feel daunting - but not anymore!

We encourage you to pick a tutorial to try this on, so you can use this tutorial side-by-side.

# Phase 1: Find a tutorial with an outdated workflow

{% include _includes/cyoa-choices.html option1="No" option2="Yes" default="No"
       text="Do you already have a tutorial and workflow you want to update?" %}

<div class="No" markdown="1">

> <hands-on-title>Check for an outdated workflow</hands-on-title>
>
> 1. **Find a {% icon hands_on %} Hands-on tutorial** in your training topic of interest (for example, Single-cell)
> 2. **Select the {% icon workflow %} workflow** from the header
> 3. **Import the workflow** to your favourite Galaxy server
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 4. Go to the **Workflow** menu and select {% icon galaxy-wf-edit %} **Edit workflow**
> 5. **Click through the tools** in the workflow and check the {% icon tool-versions %} to see if the tools are outdated.
{: .hands_on}

If the workflow has tools that are up to date (or very close!), great! That tutorial does not need updating! Try another one!

</div>

<div class="Yes" markdown="1">

> <hands-on-title>Import the workflow</hands-on-title>
>
> 1. **Navigate to your target {% icon hands_on %} Hands-on tutorial**
> 2. **Select the {% icon workflow %} workflow** from the header
> 3. **Import the workflow** to your favourite Galaxy server
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 4. Go to the **Workflow** menu and select {% icon galaxy-wf-edit %} **Edit workflow**
{: .hands_on}

</div>

# Phase 2: Check that nobody else is working on this

It's always a good idea to check, just in case!

You can:
 - Send a message to the [GTN Matrix Channel](https://matrix.to/#/#Galaxy-Training-Network_Lobby:gitter.im) is your quickest way forward.
 - Search through the [GTN Github Repository](https://github.com/galaxyproject/training-material) for existing draft Pull Requests.
 - Check with [individual communities](https://galaxyproject.org/community/sig/), who may have their own method of tracking. For example, the [🖖🏾Single-cell & sPatial Omics Community](https://galaxyproject.org/community/sig/singlecell/) have a shared [Click-Up board](https://sharing.clickup.com/9015477668/b/h/5-90152810734-2/557452707486fef) at the time of writing.

# Phase 3: Update the workflow

Now, you will update the workflow to using the latest {% icon tool-versions %} tool versions.

> <hands-on-title>Update the workflow</hands-on-title>
>
> 1. Select {% icon galaxy-wf-options %} **Workflow Options** from the **Workflow Editor**
> 2. Select {% icon upgrade_workflow %} **Upgrade workflow** to automatically update the tools.
> 3. Address any issues that may arise.
> 4. **Save** the workflow.
{: .hands_on}


# Phase 4: Test & fix the workflow

It's crucial to test the workflow, as often times the outputs will be different due to the new tool versions. It can also transpire that the newer tool versions lead to errors, either because they don't work or because you need to change parameter settings that were previously unavailable or not required. Testing is key!

> <hands-on-title>Test the workflow</hands-on-title>
>
> 1. Import the input datasets from the tutorial you are upgrading (follow the instructions on the tutorial itself for this).
> 2. Run your updated workflow on the input datasets.
> 3. Address any issues that may arise, and **NOTE DOWN** all changes.
> 4. Ensure the workflow meets workflow best practices using the {% icon galaxy-wf-best-practices %} **Best Practices** button and add your name as a contributor, if not there already.
>
>    {% snippet faqs/galaxy/workflows_best_practices.md %}
>
> 5. Add yourself as an author of the workflow.
> 6. **Save** the `updated workflow`.
> 7. Run your `updated workflow` on the input datasets in a fresh history.
> 7. **Save** this history as an `answer key history` & make your history publicly available in Published Histories.
>
>    {% snippet faqs/galaxy/histories_sharing.md %}
>
{: .hands_on}

# Phase 5: Update the tutorial

> <warning-title>If you run out of time</warning-title>
> You have already completed a large chunk of work (`updated workflow`+ `answer key history`) to get here, and we don't want to lose it!
> Updating the tutorial to match the `updated workflow` can be a separate contribution. So if you get to this point and run out of time, please:
> 1. Create an **issue** on the GTN Github Repository and include shareable links to your `updated workflow` & `answer key history`
> 2. Message on the GTN Matrix channel with a link to your issue and explaining which tutorial you updated.
> If, however, you are able to finish the task yourself, please read on!
{: .warning}

Note that you will need to have done either the [Contributing to the GTN Github tutorial using command line]({% link topics/contributing/tutorials/github-command-line-contribution/tutorial.md %}) or the [Contributing to the GTN Github tutorial using Github Desktop]({% link topics/contributing/tutorials/github-interface-contribution/tutorial.md %}). It's helpful as well to understand the folder structure in the GTN, particularly how images go in an image folder either in a tutorial or in the parent folder. Each `topic` has roughly the following structure:

Each topic has the following structure:

```
├── README.md
├── metadata.yaml
├── images
├── docker
│   ├── Dockerfile
├── slides
│   ├── index.html
├── tutorials
│   ├── tutorial1
│   │   ├── tutorial.md
│   │   ├── images
│   │   ├── slides.html
│   │   ├── data-library.yaml
│   │   ├── workflows
│   │   │   ├── workflow.ga
```

The `tutorial.md` is what you'll be editing, however you will also at the end upload a `workflow.ga` file and likely some image files.

> <hands-on-title>Update the tools in the tutorial</hands-on-title>
>
> 1. **Update tool versions**: Each {% icon hands_on %} *Hands-on* step in the tutorial will likely have tool (s) with versions in the text. Update these versions / links to be equivalent to your `updated workflow`.
> 2. **Update tool instructions**: (Simultaneously) update the text to address any differences arising during the update, i.e. new parameters to set or other changes.
> 3. **Update images**: Wherever images are used to show tool outputs, these will need updating. Use your final `answer key history` for this.
> 4. **Update text**: Wherever results are referenced in the tutorial text, update these numbers (and possibly interpretation) to reflect the new `answer key history`.
> 5. **Update header**: In the metadata at the beginning of the tutorial, there are likely `answer key history` links. Update this by adding the link to your `answer key history`.
> 6. **Update workflow file**: In the tutorial folder in the `training-material` repository, there is a subfolder titled `workflows`. Download your `updated workflow` as a file and deposit that file there.
> 7. Check if there are any links to workflows/histories in the tutorial text, and if so, update them.
> 7. Add your name as a *contributor* to the tutorial as an **editor** in the metadata.
{: .hands_on}

You may find this tutorial to be a helpful reference in the Markdown content of a GTN tutorial: [Creating content in Markdown]({% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md %})

# Phase 6: Make a pull request and (optional) update workflow testing

At this point, you're welcome to make your Pull Request for the updated tutorial. However, it is likely to fail linting if the workflow testing is not also updated. This can be tricky, so we'd rather you make the Pull Request with all the work you did than let this stop you.

But if you can make the workflow tests, that would be amazing!
Here's the tutorial: [Creating workflow tests using Planemo]({% link faqs/gtn/gtn_workflow_testing.md %})


{% icon congratulations %} Congratulations! You've made it to the end! Thank you so much for contributing to the sustainability of Galaxy tutorials!


topic: contributing
tutorial: schemas
title: GTN Metadata
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: []
tags_json: []
content_hash: f7e16da7a5691380af2360dd508317a6
last_modified: 2026-05-15T22:28:10.861828
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/schemas/tutorial.html

content:
## Training Materials

{% assign kid_key = "Tutorial Schema" %}
{% assign kid_val = site.data['schema-tutorial'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Slides Schema" %}
{% assign kid_val = site.data['schema-slides'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

## GTN Resources

{% assign kid_key = "FAQ Schema" %}
{% assign kid_val = site.data['schema-faq'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Topic Schema" %}
{% assign kid_val = site.data['schema-topic'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Quiz Schema" %}
{% assign kid_val = site.data['schema-quiz'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Learning Pathway Schema" %}
{% assign kid_val = site.data['schema-learning-pathway'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "News" %}
{% assign kid_val = site.data['schema-news'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Events" %}
{% assign kid_val = site.data['schema-event'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Events (External)" %}
{% assign kid_val = site.data['schema-event-external'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

## Contributors

{% assign kid_key = "Contributor Schema" %}
{% assign kid_val = site.data['schema-contributors'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Organisation Schema" %}
{% assign kid_val = site.data['schema-organisations'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Grant Schema" %}
{% assign kid_val = site.data['schema-grants'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}


topic: contributing
tutorial: create-new-tutorial-content
title: Creating content in Markdown
difficulty: intermediate
time_estimation: 15m
requirements_json: []
tags_json: []
content_hash: fbf0d5a678fc9c75e0addc241a568eab
last_modified: 2026-05-27T09:01:47.918269
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html

content:
Once we have set up the infrastructure, we are ready to write the tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

The tutorial's content should be placed in the file `tutorial.md`. Its syntax and structure are simple, and will have the following structure:

```markdown
---
layout: tutorial_hands_on

title: Title of the tutorial
zenodo_link: ''
questions:
- Which biological questions are addressed by the tutorial?
- Which bioinformatics techniques are important to know for this type of data?
objectives:
- The learning objectives are the goals of the tutorial
- They will be informed by your audience and will communicate to them and to yourself
  what you should focus on during the course
- They are single sentences describing what a learner should be able to do once they
  have done completed tutorial
- You can use Bloom's Taxonomy to write effective learning objectives
time_estimation: ''
key_points:
- The take-home messages
- They will appear at the end of the tutorial
contributors:
- contributor1
- contributor2
---

introduction text

> <agenda​-title></agenda​-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}

# Section 1

blabla

## Subsection 1

blabla

# Section 2

blabla

## Subsection 2

blabla

# Conclusion

blabla
```

# Metadata

The `tutorial.md` needs to start with some metadata at the top:

- `layout: tutorial_hands_on`: keep the default
- `title`: title of the tutorial (it will appear on the tutorial page and the topic page)
- `level`: `Introductory`, `Intermediate` or `Advanced`
- `zenodo_link`: link on Zenodo to the input data for the tutorial
- `contributions`: eveybody who has contributed to this tutorial (usernames must match those in `CONTRIBUTORS.yaml` file)

> <hands-on-title>Fill the basic metadata</hands-on-title>
>
> 1. Update the tutorial information in the header section of your tutorial:
>
>     ```
>     title: "Similarity search with BLAST"
>     ```
> 2. (Optional) Add the Zenodo link (if created)
>
{: .hands_on}

This information is used to display the data from the topic and tutorial page. They are also used to check which information is missing for the tutorials.

We also define metadata related to the pedagogical content of the tutorial, which will appear at the top ("Overview" box) and bottom of the online tutorial:

{% assign kid_key = "Tutorial Schema" %}
{% assign kid_val = site.data['schema-tutorial'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

For this category of metadata, we have taken inspiration from what Software Carpentry has done and particularly what they described in their [Instructor training](https://swcarpentry.github.io/instructor-training/).

> <hands-on-title>Fill out the pedagogical metadata</hands-on-title>
>
> 1. Define 2 questions that will be addressed during the tutorial and add them to the metadata
> 2. Define 2 learning objectives for the tutorial and add them to the metadata
{: .hands_on}

> <comment-title>When filling the pedagogical metadata</comment-title>
> We recommend that you fill out the *questions* and the *learning objectives* before starting writing the tutorial content. You can still refine them afterwards, but it will help to guide you in developing your tutorial, and gives you some time to think beforehand on what topics are worth being covered.
>
> For the take-home messages, it is easier to define them once the tutorial is written and you identified the issues.
{: .comment}


## Listing contributors

All tutorials and slides must give credit to all contributors. This can be any type of contribution, adding them in GitHub, creating images for it, etc.

1. Make sure all contributors are listed in the [`CONTRIBUTORS.yaml`](https://github.com/galaxyproject/training-material/blob/main/CONTRIBUTORS.yaml) file.
   Each contributor is defined in this file like:

   ```yaml
   contributor-username:                 # GitHub username (if the contributor has one)
     name: Full Name                     # mandatory
     joined: 2020-06                     # mandatory
     email: saskia.hiltemann@gmail.com   # optional
     twitter: shiltemann                 # optional
     linkedin: shiltemann                # optional
     gitter: shiltemann                  # optional
     orcid: 0000-0003-3803-468X          # optional
     bio: Researcher at EMC              # optional
   ```

2. Add all contributors to the metadata of the tutorial or slide deck:

   ```yaml
   contributors:
     - contributor-username
     - shiltemann
     - hexylena
   ```

   Make sure these names match the usernames you used in the `CONTRIBUTORS.yaml` file.

3. **Optional:** Specifying types of contributions. If you want to give more detailed credit for conributions, you can do the following (instead of step 2 above)

   ```yaml
   contributions:
     authorship:
       - shiltemann
     editing:
       - bebatut
       - hexylena
     funding:
       - carpentries
     testing:
       - userX
     ux:
       - userY
     infrastructure:
       - userZ
   ```

   To define a funding body in the `CONTRIBUTORS.yaml` there are a few extra fields available:

   ```yaml
   gallantries:
     name: Gallantries Project
     joined: 2020-09
     avatar: "https://www.erasmusplus.nl/assets/images/logo.png"
     github: false
     funder: true
     funding_id: 2020-1-NL01-KA203-064717
     funding_statement: |
        This project ([`2020-1-NL01-KA203-064717`](https://erasmus-plus.ec.europa.eu/projects/search/details/2020-1-NL01-KA203-064717)) is funded with the support of the Erasmus+ programme of the European Union. Their funding has supported a large number of tutorials within the GTN across a wide array of topics.
        ![eu flag with the text: with the support of the erasmus programme of the european union](https://gallantries.github.io/assets/images/logosbeneficaireserasmusright_en.jpg)
   ```

   Funding bodies will be credited at the bottom of the tutorial with the appropriate funding statement, and will get a page in the hall of fame listing all tutorials that list them as a funder.

   For an example of how this all looks, see the [R basics tutorial]({% link topics/data-science/tutorials/r-basics/tutorial.md %}) (top and bottom of the tutorial).

{% assign kid_key = "Contributions Schema" %}
{% assign kid_val = site.data['schema-tutorial']['mapping']['contributions'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "CONTRIBUTORS Schema" %}
{% assign kid_val = site.data['schema-contributors'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

# Content

The tutorial's content is written directly after the section of metadata. This is written in Markdown, a simple markup language.

> <comment-title>Markdown</comment-title>
>
> Check [this cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to learn more how to use Markdown.
{: .comment}

The Markdown content is then transformed into a user friendly webpage through a templating system. With this approach, there is no need to add the name of every tutorial each time, since they are automatically added based on the tutorial's metadata.

To help developing the tutorial, we recommend to create a workflow of the different steps of the tutorial inside Galaxy first, and then you can create the structure of the tutorial automatically from that:

> <hands-on-title>Create the structure of the tutorial from a workflow</hands-on-title>
>
> 1. Create a small workflow with one or two steps on a running Galaxy instance
> 2. Add the topic name as Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor.
> 3. Get the workflow id
>    1. Go the "Share" page of the workflow
>    2. Copy the information after `id=` in the URL of the page
> 4. Get your API key for this Galaxy instance
>    1. Click on **User** --> **Preferences**
>    2. Click on **Manage API key**
>    3. Click on **Create a new key** (if none is available)
>    4. Copy the API key
> 5. Generate the skeleton of the tutorial locally
>
>    ```
>    $ planemo training_generate_from_wf \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --galaxy_url "URL to Galaxy instance in which you created the workflow" \
>             --galaxy_api_key "Your API key on the Galaxy instance" \
>             --workflow_id "ID of the workflow on the Galaxy instance" \
>             --zenodo_link "URL to the Zenodo record (Optional)"
>    ```
>
>    > <comment-title>Using a local workflow</comment-title>
>    > It is also possible to download the workflow locally (with the `.ga` extension), and then run a slightly different command:
>    >
>    > ```
>    > $ planemo training_generate_from_wf \
>    >          --topic_name "my-topic" \
>    >          --tutorial_name "my-new-tutorial" \
>    >          -- workflow PATH/to/the/file.ga \
>    >          --zenodo_link "URL to the Zenodo record (Optional)"
>    > ```
>    {: .comment}
>
> 6. Inspect the generated `tutorial.md`
{: .hands_on}

The generated tutorial is structured with:

- An introduction, to give an overview of the tutorial with its use cases, data, and methods
- Multiple sections representing the steps of the analysis, complete with automatically generated hands-on blocks, as practicing is a vital part of the learning process
- A conclusion to summarize what has been done in the tutorial (with a graphic)

> <hands-on-title>Filling out the structure of the tutorial</hands-on-title>
>
> 1. Fill out the "Introduction" with a general introduction of the tutorial and a small description of the dataset (goals)
> 2. Rename/restructure the sections with several levels and more explication
> 3. Add some theory about the tool used to introduce each section
> 4. Add a small conclusion and relate the results to the original question
>
{: .hands_on}

## Self Study Tutorials

By following all of the guidelines in this file you can be sure that your tutorial will be optimised for self-study.
The GTN framework encourages the use of snippets, and PTDK ensures tutorials have fully detailed parameters learners should configure.
If you make use of snippets in appropriate places, learners can easily follow a tutorial despite different skill levels with Galaxy.

Additionally use of [Question and Solution boxes](#questions-and-solutions-boxes) can ensure that students can self-check their understanding and progress as they progress through the tutorial.
Generally we recommend one or more questions (and associated correct solutions!) after every hands-on box (which might have a one or more steps to follow.)
These questions will let students be sure their work is correct before they proceed too far.
As such you should design your questions carefully in order to catch common and likely failure modes that learners may encounter.
If a student might forget to select a less-common or deeply-nested parameter, be sure that your question following that hands on tests that properly, and if possible explain that in the solutions.

## Adding images with captions

To add an image in Markdown file, we need to use the markdown syntax for this: {% raw %}`!​[proper alt text describing the image for visually impaired learners](../../images/image.png)`{% endraw %}.

We have also added a small plugin to handle captions for each image:

![A textual description of the image](../../images/image_caption_screenshot.png "Example of an image with a caption ")<!-- Adding a space to the caption to not trigger figurigy skip_titles -->

The prefix "Figure 1." is automatically added before its caption. This is done with the following Markdown syntax:

{% raw %}
```markdown
!​[A textual description of the image](../images/image.png "Example of an image with a caption")
```
{% endraw %}

We can also cross-reference images inside our Markdown with an anchor. For example, we can link to [the previous figure](#figure-1) using `[the display text]​(#figure-nb)` (changing `nb` with the image's number).

### Guidelines on Alt vs Figcaption Text

> While both the alt attribute and the figcaption element provide a way to
> describe images, the way we write for them is different. **`alt` descriptions
> should be functional; `figcaption` descriptions should be editorial or
> illustrative.**
>
{: .quote cite="https://thoughtbot.com/blog/alt-vs-figcaption" author="thoughtbot.com"}

As an example for this image:

![alt text]({{site.baseurl}}/topics/microbiome/images/plasmid-metagenomics-nanopore/sequence_method.jpg "Example of an image with a caption ")

{% raw %}
```markdown
!​[Alt text (shown when image cannot be displayed)](path/to/image.png "Example of an image with a caption")
```
{% endraw %}

Field          | Appropriate Contents
----           | -----
alt text       | Image of cell membrance with an embedded protein with central pore. DNA is shown splitting and entering the pore, an electrical signal comes out reading A C T or G.
figure caption | Using nanopore sequencing, a single molecule of DNA or RNA can be sequenced without the need for PCR amplification or chemical labeling of the sample. (Image from: <a href="https://nanoporetech.com/sites/default/files/s3/white-papers/WGS_Assembly_white_paper.pdf?submissionGuid=40a7546b-9e51-42e7-bde9-b5ddef3c3512">Nanopore sequencing: The advantages of long reads for genome assembly</a>)

## Writing mathematical expressions

Mathematical expressions can be written in LaTeX, and are automatically rendered with [MathJax](https://www.mathjax.org/).

Surround your math expression with two `$` signs on each side (like in LaTeX math blocks):

- inline expressions, *e.g.* `$$ 5 + 5 $$` will be rendered as $$ 5 + 5 $$
- block expressions, *e.g.* `$$ 5 + 5 $$` will be rendered in its own line block as

   $$ 5 + 5 $$

- Note: if inline mode is not working correctly, you can force it by using the following delimiters instead of dollar signs: `\\( 5 +5 \\)`

Dollar signs are therefore *reserved characters* for instructing the templating system to open/close LaTeX math blocks. If you want to use a `$` within your expression, you will need to *escape* it: `$$ a + 3\$ = 5\$ $$` will be rendered as: $$ a + 3\$ = 5\$ $$


LaTeX code that uses the pipe symbol `|` in inline math statements may lead to a line being recognized as a table line by the templating system.
This can be avoided by using the `\vert` command instead of `|`

## Tables and Matrices

Tables can be generated using markdown by using the `|` symbol to indicate column dividers, and `--` for table headers:

{% raw %}
```markdown
|       | Obs1 | Obs2 | Obs3 |
|------ |--------------------|
| Feat1 | 0    | 1    | 2    |
| Feat2 | 1    | 2    | 3    |
| Feat3 | 2    | 3    | 4    |
```
{% endraw %}

When rendered, they will take the full width of the page:

|       | Obs1 | Obs2 | Obs3 |
|------ |--------------------|
| Feat1 | 0    | 1    | 2    |
| Feat2 | 1    | 2    | 3    |
| Feat3 | 2    | 3    | 4    |

This does not appear to be visually appealing when representing matrices, which is why a matrix box can be used instead:

{% raw %}
```markdown
> |       | Obs1 | Obs2 | Obs3 |
> | ----- |--------------------|
> | Feat1 | 0    | 1    | 2    |
> | Feat2 | 1    | 2    | 3    |
> | Feat3 | 2    | 3    | 4    |
{: .matrix}
```
{% endraw %}

The rendered table is then given as a minimum-width and centred matrix:

> |       | Obs1 | Obs2 | Obs3 |
> | ----- |--------------------|
> | Feat1 | 0    | 1    | 2    |
> | Feat2 | 1    | 2    | 3    |
> | Feat3 | 2    | 3    | 4    |
{: .matrix}

## Internally linking to other training material

If you want to link to other training material within your text, please use the {%raw%}`{​% link path/to/file.ext %​}`{%endraw%} tag:

{%raw%}
```markdown
[link text]( {% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %} )
```
{%endraw%}

(Note the `.md` extension, and not `.html`, always provide the file name here, it will automatically be converted to the correct link)

If you want to link to a specific section in a tutorial using an anchor (e.g. `#getting-started`), place it outside of the {%raw%}`{​% link %​}`{%endraw%} tag:

{%raw%}
```markdown
[link text]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}#section-name)
```
{%endraw%}


# Improving the learning experience with Boxes

To improve the learning experience in our tutorial, we define some boxes to highlight content.

These boxes are defined always with the same structure:

{% raw %}
```markdown
> <type​-title>Name of the box</type​-title>
> text goes here!
{: .type}
```
{% endraw %}

where type is something like `tip`, so `<tip-title>` and `{: .tip}`. You must follow this structure **exactly** for it to be rendered correctly.

## **Overview** box

This box at the top of each tutorial is automatically generated using the metadata we defined in the topic's metadata file:

> <overview-title>Overview</overview-title>
>
> **{% icon question %} Questions**
> - Which biological questions are addressed by the tutorial?
> - Which bioinformatics techniques are important to know for this type of data?
>
> **{% icon objectives %} Objectives**
> - The learning objectives are the goals of the tutorial
> - They will be informed by your audience and will communicate to them and to yourself what you should focus on during the course
> - They are single sentences describing what a learner should be able to do once they have completed the tutorial
> - You can use Bloom's Taxonomy to write effective learning objectives
>
> {% icon requirements %} Requirements
> - [Galaxy introduction]({% link topics/introduction/index.md %})
>
> {% icon time %} Time estimation: '1H'
>
{: .overview}

> <hands-on-title>Checking the metadata</hands-on-title>
>
> 1. Check that the metadata added previously are correctly filling the overview box
>
>    > <question-title></question-title>
>    >
>    > What metadata hasn't been added to this box?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The take-home messages are not added to this box but into the last box of the tutorial
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

## **Agenda** box

In most tutorials, the second box is the agenda box, placed at the end of the introduction. It shows the table of contents for the tutorial

{% raw %}
```markdown
> <agenda​-title></agenda​-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}
```
{% endraw %}

There is no need to fill out the list; this will be done automatically based off of your tutorial's section titles. First and second level headings (lines starting with `#` and `##` in markdown) will be shown in the agenda.

If a section is not important enough to be shown in the agenda, consider making it a third-level heading (`###`) instead.

The agenda will be rendered as follows:

> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## **Hands-on** box

We find that having users walk through the tutorial, doing all of the steps is important for learning the concepts taught. We therefore emphasize this by regularly adding hands-on sections, where trainees are encouraged to do the analysis by themselves. We have designed some special boxes to make these sections easier to find.

{% raw %}
```markdown
> <hands​-on-title>Spliced mapping</hands​-on-title>
>
> 1. **RNA STAR** {% icon tool %}: Map your reads on the reference genome with
>    - *"Single-end or paired-end reads"*:  `Paired-end (as individual datasets)`
>    - *"RNA-Seq FASTQ/FASTA file, forward reads"*: the generated `trimmed reads pair 1` files (multiple datasets)
>    - *"RNA-Seq FASTQ/FASTA file, reverse reads"*: the generated `trimmed reads pair 2` files (multiple datasets)
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>    - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>    - *"Select reference genome"*: `Drosophila Melanogaster (dm6)`
>    - *"Gene model (gff3,gtf) file for splice junctions"*: the imported `Drosophila_melanogaster.BDGP6.87.gtf`
>    - *"Length of the genomic sequence around annotated junctions"*: `36`
>
>        This parameter should be length of reads - 1
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}: Aggregate the STAR logs with
>      - *"Which tool was used generate logs?"*: `STAR`
>      - *"Type of FastQC output?"*: `Log`
>      - *"STAR log output"*: the generated `log` files (multiple datasets)
{: .hands​_on}
```
{% endraw %}

For consistency please use:

- {% raw %}`{% icon hands_on %}`{% endraw %} icon to define that is an hands-on
- Short imperative sentences to make it easy to identify the tasks
- Name of the tool in bold followed by {% raw %}`{% icon tool %}`{% endraw %} icon to make it easy to identify a Galaxy tool
- Parameters for the tool as a sublist

This will be rendered like:


> <hands-on-title>Spliced mapping</hands-on-title>
>
> 1. **RNA STAR** {% icon tool %}: Map your reads on the reference genome with
>    - *"Single-end or paired-end reads"*:  `Paired-end (as individual datasets)`
>    - *"RNA-Seq FASTQ/FASTA file, forward reads"*: the generated `trimmed reads pair 1` files (multiple datasets)
>    - *"RNA-Seq FASTQ/FASTA file, reverse reads"*: the generated `trimmed reads pair 2` files (multiple datasets)
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>    - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>    - *"Select reference genome"*: `Drosophila Melanogaster (dm6)`
>    - *"Gene model (gff3,gtf) file for splice junctions"*: the imported `Drosophila_melanogaster.BDGP6.87.gtf`
>    - *"Length of the genomic sequence around annotated junctions"*: `36`
>
>        This parameter should be length of reads - 1
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}: Aggregate the STAR logs with
>      - *"Which tool was used generate logs?"*: `STAR`
>      - *"Type of FastQC output?"*: `Log`
>      - *"STAR log output"*: the generated `log` files (multiple datasets)
{: .hands_on}

There are also some predefined **parameter icons** available which can be used to indicate the *type* of parameter. These are optional,
but may be helpful in some cases (for example to distinguish between single file inputs and collection inputs).

The available icons are:

{% raw %}
```markdown
> <hands​-on-title>My Step</hands​-on-title>
>
> 1. **My Tool** {% icon tool %} with the following parameters
>    - {% icon param-text %} *"My text parameter"*: `my value`
>    - {% icon param-file %} *"My input file"*: `my file`
>    - {% icon param-files %} *"My multiple file input or collection"*: `my collection`
>    - {% icon param-select %} *"My select menu"*: `my choice`
>    - {% icon param-check %} *"My check box"*: `yes`
>    - {% icon param-toggle %} *"My toggle button"*: `Yes`
>    - {% icon param-repeat %} **My repeat parameter**
>      - *"param1"*: `42`
{: .hands​_on}
```
{% endraw %}

which, when rendered, look like:

> <hands-on-title>My Step</hands-on-title>
>
> 1. **My Tool** {% icon tool %} with the following parameters
>    - {% icon param-text %} *"My text parameter"*: `my value`
>    - {% icon param-file %} *"My input file"*: `my file`
>    - {% icon param-files %} *"My multiple file input or collection"*: `my collection`
>    - {% icon param-select %} *"My select menu"*: `my choice`
>    - {% icon param-check %} *"My check box"*: `yes`
>    - {% icon param-toggle %} *"My toggle button"*: `Yes`
>    - {% icon param-repeat %} **My repeat parameter**
>      - *"param1"*: `42`
{: .hands_on}

## **Questions** and **Solutions** boxes

Questions can be added to force trainees to think about what they are currently doing, and to put things in perspective.
They can also help the instructors by exposing and clarifying common scenarios, errors, or applications.

{% raw %}
```markdown
> <question​-title></question​-title>
>
> 1. Why are some tests filtered?
> 2. Does it improve the *p*-value distribution?
>
> > <solution​-title></solution​-title>
> >
> > 1. Sol for the first question
> > 2. Sol for the second question
> >
> {: .solution}
{: .question}
```
{% endraw %}

Which will be rendered as:

> <question-title></question-title>
>
> 1. Why are some tests filtered?
> 2. Does it improve the *p*-value distribution?
>
> > <solution-title></solution-title>
> >
> > 1. Sol for the first question
> > 2. Sol for the second question
> >
> {: .solution}
{: .question}

Questions should be quick to answer. You can directly ask a question and expect an answer, or you can provide some answers and create multiple choices questions (MCQs).
With well chosen wrong answers, MCQs can do much more than just measure how much someone knows, such as exposing common misconceptions and mistakes.

In the box below, initially hidden, we add the correct answer and possibly any additional explanation. Self-trainees can then check the solution and its explanation.

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching-selfstudy.md %}


## **Tips** box

Tips boxes are really just for 'tips', usually hints regarding Galaxy operations that users may or may not be familiar with. If you want to provide extended discussion or links to external materials then consider the comment and detail boxes instead.

{% raw %}
```markdown
> <tip​-title>Importing data via links</tip​-title>
>
> * Copy the link location
> * Open the Galaxy Upload Manager
> * Select **Paste/Fetch Data**
> * Paste the link into the text field
> * Press **Start**
{: .tip​}
```
{% endraw %}

Rendered:

> <tip-title>Importing data via links</tip-title>
>
> * Copy the link location
> * Open the Galaxy Upload Manager
> * Select **Paste/Fetch Data**
> * Paste the link into the text field
> * Press **Start**
{: .tip}

## **Comments** boxes

{% raw %}
```markdown
> <comment​-title></comment​-title>
> - Edit the "Database/Build" to select "dm3"
> - Rename the datasets according to the samples
{: .comment}
```
{% endraw %}

Rendered:

> <comment-title></comment-title>
> - Edit the "Database/Build" to select "dm3"
> - Rename the datasets according to the samples
{: .comment}

## **Details** box

The detail box is used to give more background explanation on the subject. By default the box is collapsed, trainees can expand it if they wish to know extra information about a topic.

{% raw %}
```markdown
> <details​-title>More details on the ....</details​-title>
>
> Add more details in Markdown...
>
{: .details​}
```
{% endraw %}

Rendered:

> <details-title>More details on the ....</details-title>
>
> Add more details in Markdown...
>
{: .details}

## **Key points** box

This last box of the tutorial is automatically created with the take-home messages defined in the topic's metadata

To render the boxes correctly, the syntax needs to be correct. If it doesn't work, have a look at similar tutorials and get inspiration.

## **Warning** box

{% raw %}
```markdown
> <warning​-title>Danger: You can lose data!</warning​-title>
> Something really bad can happen here!
{: .warning}
```
{% endraw %}

Rendered:

> <warning-title>Danger: You can lose data!</warning-title>
> Something really bad can happen here!
{: .warning}


## Nested boxes

Boxes can be nested, *e.g.* for having tips inside a hands-on:

{% raw %}
```markdown
> <hands​-on-title>Defining the topic for the tutorial</hands​-on-title>
>
> 1. Search for NCBI Blast+ on the [ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. Check in which category it is
>
>    > <question​-title></question​-title>
>    >
>    > In which topic will you put the tutorial?
>    >
>    > > <solution​-title></solution​-title>
>    > >
>    > > If we search for [NCBI Blast+ in the ToolShed](https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/7538e2bfcd41), it is attributed to 2 categories (bottom): "Next Gen Mappers" and "Sequence Analysis".
>    > > We decided to put it in "Sequence analysis" because this is the most general one for this tutorial.
>    > {: .solution​}
>    {: .question​}
{: .hands​_on}
```
{% endraw %}

## **Code** box

We have added code in/out boxes to help you show commands, and their effects, when running command line commands.

Normally a single column, with the boxes above one another, it will automatically split side-by-side over a given width (1200px);

{% raw %}
```markdown
> > <code​-in-title>Bash</code​-in-title>
> > ```bash
> > cat /tmp/test.ini
> > ```
> {: .code-in}
>
> > <code​-out-title></code​-out-title>
> > The file should look like:
> >
> > ```ini
> > [example]
> > server_name = Dogs!
> > listen = 192.168.0.2
> > apikey = super-secret-api-key-wow!
> > ```
> {: .code-out}
{: .code-2col}
```
{% endraw %}

Rendered (try it! resize your browser)

> > <code-in-title>Bash</code-in-title>
> > ```bash
> > cat /tmp/test.ini
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > The file should look like:
> >
> > ```ini
> > [example]
> > server_name = Dogs!
> > listen = 192.168.0.2
> > apikey = super-secret-api-key-wow!
> > ```
> {: .code-out}
{: .code-2col}

If you leave off the `{: .code-2col}`, it will render as a single column always.

{% raw %}
```markdown
> <code​-in-title>Bash</code​-in-title>
> ```bash
> cat /tmp/test.ini
> ```
{: .code-in}

> <code​-out-title></code​-out-title>
> The file should look like:
>
> ```ini
> [example]
> server_name = Dogs!
> listen = 192.168.0.2
> apikey = super-secret-api-key-wow!
> ```
{: .code-out}
```
{% endraw %}

Rendered:

> <code-in-title>Bash</code-in-title>
> ```bash
> cat /tmp/test.ini
> ```
{: .code-in}

> <code-out-title></code-out-title>
> The file should look like:
>
> ```ini
> [example]
> server_name = Dogs!
> listen = 192.168.0.2
> apikey = super-secret-api-key-wow!
> ```
{: .code-out}

## **Quote** boxes

{% raw %}
```markdown
> If you don't know where you're going, you might not get there.
{: .quote cite="https://en.m.wikiquote.org/wiki/Yogi_Berra" author="Yogi Berra"}
```
{% endraw %}

Rendered:

> If you don't know where you're going, you might not get there.
{: .quote cite="https://en.m.wikiquote.org/wiki/Yogi_Berra" author="Yogi Berra"}

The citation and author parameters are both optional. If provided the `cite` key must be a URL.

> If you don't know where you're going, you might not get there.
{: .quote cite="https://en.m.wikiquote.org/wiki/Yogi_Berra"}

> If you don't know where you're going, you might not get there.
{: .quote author="Yogi Berra"}

# Additional Features to Improve Learning

Here we cover additional features you can use throughout your tutorials to improve the learning experience.

## Tool Links

With the new [GTN in Galaxy Webhook](https://github.com/galaxyproject/galaxy/pull/10024), trainees can view training directly within Galaxy. As part of this, we enable those trainees to click on tools, and have those tools directly activated in Galaxy, enabling for a seamless training experience for trainees.

![GIF of a user using the GTN in Galaxy webhook.](../../images/88277962-ddda4a80-cce1-11ea-92cd-41b1df063db0.gif "A gif showing how the GTN in Galaxy webhook works. A student clicks the learning hat icon in the masthead of a Galaxy server, and an overlay is activated showing the GTN website. Within the GTN they can browse around and their place in tutorials is saved. While following a tutorial the student reches a step which instructs them to run a specific tool. Instead of the normal experience searching for a tool (quite difficult on large servers), they click a blue button and the tool is activated in Galaxy, and the overlay is closed. The student can reactivate the overlay at any time and return to their place in the tutorial.")

To enable these in your tutorial you can use the following syntax:

{% raw %}
```
- {% tool MultiQC %}
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}
- {% tool [Import some data](upload1) %}
```
{% endraw %}

Which will be rendered as:

- {% tool MultiQC %}
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}
- {% tool [Import some data](upload1) %}

When viewed through Galaxy, students will see:

<span data-tool="upload1" title="Tested with upload1" class="tool galaxy-proxy-active"><strong>Import some data</strong> <i class="fas fa-wrench" aria-hidden="true"></i><i aria-hidden="true" class="fas fa-cog"></i><span class="visually-hidden">Tool: upload1</span></span>

### How to find these IDs?

The easiest way is to use planemo to generate the training from a workflow. In recent versions of planemo, this is managed automatically.

The alternative is to figure out the ID for the tool you want to use:

1. Find your tool in Galaxy, and click to access the tool form.
2. Click on Options at the top right
3. Click on Share
4. The URL shown will be something like `https://usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/galaxyp/mz_to_sqlite/mz_to_sqlite/2.0.4+galaxy1`
5. Keep only the part after the `=`, so `toolshed.g2.bx.psu.edu/repos/galaxyp/mz_to_sqlite/mz_to_sqlite/2.0.4+galaxy1` in this example

![Finding the tool ID](../../images/tool-id.png)

## Example Histories

If you have example input histories for your tutorial, perhaps for specific servers where trainees will often follow a tutorial but want to skip a slow input step, then you can provide example histories as part of your tutorial.

> <code-in-title>Tutorial Frontmatter</code-in-title>
> ```yaml
> answer_histories:
>   - label: "UseGalaxy.eu"
>     history: https://humancellatlas.usegalaxy.eu/u/j.jakiela/h/generating-a-single-cell-matrix-using-alevin-3
>   - label: "Older Alevin version"
>     history: https://humancellatlas.usegalaxy.eu/u/wendi.bacon.training/h/cs1pre-processing-with-alevin---answer-key
>     date: 2024-01-01
> input_histories:
>   - label: "UseGalaxy.eu"
>     history: https://humancellatlas.usegalaxy.eu/u/wendi.bacon.training/h/cs1pre-processing-with-alevin---input-1
> ```
{: .code-in}

> <code-out-title>Rendered Tutorial</code-out-title>
> ![a screenshot of the GTN metadata box showing a dropdown for input histories and answer histories. The answer histories features two examples one on UseGalaxy.eu and an older alevin one, each with a date. In the dropdown is also a link to an FAQ titled how to use this](images/example-histories.png)
{: .code-out}

## Workflows

In some tutorials you aren't as interested in teaching users the individual steps for analysing data, but rather want to focus on some downstream aspects of analysis, or to showcase the best practice workflows that are already available for a user to use! In those cases it can be useful to have a nicer way of inviting the user to execute those steps.

If you are accessing these in tutorial mode, they should function as button that, when clicked, launch the workflow directly. Outside of tutorial mode, they will link to our redirection service which will let you supply which Galaxy you plan to use.

Note that if for some reason the 'fancy' method doesn't work, there are fallback tip boxes to help a user execute a similar procedure manually.

### GTN Workflows

If you've included the workflow in the GTN but haven't uploaded to a repository yet:

{% raw %}
```markdown
{% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Data_QC.ga" title="Galaxy Workflow Data QC" %}
```
{% endraw %}

Rendered:

{% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Data_QC.ga" title="Galaxy Workflow Data QC" %}


### WorkflowHub

You can use a dedicated snippet to invite users to run a WorkflowHub workflow:

{% raw %}
```markdown
{% snippet faqs/galaxy/workflows_run_wfh.md title="mRNA-Seq BY-COVID Pipeline" wfhub_id="685" version="1" %}
```
{% endraw %}

Rendered:

{% snippet faqs/galaxy/workflows_run_wfh.md title="mRNA-Seq BY-COVID Pipeline" wfhub_id="685" version="1" %}

Note that it links to a specific workflow, on any Galaxy server. When this tutorial is opened from within the Tutorial Mode, that link will change to one on the current server, removing the intermediate step.

### Dockstore

Please note that the dockstore ID should be provided without the `#workflow/` prefix, so starting from `github.com`.

{% raw %}
```markdown
{% snippet faqs/galaxy/workflows_run_ds.md title="K-mer Profiling HiFi" dockstore_id="github.com/iwc-workflows/kmer-profiling-hifi-VGP1/main" version="v0.1.5" %}
```
{% endraw %}

Rendered:

{% snippet faqs/galaxy/workflows_run_ds.md title="K-mer Profiling HiFi" dockstore_id="github.com/iwc-workflows/kmer-profiling-hifi-VGP1/main" version="v0.1.5" %}

This snippet has the same behaviour, it will use my.galaxy.training links to make them server independent, but in Tutorial Mode it will open on the current server.


## FAQs (snippets)

Many common questions or instructions may be useful to share between different tutorials. For example instructions on how to start a new history or importing data. To make these types of snippets easier to re-use and avoid duplication, they are available in the form of *snippets*.

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching.md %}

### Finding snippets
These are available in folders named `faqs`, either at the project level, topic level, or tutorial level.

- **Project-level FAQs:** `faqs/`
  - `faqs/galaxy/` for general Galaxy questions
  - `faqs/gtn/` for questions regarding the GTN website itself

- **Topic-level FAQs:** `topics/<topic>/faqs/`
  - for questions pertaining to that specific topic

- **Tutorial-level FAQs:** `topics/<topic>/tutorials/<tutorial>/faqs/`
  - for questions pertaining to that specific tutorial
  - if this is present, it is linked to from the tutorial overview box at the top, and from the end of the tutorial


### Including FAQs/snippets in your tutorials

To include one of these snippets in your tutorial, you can use the following syntax:

```
{% raw %}{% snippet faqs/galaxy/histories_create_new.md %}{% endraw %}
```

Which will be rendered as:

{% snippet faqs/galaxy/histories_create_new.md %}

The advantage of this approach is that when the Galaxy interface updates, we only have to update the snippet, rather than every tutorial. Please try to use snippets whenever you can!

You could also specify the box type you want the snippet to be rendered in:

```
{% raw %}{% snippet faqs/galaxy/histories_create_new.md box_type="hands_on" %}{% endraw %}
```

{% snippet faqs/galaxy/histories_create_new.md box_type="hands_on" %}

or without a box altogether:

```
{% raw %}{% snippet faqs/galaxy/histories_create_new.md box_type="none" %}{% endraw %}
```

{% snippet faqs/galaxy/histories_create_new.md box_type="none" %}



### Creating new FAQs/snippets

Do you want to include something in your tutorial that you think might be useful in other tutorials as well? Or are you answering a frequently asked question? Consider creating a snippet for it

Each snippet (question) is a separate file, with some metadata, residing in one of the `faqs` folders:

```yaml
---
title: How do I run a workflow?
area: workflows      # FAQs will be grouped by these areas on the FAQ page
box_type: tip        # tip/comment/hands_on/question; optional, if you want the content to be in a box
layout: faq          # if you set this the snippet will get its own page and be included in the FAQs page
contributors: [annefou,nomadscientist] # anybody who has contributed to the FAQ over time
---

Here you can write the snippet / answer to the FAQ in Markdown

- Go to `Workflows` on the top menu bar
- Click on ..
- ..

```

{% assign kid_key = "FAQ Schema" %}
{% assign kid_val = site.data['schema-faq'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching-faqs.md %}

### FAQ pages

All FAQs will also be collected on their own page, this makes it easy for and teachers to prepare the session, and for participants to quickly find the answers to common questions.


To do this, create a file named `index.md` inside the faq folder:

```yaml
---
layout: faq-page
---
```

If you would like to enforce an order of the faq areas, you can do so:

```yaml
---
layout: faq-page
area_order: [introduction, learners, instructors, contributors, other]
---
```

(just make sure you list all existing areas in the folder)

If a tutorial-level FAQ page exists (`topics/<topic>/tutorials/<tutorial>/faqs/index.md`) it will be automatically linked to from the overview box at the top of the tutorial, and at the end of the tutorial. Have a look at this tutorial to see it in action.

## Footnotes

> > <code-in-title>Markdown</code-in-title>
> >
> > ```
> > Footnotes[^test] can be used to insert more content or explanation as reference material to your content. You can use the same footnote reference multiple time, and the footnote will include backlinks to return to the correct place in the text.[^test]
> > ```
> > {: .pre-break-lines}
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > Footnotes[^test] can be used to insert more content or explanation as reference material to your content. You can use the same footnote reference multiple time, and the footnote will include backlinks to return to the correct place in the text.[^test]
> {: .code-out}
{: .code-2col}

[^test]: The wikipedia definition of a footnote is: "A note is a string of text placed at the bottom of a page in a book or document or at the end of a chapter, volume or the whole text. The note can provide an author's comments on the main text or citations of a reference work in support of the text. Footnotes are notes at the foot of the page while endnotes are collected under a separate heading at the end of a chapter, volume, or entire work. Unlike footnotes, endnotes have the advantage of not affecting the layout of the main text, but may cause inconvenience to readers who have to move back and forth between the main text and the endnotes."

## Icons

To use these icons, take the name of the icon, 'details' in this example, and write something like this in your tutorial:

```markdown
{% raw %}{% icon details %}{% endraw %}
```

Some icons have multiple aliases, any may be used, but we'd suggest trying to choose the most semantically appropriate one in case Galaxy later decides to change the icon.

<div class="row">
{% assign icon_groups = site['icon-tag'] | group_icons %}
{% for icon in icon_groups %}
	<div class="col-md-2 col-sm-3" style="text-align: center">
		<div style="font-size: 400%">{% icon_var icon[0][0] %}</div>
		<div>{% for z in icon[0] %}{{ z }}{%unless forloop.last%},{%endunless%} {% endfor %}</div>
	</div>
{% endfor %}
</div>

## Abbreviations

Oftentimes there are terms you'll use over and over again where there is an acronym or abbreviation you'll use consistently. However for learners new to the material, they might need to scroll up to the first definition every time to remember what it meant. It would be annoying as an author to have to re-define it every time, so we've implemented some very simple syntax to allow you to create a list of definitions and then use those in the text.

In your tutorial metadata you can add an abbreviations section like:

```yaml
---
title: My awesome tutorial
...
abbreviations:
  API: Application Programming Interface
  JSON: JavaScript Object Notation
---
```

And in your text you can use braces to refer to the term

> > <code-in-title>Markdown</code-in-title>
> > <code>
> > The `/jobs` &lbrace;API&rbrace; will return &lbrace;JSON&rbrace;. When we call the &lbrace;API&rbrace; we'll get back this result &lbrace;JSON&rbrace;.
> > </code>
> {: .code-in}
>
> > <code-out-title></code-out-title>
> >
> > The `/jobs` {API} will return {JSON}. When we call the {API} we'll get back this result {JSON}.
> >
> {: .code-out}
{: .code-2col}

{% assign kid_key = "Abbreviations Schema" %}
{% assign kid_val = site.data['schema-tutorial']['mapping']['abbreviations'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

## Choose Your Own Tutorial

Sometimes you're writing a large tutorial and at one small step there are multiple paths or multiple ways to get the data you want, and you'd like to showcase them all! You could write them all out in order, Option 1...2...etc, however maybe you want it to be a bit more interactive and focus only on one option at a time, so a user doesn't get distracted by the other options.

Include this markdown where you want your user to choose between the multiple paths:

<!-- GTN:IGNORE:041 we cannot tell code samples from text so we get dupe warnings here. -->

> <code-in-title>Markdown</code-in-title>
> {% raw %}
> ```
> {% include _includes/cyoa-choices.html option1="Ananas of course" option2="Avocados" default="Avocados"
>        text="Here is why some people choose Ananas. Other times you want Avocados as they fit the menu better." %}{% endraw %}
> ```
{: .code-in}

{% include _includes/cyoa-choices.html option1="Ananas of course" option2="Avocados" default="Avocados" text="Here is why some people choose Ananas. Other times you want Avocados as they fit the menu better." %}

And then they can wrap the relevant sections with a `div` block with the relevant class (if you have space in your option, use `-` in the class). You **must** set `markdown="1"` as well to have the inner contents rendered corretly.

**NB**: If you do not set a default, then on the very first page load, both options will be shown in their entirety. As soon as the user selects one of the options by clicking the relevant button, then the list is filtered. The user's browser generally will remember which button was selected across navigation and page reloads.

> > <code-in-title>Markdown</code-in-title>
> > ```
> > <div class="Ananas-of-course" markdown="1">
> > - 🍍 are fantastic
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> > <div class="Avocados" markdown="1">
> > - 🥑 are amazing
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> > ```
> >
> {: .code-in}
>
> > <code-out-title></code-out-title>
> >
> > <div class="Ananas-of-course" markdown="1">
> > - 🍍 are fantastic
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> > <div class="Avocados" markdown="1">
> > - 🥑 are amazing
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> >
> {: .code-out}
{: .code-2col}

> <tip-title>Why ananas-of-course? Name munging</tip-title>
> CYOAs work by filtering the content based on the class name. Class names cannot contain spaces, so, we need to replace all whitespace with `-`.
> The exact algorithm we use to create 'safe' IDs is:
>
> 1. Remove the following character `"`, `'`, `/`, `;`, `:`, `,`, `.`, `!`, `@`, `#`, `$`, `%`, `^`, `&`, `*`, `(`, `)` from the text.
> 2. Replace all whitespace with `-`
> 3. Replace multiple `-`s with a single `-`
>
> This results in retaining capitalisation. Hence, `Ananas of course` becomes `Ananas-of-course`.
> For a more complicated example `(RNA) inhibitors` would become `RNA-inhibitors`.
{: .tip}

This can also be used inline: My favourite fruit is an <span class="Ananas-of-course">🍍</span><span class="Avocados">🥑</span>.

> <tip-title>Multiple, Disconnected CYOAs</tip-title>
> If you wish to have multiple CYOAs in a single tutorial, you are free to do that! However you must:
>
> 1. Ensure that all options are disjoint, there should not be any shared terms! (I.e if the both CYOAs need to use "STAR", please find a different way to phrase it, or even use "STAR ", it just needs to be different.)
> 2. Provide a disambiguation term for them, passed as a parameter to all, or all but one, includes.
>
> This disambiguation term will affect the URL parameter, which will become `?gtn-cyoa{term}={value}`
>
> E.g.:
>
> ```
> {% raw %}
> {% include _includes/cyoa-choices.html option1="Oui" option2="Non" default="Oui" text="Vos données ESTAMP sont prêtes ?" %}
> {% include _includes/cyoa-choices.html option1="Yes" option2="No" default="Yes" text="Do the thing?" disambiguation="english" %}
> {% endraw %}
> ```
{: .tip}

> <tip-title>Using every branch</tip-title>
> We, the GTN, don't always know if you *intend* to have not used a specific branch. As such we recommend that if you have a branch that has no steps, you should include a `div` block with the class name of that branch, just so we know for sure you meant it to be empty.
>
> E.g. 
>
> ```
> {% raw %}
> {% include _includes/cyoa-choices.html option1="Yes, they don't pass QC" option2="No" default="Yes, they don't pass QC" text="Do you think your samples need to be filtered?" %}
> 
> <div class="Yes-they-dont-pass-QC" markdown="1">
> Some content here!
> </div>
> 
> <div class="No" markdown="1">
> <!-- intentionally empty. This comment itself isn't necessary, but we recommend adding the div. -->
> </div>
> 
> {% endraw %}
> ```
{: .tip}

### URL Parameter

The branch can be selected via URL parameter e.g. for courses, to prevent users selecting the wrong path. Just supply `?gtn-cyoa=Ananas` (or your preferred value) on the tutorial URL.

- [See this page with Ananas](?gtn-cyoa=Ananas#choose-your-own-tutorial)
- [See this page with Avocados](?gtn-cyoa=Avocados#choose-your-own-tutorial)

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching-cyoa.md %}

# Citations
If you would like to cite any articles, books or websites in your tutorial, you can do so by adding a file called `tutorial.bib` next to your `tutorial.md` file. In this file you may enter [bibtex](http://www.bibtex.org/Using/) formatted citations. An e

tools_json:
["MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0", "Import some data", "upload1"]


topic: contributing
tutorial: admin-knitting
title: Updating diffs in admin training
difficulty: intermediate
time_estimation: 5m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 0465728964d9ef87b105a21cf3b9f90b
last_modified: 2026-05-15T22:28:10.803914
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/admin-knitting/tutorial.html

content:
# Updating Diffs


The admin training was recently converted completely to use diffs. This was
done so we could build a git repository, and time travel to any point within
admin training, in support of students and testing.

Here's some common editing scenarios to help you out.

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## How it Works

Tutorials are written with diffs that look like this:

    ```diff
    --- /dev/null
    +++ b/ansible.cfg
    @@ -0,0 +1,4 @@
    +[defaults]
    +interpreter_python = /usr/bin/python3
    +inventory = hosts
    +retry_files_enabled = false
    ```
    {: data-commit="Add ansible.cfg"}

The `data-commit` will be turned into the commit message, so, you can write something nice here. Otherwise it's a pretty standard diff. For tutorials which are written in this way, we can extract all of the diffs.

### Frogging

In knitting you sometimes need to rip out the stitches, a process sometimes called frogging.

> If they are not secured, the loops of a knitted course will come undone when their yarn is pulled; this is known as ripping out, unravelling knitting, or humorously, frogging (because you 'rip it', this sounds like a frog croaking: 'rib-bit').
{: .quote cite="https://en.wikipedia.org/wiki/Knitting"}

```console
$ python bin/knit-frog.py topics/admin/tutorials/singularity/tutorial.md /tmp/03-singularity
/tmp/03-singularity-commit-0000-add-golang-and-singulary-ansible-roles.patch
/tmp/03-singularity-commit-0001-configure-golang-and-singularity.patch
/tmp/03-singularity-commit-0002-add-the-roles-to-the-playbook.patch
/tmp/03-singularity-commit-0003-configure-the-container-and-dependency-resolvers.patch
/tmp/03-singularity-commit-0004-configure-the-dependency-resolvers.patch
/tmp/03-singularity-commit-0005-configure-the-container-resolver.patch
/tmp/03-singularity-commit-0006-update-the-job-conf-xml-with-singularity-destination.patch
```

This rips out the diffs and writes them into patch files.

### Knitting

With a series of patch files, we have a similar script to knit them back together.

```console
$ python bin/knit.py topics/admin/tutorials/singularity/tutorial.md --patches /tmp/03-singularity*
```

This takes the patch files and lines them up in the tutorial. It is **not intelligent**. You should have the same number of patch files as you do diffs with `data-commit`. Up to you to manage that.

## Automation

But mostly you don't need to care about the `knit.py` or `knit-frog.py`, we have `knit-automated.sh` which takes care of calling those properly. You can call it with `export` to write the patches into `/tmp/git-gat/`

```
$ ./bin/knit-automated.sh export
/tmp/git-gat/0-ansible-galaxy-commit-0000-add-requirements.patch
/tmp/git-gat/0-ansible-galaxy-commit-0001-add-ansible-cfg.patch
/tmp/git-gat/0-ansible-galaxy-commit-0002-add-hosts.patch
/tmp/git-gat/0-ansible-galaxy-commit-0003-add-initial-group-variables-file.patch
/tmp/git-gat/0-ansible-galaxy-commit-0004-add-initial-galaxy-playbook.patch
/tmp/git-gat/0-ansible-galaxy-commit-0005-add-pip--miniconda--galaxy-to-playbook.patch
/tmp/git-gat/0-ansible-galaxy-commit-0006-configure-miniconda-and-galaxy.patch
[...]
/tmp/git-gat/6-pulsar-commit-0000-add-requirements.patch
/tmp/git-gat/6-pulsar-commit-0001-configure-rabbitmq.patch
/tmp/git-gat/6-pulsar-commit-0002-add-role.patch
/tmp/git-gat/6-pulsar-commit-0003-add-pulsar-group-variables.patch
/tmp/git-gat/6-pulsar-commit-0004-add-pulsar-host.patch
/tmp/git-gat/6-pulsar-commit-0005-add-pulsar-playbook.patch
/tmp/git-gat/6-pulsar-commit-0006-add-pulsar-plugin.patch
/tmp/git-gat/6-pulsar-commit-0007-add-pulsar-destination.patch
/tmp/git-gat/6-pulsar-commit-0008-send-bwa-and-bwa-mem-to-pulsar.patch
```

And there you can build a repository from them!

```
[hxr@mk:/tmp/git-gat]$ git init
[hxr@mk:/tmp/git-gat]$ git am -3 *
Applying: admin/ansible-galaxy/0000: Add requirements
applying to an empty history
Applying: admin/ansible-galaxy/0001: Add ansible.cfg
Applying: admin/ansible-galaxy/0002: Add hosts
Applying: admin/ansible-galaxy/0003: Add initial group variables file
Applying: admin/ansible-galaxy/0004: Add initial galaxy playbook
Applying: admin/ansible-galaxy/0005: Add pip, miniconda, galaxy to playbook
[...]
Applying: admin/pulsar/0004: Add pulsar host
Applying: admin/pulsar/0005: Add pulsar playbook
Applying: admin/pulsar/0006: Add pulsar plugin
Applying: admin/pulsar/0007: Add pulsar destination
Applying: admin/pulsar/0008: Send bwa and bwa-mem to pulsar
```

Et voila!

```console
$ rm *.patch
$ tree
.
├── ansible.cfg
├── files
│   └── galaxy
│       ├── config
│       │   ├── dependency_resolvers_conf.xml
│       │   └── tool_destinations.yml
│       ├── dynamic_job_rules
│       │   ├── map_resources.py
│       │   └── my_rules.py
│       └── tools
│           └── testing.xml
├── galaxy.yml
├── group_vars
│   ├── all.yml
│   ├── galaxyservers.yml
│   └── pulsarservers.yml
├── hosts
├── pulsar.yml
├── requirements.yml
└── templates
    ├── galaxy
    │   └── config
    │       ├── container_resolvers_conf.xml.j2
    │       ├── job_conf.yml.j2
    │       └── job_resource_params_conf.xml.j2
    └── nginx
        ├── galaxy.j2
        └── redirect-ssl.j2

10 directories, 18 files
```

When you're done editing, you can run the `./bin/knit-automated.sh import`, it will

- extract the history as patches (using `git format-patch`)
- knit them into the tutorials

> <warning-title>Danger: Don't mess with the commit names at this step</warning-title>
> They're used for figuring out which commit a patch file belongs to. Sorry!
{: .warning}


# Git-Gat Knick-Knacks: Some Common Scenarios

## I want to update the text of a specific commit

> <question-title>Example problem</question-title>
> I want to edit the commit titled "Add production facing vars" to add an addtional variable in admin/ansible-galaxy.
{: .question}

> <hands-on-title>Make the change you want to see in the GAT</hands-on-title>
> 1. Start by exporting
>
>    ```
>    $ bash bin/knit-automated.sh export
>    ```
>
> 2. Build the repository, and apply the patches to get the current state.
>
>    ```
>    [hxr@mk:/tmp/git-gat]$ git init
>    [hxr@mk:/tmp/git-gat]$ git am -3 *.patch
>    Applying: admin/ansible-galaxy/0000: Add requirements
>    applying to an empty history
>    Applying: admin/ansible-galaxy/0001: Add ansible.cfg
>    Applying: admin/ansible-galaxy/0002: Add hosts
>    Applying: admin/ansible-galaxy/0003: Add initial group variables file
>    Applying: admin/ansible-galaxy/0004: Add initial galaxy playbook
>    Applying: admin/ansible-galaxy/0005: Add pip, miniconda, galaxy to playbook
>    Applying: admin/ansible-galaxy/0006: Configure miniconda and galaxy
>    Applying: admin/ansible-galaxy/0007: Configure galaxy config
>    [...]
>    [hxr@mk:/tmp/git-gat]$ rm *.patch
>    ```
>
> 3. Find the commit:
>
>    ```
>    $ git log --oneline | grep "Add production facing vars"
>    2f5b416 admin/ansible-galaxy/0022: Add production facing vars
>    ```
> 4. Edit rebase one upstream of that commit
>
>    ```
>    $ git rebase -i 2f5b416~
>    ```
>
>    Mark it for editing
>
>    ```
>    edit 2f5b416 admin/ansible-galaxy/0022: Add production facing vars
>    pick a352bec admin/ansible-galaxy/0023: Add nginx x-accel-redir and g-i-g webhook config to nginx
>    pick 59f9ff0 admin/singularity/0000: Add golang and singulary ansible roles
>    pick f38ddbf admin/singularity/0001: Configure golang and singularity
>    ```
>
>    It should stop there
>
>    ```
>    Stopped at 2f5b416...  admin/ansible-galaxy/0022: Add production facing vars
>    You can amend the commit now, with
>
>      git commit --amend
>
>    Once you are satisfied with your changes, run
>
>      git rebase --continue
>    ```
>
> 5. Open your file, make your change. Here I'm showing the change I made with `git diff`
>
>    ```
>    $ git diff | cat
>    diff --git a/group_vars/galaxyservers.yml b/group_vars/galaxyservers.yml
>    index 15fbf1e..8fe4343 100644
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -53,6 +53,8 @@ galaxy_config:
>         expose_user_name: true
>         expose_dataset_path: true
>         expose_potentially_sensitive_job_metrics: true
>    +    # NFS workarounds
>    +    retry_job_output_collection: 3
>         # Debugging
>         cleanup_job: onsuccess
>         allow_user_impersonation: true
>    ```
>
>
> 6. The status probably looks like this, the file that was changed in that commit is marked as modified
>
>    ```
>    $ git status
>    interactive rebase in progress; onto 7284036
>    Last command done (1 command done):
>       edit 2f5b416 admin/ansible-galaxy/0022: Add production facing vars
>    Next commands to do (46 remaining commands):
>       pick 3c6a72e admin/ansible-galaxy/0023: Add nginx x-accel-redir and g-i-g webhook config to nginx
>       pick a893955 admin/singularity/0000: Add golang and singulary ansible roles
>      (use "git rebase --edit-todo" to view and edit)
>    You are currently editing a commit while rebasing branch 'main' on '7284036'.
>      (use "git commit --amend" to amend the current commit)
>      (use "git rebase --continue" once you are satisfied with your changes)
>
>    Changes to be committed:
>      (use "git restore --staged <file>..." to unstage)
>            modified:   group_vars/galaxyservers.yml
>    ```
>
> 7. Ammend the commit, and continue the rebase
>
>    ```
>    $ git commit --amend
>    [detached HEAD ed290de] admin/ansible-galaxy/0022: Add production facing vars
>     Author: The Galaxy Training Network <galaxytrainingnetwork@gmail.com>
>     Date: Mon Feb 15 14:06:56 2021 +0100
>     1 file changed, 22 insertions(+)
>    $ git rebase --continue
>    Successfully rebased and updated refs/heads/main.
>    ```
>
> 8. Import the changes back in the GTN repository
>
>    ```
>    ./bin/knit-automated.sh import
>    ```
>
{: .hands_on}


The above shows commands with outputs, the rest will be more abbreviated

## Upgrading an existing tutorial

> <question-title>Example problem</question-title>
> I have a new tutorial in the schedule and I want to upgrade it to use this feature
{: .question}

> <hands-on-title>Make the change you want to see in the GAT</hands-on-title>
>
> 1. Start by exporting
>
>    ```
>    $ bash bin/knit-automated.sh export
>    ```
>
> 2. Build the repository... **up to the point where your tuto is**
>
>    ```
>    [hxr@mk:/tmp/git-gat]$ git init
>    [hxr@mk:/tmp/git-gat]$ git am -3 0* 1* 2*
>    Applying: admin/ansible-galaxy/0000: Add requirements
>    applying to an empty history
>    Applying: admin/ansible-galaxy/0001: Add ansible.cfg
>    Applying: admin/ansible-galaxy/0002: Add hosts
>    Applying: admin/ansible-galaxy/0003: Add initial group variables file
>    Applying: admin/ansible-galaxy/0004: Add initial galaxy playbook
>    Applying: admin/ansible-galaxy/0005: Add pip, miniconda, galaxy to playbook
>    Applying: admin/ansible-galaxy/0006: Configure miniconda and galaxy
>    Applying: admin/ansible-galaxy/0007: Configure galaxy config
>    [...]
>    ```
>
> 3. Go diff-by-diff in your tutorial, making the change, and then making a new commit on top for each diff. For each of these, add the `{: data-commit="Something usful"}` below the commit.
>
>    Do they have a `{ % endraw % }`? then move that above the backticks.
>
> 4. Format them as patches
>
>    ```
>    git format-patch <the last commit before you started committing>
>    ```
>
> 5. Knit them together
>
>    ```
>    python bin/knit.py topics/admin/tutorials/<yourtuto>/tutorial.md --patches /tmp/<the-patches-you-exported>*
>    ```
>
> 6. That's part one done! Now we need to finish the rest of the patches.
>
> 7. Update `./bin/knit-automated.sh` to include your tutorial in the right order.
>
> 8. Go through an export-import cycle
>
>    ```
>    $ rm -rf /tmp/git-gat/
>    $ ./bin/knit-automated.sh export
>    $ cd /tmp/git-gat/
>    $ git am -3 *.patch # Resolve any conflicts
>    $ cd -
>    $ ./bin/knit-automated.sh import
>    ```
>
> 9. Done!
{: .hands_on}

## I want to add a new commit

> <hands-on-title>Make the change you want to see in the GAT</hands-on-title>
>
> 1. Add your commit where it belongs in the tutorial
>
>    <pre>
>    {% raw %}
>    { raw }
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -22,3 +22,7 @@
>       version: 0.0.2
>     - src: galaxyproject.slurm
>       version: 0.1.3
>    +- name: usegalaxy_eu.rabbitmq
>    +  version: 0.1.0
>    +- src: galaxyproject.pulsar
>    +  version: 1.0.7
>    { endraw }
>    ```
>    {: data-commit="Add requirements"}
>    {% endraw %}
>    </pre>
>
>    Use your imagination to pretend the raw/endraw tags are correct with the missing percent signs.
>
> 2. Now in the above diff we've written some --- and +++ and more metadata we don't really have. So: just make it up, the commit will fail to apply, and we'll go fix it. You can really write *anything* here you want, and the diff will fail which is what we want.
>
> 3. Now, go apply the commits
>
>    ```
>    $ ./bin/knit-automated.sh export
>    $ cd /tmp/git-gat/
>    $ git am -3 *.patch # Resolve any conflicts
>    ```
>
> 4. This will fail. Here's how to fix it:
>
>    1. Run `git am --show-current-patch` to see the above diff you've inserted (or the otherwise incorrect diff that's a downstream result of your new change).
>    2. Make the correct change to the file on disk.
>    3. `git add <path>`
>    3. `git am --continue`
>
> 5. You'll probably have to do this quite a few more times, diffs have a habit of affecting downstream diffs (e.g. changing a version number or being included in the diff context.)
>
> 6. Once you've finished all of them, you'll just `./bin/knit-automated.sh import` and commit the resulting changes!
{: .hands_on}

The alternative to the above approach is

1. Export
2. git rebase and make the corresponding change in the history
3. Let the rebase finish
4. Insert a 'dummy' diff (raw/open code+diff/endraw/close code/data-commit tag) without any content and then run `import` which should insert it in the correct place.

# Errors that can occur and how to fix them


## sha1 information is lacking or useless

This is because there's a mismatch in the commits you're trying to apply and what the merge commit is seeing. Usually you won't notice. **fix by rebasing, and roundtripping the commits.**

```
Applying: admin/pulsar/0012: Send bwa and bwa-mem to pulsar
Tagging step-17
Committing 18-gxadmin-commit-0000-add-requirement.patch
18-gxadmin-commit-0002-add-the-gxadmin-role.patch
Applying: admin/gxadmin/0000: Add requirement
error: sha1 information is lacking or useless (requirements.yml).
error: could not build fake ancestor
Patch failed at 0001 admin/gxadmin/0000: Add requirement
hint: Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
```

Here you grumble at how picky it is.

```
$ git am --show-current-patch
From: The Galaxy Training Network <galaxytrainingnetwork@gmail.com>
Date: Mon, 15 Feb 2021 14:06:56 +0100
Subject: admin/gxadmin/0000: Add requirement


--- a/requirements.yml
+++ b/requirements.yml
@@ -26,3 +26,5 @@
   version: 0.1.0
 - src: galaxyproject.pulsar
   version: 1.0.6
+- src: galaxyproject.gxadmin
+  version: 0.0.8
--
2.25.1
```

That looks right, doesn't it? But if we grep we'll see

```
$ ag -Q '1.0.6' topics/admin
topics/admin/tutorials/gxadmin/tutorial.md
55:>       version: 1.0.6

topics/admin/tutorials/monitoring/tutorial.md
80:>       version: 1.0.6

topics/admin/tutorials/pulsar/tutorial.md
149:>    +  version: 1.0.6
```

again right, but is it? No. Github is testing the merge commit. Rebase your commits, and now you'll see:


```
$ ag -Q '1.0.6' topics/admin
topics/admin/tutorials/gxadmin/tutorial.md
55:>       version: 1.0.6

topics/admin/tutorials/monitoring/tutorial.md
80:>       version: 1.0.6

$ ag -Q '1.0.7' topics/admin
topics/admin/tutorials/pulsar/tutorial.md
149:>    +  version: 1.0.7
```

Aha, two different version. This is what breaks the hashes (hence the error message.)

Roundtripping the commits and fixing up everywhere there was a single character difference in the diff context (UGH) will fix it.


topic: contributing
tutorial: meta-analysis-data
title: Single Cell Publication - Data Analysis
difficulty: advanced
time_estimation: 1H
tools_json: []
requirements_json: ["[]"]
tags_json: ["Ruby"]
content_hash: 840ed36fe0fe17feace2e80923f5a6eb
last_modified: 2026-05-15T22:28:10.846290
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/meta-analysis-data/tutorial.html

content:
Extracting data from the GTN's Git history isn't that difficult, but it requires some internal knowledge of how the GTN's Jekyll-based codebase works. Here we'll document what we've done!

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Our imports and metadata (all merged github PRs), and the list of all historical names of single-cell tutorial folders.

The author recommends running this code in a 'Jekyll Console' context. Jekyll does not natively have support for a console, but [there is an open Pull Request to add it](https://github.com/jekyll/jekyll/pull/8164). We recommend you install this yourself to most easily run the following code. You can do that by:

> <hands-on-title>Installing Jekyll's Console</hands-on-title>
> 1. View [the open Pull Request to add it](https://github.com/jekyll/jekyll/pull/8164)
> 2. Download `lib/jekyll/commands/console.rb` to somewhere on your computer.
> 3. Find out where jekyll is installed:
>    
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > gem which jekyll
>    > ```
>    {: .code-in}
>    
>    > <code-out-title></code-out-title>
>    > ```bash
>    > /home/user/galaxy/training-material/.direnv/ruby/gems/jekyll-4.3.3/lib/jekyll.rb
>    > ```
>    {: .code-out}
>    
>    That means the commands should be in:
>    
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ls /home/user/galaxy/training-material/.direnv/ruby/gems/jekyll-4.3.3/lib/jekyll/commands/
>    > ```
>    {: .code-in}
>    
>    > <code-out-title></code-out-title>
>    > ```bash
>    > build.rb  clean.rb  doctor.rb  help.rb  new.rb  new_theme.rb  serve  serve.rb
>    > ```
>    {: .code-out}
> 
> 4. Copy the `console.rb` you downloaded to that folder
> 
>     ```bash
>     cp ~/Downloads/console.rb $(dirname $(gem which jekyll))/jekyll/commands/
>     ```
> 
>     This was written with commands in case folks want to copy paste it, to reduce error.
>
> 5. Launch the console with 
>    
>    ```bash
>    jekyll console
>    ```
> 
{: .hands_on}

## Which PRs are Single Cell RPs?

```ruby
require 'yaml'
data = YAML.load_file('metadata/github.yml')
```

<!-- GTN:IGNORE:003 -->

Let's fetch data from within the GTN's infrastructure. You can see documentation for some of our APIs in the [RDoc](https://training.galaxyproject.org/training-material/gtn_rdoc/). E.g. here is how we document [`TopicFilter.list_materials_structured`](https://training.galaxyproject.org/training-material/gtn_rdoc/TopicFilter.html#method-c-list_materials_structured).

```ruby
# Obtain all single cell materials
mats = TopicFilter
 .list_materials_structured(site, 'single-cell')
 .map { |k, v| v['materials'] }
 .flatten
 .uniq { |x| x['id'] }

# And flatten them into a useful list of old folder names
@sc = mats.map{|x| x['ref_tutorials'].map{|t| t['redirect_from']} + x['ref_slides'].map{|t| t['redirect_from']}}
    .flatten.uniq
    .reject{|x| x=~ /\/short\//} # /short/ is a folder of redirects
    .map{|x| x.split('/')[1..-2].join('/')} # Remove the filename.

```

> <tip-title>Jekyll console only needed for this one command</tip-title>
> The `site` object is currently required for calculating this list of structured materials. This is only available in the Jekyll console, so you'll need to run this command in a Jekyll console.
> All of the subsequent steps can be run in a normal Ruby environment, but you might as well keep running it in the Jekyll console anyway.
{: .tip}

Let's go ahead and patch array to let us calculate a mean, because laziness is great actually

```ruby
class Array
  def mean
    return self.sum / (0.0 + self.length)
  end
end
```

Here's how we'll define what is or isn't a single cell tutorial, based on URL:

```ruby
def is_sc(path)
  p = path.gsub(/-ES/, '_ES').gsub(/-CAT/, '_CAT'); 

  if p =~ /\/single-cell\// then
    return 1
  end

  # If anything in @sc is a prefix for p, then it's a single-cell file.
  if @sc.any?{|sc| p.start_with?(sc)} then
    return 1
  end
  
  return 0
end
```

Now let's obtain everything that IS a single cell PR. Here we define everything with 50% or more of the files being "single cell" files, as a single cell PR. How did we arrive at 50%? We made some plots and spot-checked individual results to see what made sense.

```ruby
sc_prs = data
  .reject{|num, pr| pr['author']['login'] == 'github-actions'} # Remove all automation
  .map{|num, pr| 
    [
      num,
      pr['files']
        .reject{|f| f['path'].split('/')[2] == 'images' && f['path'] !~ /scrna/ }
        .reject{|f| f['path'] =~ /^assets/}
        .map{|f| is_sc(f['path'])}
      .mean
    ]
  }
  .reject{|num, sc| sc < 0.5}
  # Reject NaN
  .select{|num, sc| sc == sc}
```

Output number 1 done!

```ruby
File.open('dist.txt', 'w') do |f|
  f.puts "num\tdist"
  sc_prs.each do |num, sc|
    f.puts "#{num}\t#{sc}"
  end
end
```

## Classifying PR Content

Let's write a classifier for each file type, to enhance our statistics:

```ruby
def classify(path)
  if path =~ /tutorial[A-Z_]*?\.md/ then
    return 'tutorial'
  elsif path =~ /slides.*\.html/ then
    return 'slides'
  elsif path =~ /faqs.*md/ || path =~ /^snippets/ then
    return 'faq'
  elsif path =~ /metadata.yaml/ || path == 'CONTRIBUTORS.yaml' || path =~ /index.md$/ || path =~ /README.md$/ then
    return 'metadata'
  elsif path =~ /\/workflows\// then
    return 'workflows'
  elsif path =~ /data-(library|manager)/ then
    return 'data-library'
  elsif path =~ /.bib$/ then
    return 'bibliography'
  elsif path =~ /\/images\// then
    return 'image'
  elsif path =~ /tutorials\/.*md/ then
    return 'tutorial'
  elsif path =~ /_plugins/ || path =~ /^bin/ || path =~ /_layouts/ || path =~ /_include/ || path == '_config.yml' || path =~ /assets/ || path =~ /Gemfile/ || path =~/shared/ then
    return 'framework'
  elsif path =~ /metadata\/.*.yaml/ then
    return 'metadata'
  elsif path =~ /metadata\/.*.csv/ || path =~ /Dockerfile/ then
    return 'ignore'
  elsif path =~ /^news/ then
    return 'news'
  end

  # This will raise an exception which will ensure we catch the case where we
  # haven't defined a classification rule for a file yet.
  1/ 0
end
```

And now we'll classify each of the single cell pull requests by their file type:

```ruby
results = []
results << [
  "num", "path", "class", "additions", "deletions", "createdAt", "mergedAt"
]
sc_prs.each do |num, _|
  data[num]['files'].reject{|f| f['path'] =~ /test-data/}.each do |f|
    results << [
      num, f['path'], 
      classify(f['path']),
      f['additions'], f['deletions'],
      data[num]['createdAt'], data[num]['mergedAt']
    ]
  end
end
```

Output number 2!

```ruby
# save to file.csv
File.open('sc.tsv', 'w') do |f|
  results.each do |r|
    f.puts r.join("\t")
  end
end
```

Preview of that data:

num | path | class | additions | deletions | createdAt | mergedAt
--- | ---- | ----- | --------- | --------- | --------- | --------
5484 | topics/single-cell/faqs/single_cell_omics.md | faq | 3 | 3 | 2024-10-29T12:24:51Z | 2024-10-29T12:47:23Z
5473 | topics/single-cell/tutorials/alevin-commandline/tutorial.md | tutorial | 48 | 47 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case-jupyter_basic-pipeline/tutorial.md | tutorial | 3 | 2 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_FilterPlotandExplore_SeuratTools/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md | tutorial | 6 | 5 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md | tutorial | 4 | 4 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_alevin/tutorial.md | tutorial | 3 | 2 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_monocle3-rstudio/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_trajectories/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5447 | topics/single-cell/tutorials/GO-enrichment/tutorial.md | tutorial | 7 | 3 | 2024-10-11T11:22:34Z | 2024-10-11T13:36:22Z
5445 | topics/single-cell/tutorials/scrna-case-cell-annotation/slides.html | slides | 1 | 1 | 2024-10-11T10:52:08Z | 2024-10-12T15:44:00Z
5443 | topics/single-cell/faqs/single_cell_omics.md | faq | 4 | 2 | 2024-10-11T10:16:18Z | 2024-10-12T15:43:24Z
5416 | topics/single-cell/metadata.yaml | metadata | 4 | 2 | 2024-10-07T22:26:36Z | 2024-10-12T15:35:11Z
5416 | topics/single-cell/tutorials/GO-enrichment/tutorial.md | tutorial | 1 | 0 | 2024-10-07T22:26:36Z | 2024-10-12T15:35:11Z

## Contributors over time

Additionally we want to figure out how our contributions and contributors changed over time.

```ruby
require 'yaml'
require 'date'
require 'pp'
```

Here are the current classifications of contributors:

```ruby
KEYS = %w[authorship editing testing ux infrastructure translation data]
```

Let's get all the single cell tutorials:

```ruby
tutorials = Dir.glob("topics/single-cell/tutorials/*/tutorial.md")
```

We'll want to get all the timepoints from 2019 (when single cell was added) to
2025, so we'll setup an empty data structure for this.

**NB** This is NOT the best solution, this is just a simple brute force solution because the runtime is fine actually.

```ruby
timepoints = []
(2019..2025).each do |year|
  (1..12).each do |month|
    s = "#{year}-#{month}-01T00:00:00Z"
    timepoints << [
      s,
      DateTime.parse(s).to_time.to_i
    ]
  end
end
contribs_over_time = timepoints.map{|n, t| [t, KEYS.map{|k| [k, []]}.to_h]}.to_h
```

Ok, let's get the history of each tutorial

```ruby
tutorials.each do |tutorial|
  # if tutorial !~ /bulk/
  #   next
  # end

  git=`git log --follow --name-only --format="GTN_GTN %H %at" #{tutorial}`
  commits = git.split("GTN_GTN ")
  commits.reject!{|c| c.empty?}
  commits.map!{|c| 
    res = c.gsub(/\n+/, "\t").split(/\t/)
    if res.size > 2
      puts "ERROR: #{res}"
    end

    hash = res[0].split(' ')[0]
    time = res[0].split(' ')[1].to_i

    f = res[1]
    contents_at_time = `git show #{hash}:#{f}`
    begin
      contents_meta = YAML.load(contents_at_time)
    rescue
      next
    end

    if contents_meta.nil?
      next
    end

    if contents_meta.key?("contributors")
      c = {
        'authorship' => contents_meta["contributors"],
      }
    else
      c = contents_meta["contributions"]
    end

    squashed_i = DateTime.parse(Time.at(time).strftime("%Y-%m-01T00:00:00Z")).to_time.to_i

    {
      :hash => hash,
      :time => time,
      :date => Time.at(time),
      :sqsh => squashed_i, # The time rounded to the month
      :path => res[1],
      :role => c
    }
  }

  # For every commit
  commits.reverse.compact.each do |c|
    KEYS.each do |k|
      # For every role
      if c[:role].key?(k)
        # add to contribs now and at every time point in the future
        now_and_future_keys = contribs_over_time.keys.select{|t| t >= c[:sqsh] }
        now_and_future_keys.each do |t|
          contribs_over_time[t][k] << c[:role][k]
          contribs_over_time[t][k].flatten!
          contribs_over_time[t][k].uniq!
        end
      end
    end
  end
end
```

See, terrible, but it works!

```ruby
contribs_over_time.reject!{|k, v| v.values.all?{|vv| vv.empty?}}
pp contribs_over_time

File.open("sc-roles.tsv", "w") do |f|
  f.write("date\tarea\tcount\tcontributors\n")

  KEYS.each do |k|
    f.write(contribs_over_time.map{|date, roles| [date, roles[k]]}.map{|date, contribs| "#{Time.at(date).strftime("%Y-%m-01")}\t#{k}\t#{contribs.count}\t#{contribs.join(',')}"}.join("\n"))
  end
end
```

## All contributions over time

The plot of all new items over time was requested, and since the git log approach is so deeply ugly and slow, let's do something slightly different.

The date that Single Cell Tutorials, FAQ, Video, News, Events were added is requested.

The GTN produces RSS [feeds of all new content]({% link feeds/index.md %}), as well as feeds subsetted by their area (e.g. the [single cell feed](https://training.galaxyproject.org/training-material/feeds/single-cell-all.i.w.html) or the [single cell grouped by month](https://training.galaxyproject.org/training-material/feeds/single-cell-month.w.html)) so we can use that to get a list of all new single cell content over time.

```ruby
require 'net/http'
require 'date'
require './_plugins/util'
require 'uri'

# Ruby's stdlib for doing web requests is ... not great.
def request(url)
  uri = URI.parse(url)
  request = Net::HTTP::Get.new(uri)
  req_options = {
    use_ssl: uri.scheme == 'https',
  }
  Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
    http.request(request)
  end
end
```

Let's look though the data we get back:

```ruby
feed = request('https://training.galaxyproject.org/training-material/feeds/single-cell-month.xml').body
# The body of the feed has structure but it's slightly inconvenient.
# The URLs however...
# <a href="https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-preprocessing/workflows/scrna_mp_celseq.html?utm_source=matrix&amp;utm_medium=newsbot&amp;utm_campaign=matrix-news">CelSeq2: Multi Batch (mm10) (February 22, 2019)</a>
#
# That has everything and a date, can just figure out what 'type' of thing it was by the folder/name.

def classify(url)
  if url =~ /tutorial(_[A-Z_]*)?\.html/
    return 'tutorial'
  elsif url.include? '/events/'
    return 'event'
  elsif url =~ /slides(_[A-Z_]*)?\.html/
    return 'slide'
  elsif url.include? '/faqs/'
    return 'faq'
  elsif url.include? '/workflows/'
    return 'workflow'
  elsif url.include? '/news/'
    return 'news'
  end

  p url

  1/0
end

out = []

feed.scan(/<a href="([^"]+)">([^<]+)<\/a>/).each do |url, title|
  next if url.include? 'gtn-standards-rss.html'
  date = title.match(/\(([^)]+)\)$/)[1]
  date = Date.parse(date).to_s
  out << [date, classify(url)]
end
```

And that's almost done! Let's add in the video data as well:

```ruby
# Let's add in videos, they're stored in the tutorials. Thankfully these have a
# recording date we can use.
tutos = Dir.glob("topics/single-cell/tutorials/*/tutorial.md")
tutos.each do |tuto|
  meta = safe_load_yaml(tuto)
  next if meta['recordings'].nil? || meta['recordings'].empty?

  meta['recordings'].each do |rec|
    out << [rec['date'], 'video']
  end
end
```

and write it out:

```ruby
File.open("single-cell-over-time.tsv", "w") do |f|
  f.puts "date\ttype"
  out.sort_by{|d, t| d}.each do |date, type|
    f.puts "#{date}\t#{type}"
  end
end
```

With that, I think we're done! Ready to plot.

> <tip-title>Do you want to re-do this analysis? For your community?</tip-title>
> [Let us know](https://github.com/galaxyproject/training-material/issues/new?title=GTN%20Metadata%20reuse) and we can perhaps generalise this dataset such that you could more easily download it for your own community.
{: .tip}


topic: contributing
tutorial: create-new-tutorial-technical
title: Tools, Data, and Workflows for tutorials
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 2e492bf7cda91b1e9e4fb841314d1f95
last_modified: 2026-05-27T09:01:47.918390
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-technical/tutorial.html

content:
# Building a Galaxy instance specifically for your training


To be able to run the tutorial, we need a Galaxy instance where all of the needed tools and data are available. Thus we need to describe the needed technical infrastructure.

The files we define in this tutorial will be used to automatically build a Docker Galaxy flavour, and also to test if a public Galaxy instance is able to run the tool.

In this tutorial, you will learn how to create a virtualized Galaxy instance, based on Docker, to run your training - either on normal computers or cloud environments.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Extracting workflows

Once the tutorial is ready, we need to develop a workflow that represents the steps taken in the tutorial, and then extract these workflow(s) and add them to the `workflows` directory in the tutorial. Additionally we will need to add some explanation about the workflow(s) in a `README.md` file

> <hands-on-title>Extract the workflow</hands-on-title>
>
> 1. Add the topic name as a Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor.
> 2. Download the workflow for the tutorial
> 3. Save it in the `workflow` directory of the tutorial
> 4. Check that your `workflow` directory has an `index.md` with the contents:
>
>    ```yaml
>    ---
>    layout: workflow-list
>    ---
>    ```
{: .hands_on}

## Testing the workflow (recommended)

Workflow testing is a great way to get feedback that your tutorial can be run successfully on a given server. When you're giving a training this can provide peace of mind, not only are the tools installed (as is indicated by the badges we provide) but they also work.

Given the workflow you created above and have included in the tutorial folder, you'll need to create a corresponding `-test.yml` file.

> <hands-on-title>Creating the `-test.yml` file for your workflow</hands-on-title>
>
> 1. Find the correct name for the file; if your workflow was `unicycler.ga`, then your test file should be `unicycler-test.yml`, they need to share the same prefix.
>
> 2. Create the following structure:
>
>    ```yaml
>    ---
>    - doc: Test sample data for the workflow
>      job:
>        an_input_file:
>          class: File
>          location: https://....
>          filetype: fasta
>      outputs:
>        ffn:
>          asserts:
>            - that: has_text
>              text: ">A"
>            - that: has_text
>              text: ">B"
>    ```
>
>
{: .hands_on}

You'll need to edit the `job` and `outputs` sections according to your workflow's inputs and outputs. Additionally you will need to edit the steps of your workflow `.ga` file appropriately.

### Inputs

Your workflow **must** use "Data Inputs" for each input dataset. For each of these input steps in the `.ga` file, you'll need to do the following:

1. Edit the `label`
2. Edit the `name`
3. Edit the `inputs[0].name`
4. Edit the `tool_state`

In a normal workflow you have exported from Galaxy, you'll see something like

```json
{
    "id": 0,
    "input_connections": {},
    "inputs": [
        {
            "description": "",
            "name": "patient1_ChIP_ER_good_outcome.bam"
        }
    ],
    "label": null,
    "name": "Input dataset",
    "outputs": [],
    "position": {
        "left": 10,
        "top": 10
    },
    "tool_id": null,
    "tool_state": "{\"name\": \"patient1_ChIP_ER_good_outcome.bam\"}",
    "tool_version": null
}
```

You should synchronize the aforementioned fields so it looks like this:

```json
{
    "id": 0,
    "input_connections": {},
    "inputs": [
        {
            "description": "",
            "name": "good_outcome"
        }
    ],
    "label": "good_outcome",
    "name": "good_outcome",
    "outputs": [],
    "position": {
        "left": 10,
        "top": 10
    },
    "tool_id": null,
    "tool_state": "{\"name\": \"good_outcome\"}",
    "tool_version": null
}
```

This will allow you to specify `good_outcome` in your job to load a file:

```
- doc: ...
  job:
    good_outcome:
      class: File
      location: ...
      filetype: ...
```

The filetype should be the Galaxy datatype of your file, for example `fastqsanger`, `tabular`, `bam`.

### Outputs

For the outputs the process is somewhat simpler:

1. Identify a step, the outputs of which you would like to test
2. Convert the relevant `outputs` to `workflow_outputs`

   In a normal workflow you see

   ```json
   {
       "outputs": [
           {
               "type": "txt",
               "name": "ofile"
           },
           {
               "type": "txt",
               "name": "ofile2"
           }
       ],
       "workflow_outputs": []
   }
   ```

   If you want to test the contents of `ofile`, you should change it to

   ```json
   {
       "outputs": [
           {
               "type": "txt",
               "name": "ofile"
           },
           {
               "type": "txt",
               "name": "ofile2"
           }
       ],
       "workflow_outputs": [
           {"output_name": "ofile", "label": "my_output"}
       ]
   }
   ```

3. You can now use the label you chose (here `my_output`) in your test case:

   ```yaml
   - doc:
     job: ...
     outputs:
       my_output:
         asserts:
           has_text:
             text: 'some-string'
   ```

### Running the Tests

You can test the file you've written with the following command and a recent version (>=0.56.0) of planemo:

```console
planemo test \
	--galaxy_url "$GALAXY_URL" \
	--galaxy_user_key "$GALAXY_USER_KEY" \
	--no_shed_install \
	--engine external_galaxy \
	workflow.ga
```

Planemo will autodetect that the `workflow-test.yml` file and load that for the testing.

### Skipping Testing in the GTN

If for some reason you want to skip this workflow being tested in the GTN,
please add a comment with `GTN_RUN_SKIP_REASON` in the `-test.yml` file stating
the reason it is skipped.

This will also exempt you from writing output tests.

A good use case for this is you want to provide a working test, but the workflow takes upwards of 6 hours to execute (e.g. large download jobs.)

# Creating the `data-library.yaml` (recommended)

The datasets needed for a tutorial can also be integrated in the Galaxy instance inside of data libraries. These allow the datasets to be easily shared with all users of a Galaxy instance. Additionally it lets trainees avoid each re-downloading the input data.

These datasets are described in the `data-library.yaml` files:

```yaml
---
destination:
  type: library
  name: GTN - Material
  description: Galaxy Training Network Material
  synopsis: Galaxy Training Network Material. See https://training.galaxyproject.org
items:
- name: Title of the topic
  description: Summary of the topic
  items:
  - name: Title of the tutorial
    items:
    - name: 'DOI: 10.5281/zenodo....'
      description: latest
      items:
      - info: https://doi.org/10.5281/zenodo....
        url: https://zenodo.org/records/URL/files/path/to/input
        ext: galaxy-datatype
        src: url
```

> <hands-on-title>Creating the `data-library.yaml`</hands-on-title>
>
> 1. Copy the Zenodo link
> 2. Generate the `data-library.yaml` file and update the tutorial metadata with the link:
>
>    ```
>    $ planemo training_fill_data_library \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --zenodo_link "URL to the Zenodo record"
>    ```
>
> 3. Check that the `data-library.yaml` has been generated (or updated)
> 4. Check that the Zenodo link is in the metadata at the top of the `tutorial.md`
{: .hands_on}

# Creating the `data-manager.yaml` (optional)

Some of the tools may require specific databases, specifically prepared for the tool. In this case, some Galaxy tools come with "data managers" to simplify this process.

If you need such data managers for your training, then you should describe how to run them in the `data-manager.yaml` file:

```yaml
data_managers:
    - id: url to data manager on ToolShed
      params:
        - 'param1': '{{ item }}'
        - 'param2': 'value'
      # Items refer to a list of variables you want to run this data manager. You can use them inside the param field with {{ item }}
      # In case of genome for example you can run this DM with multiple genomes, or you could give multiple URLs.
      items:
        - item1
        - item2
      # Name of the data-tables you want to reload after your DM are finished. This can be important for subsequent data managers
      data_table_reload:
        - all_fasta
        - __dbkeys__
```

# Creating the Galaxy Interactive Tour (optional)

A Galaxy Interactive Tour is a way to go through an entire analysis, step by step inside Galaxy in an interactive and explorative way.
It is a great way to help users run the tutorial directly inside Galaxy. To learn more about creating a Galaxy tour please have a look at our [dedicated tour training]({{site.baseurl}}/topics/contributing/tutorials/create-new-tutorial-tours/tutorial.html).

# Testing the technical infrastructure

Once we have defined all the requirements for running the tutorial, we can test these requirements, either in a locally running Galaxy or in a Docker container. Please see our tutorial about [Setting up Galaxy for Training]({{site.baseurl}}/topics/teaching/tutorials/setup-galaxy-for-training/tutorial.html) about how to test your tutorial requirements.


# Conclusion


