# tutorials

topic: data-science
tutorial: r-advanced
title: Advanced R in Galaxy
difficulty: intermediate
time_estimation: 2H
tools_json: ["RStudio", "interactive_tool_rstudio"]
requirements_json: []
tags_json: ["R"]
content_hash: 8e524c51ea0c62122ff69b87d3d50b0c
last_modified: 2026-05-15T22:28:11.110921
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-advanced/tutorial.html

content:
With HTS-Seq data analysis, we generated tables containing list of DE genes, their expression, some statistics, etc. We can manipulate these tables using Galaxy, as we saw in some tutorials, e.g. ["Reference-based RNA-Seq data analysis"]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), and create some visualisations.

Sometimes we want to have some customizations on visualization, some complex table manipulations or some statistical analysis. If we can not find a Galaxy tools for that or the right parameters, we may need to use programming languages as R or Python.

<!-- Keeping this here because people may be familiar enough with R, that they go directly to this tutorial, and it is still useful information. -->

{% include topics/data-science/tutorials/r-basics/r_introduction.md %}
{% include topics/data-science/tutorials/r-basics/disclaimer.md %}

In this tutorial, we will take the list of DE genes extracted from DESEq2's output that we generated in the ["Reference-based RNA-Seq data analysis" tutorial]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), manipulate it and create some visualizations.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Get familiar with the annotated DE genes table in R

A substantial amount of the data we work with in science is tabular data, i.e. data arranged in rows and columns - also known as spreadsheets.

> <comment-title>Few principles when working with data</comment-title>
> We want to remind you of a few principles before we work with our first set of example data:
>
> 1. **Keep raw data separate from analyzed data**
>
>    This is principle number one, because if you can't tell which files are the original raw data, you risk making some serious mistakes (e.g. drawing conclusion from data which have been manipulated in some unknown way).
>
>    When you work with data in R, you are not changing the original file which you loaded the data from. This is different than (for example) working with a spreadsheet program where changing the value of the cell leaves you one "save"-click away from overwriting the original file. You have to purposely use a writing function (e.g. `write.csv()`) to save data loaded into R. In that case, be sure to save the manipulated data into a new file. More on this later in the lesson.
>
> 2. **Keep spreadsheet data Tidy**
>
>    The simplest principle of *Tidy* data is that we have one row in our spreadsheet for each observation or sample, and one column for every variable that we measure or report on. As simple as this sounds, it’s very easily violated. Most data scientists agree that significant amounts of their time is spent tidying data for analysis.
>
> 3. **Trust but verify**
>
>    Finally, while you don't need to be paranoid about data, you should have a plan for how you will prepare it for analysis. **This a focus of this lesson**. You probably already have a lot of intuition, expectations, assumptions about your data - the range of values you expect, how many values should have been recorded, etc. Of course, as the data get larger our human ability to keep track will start to fail (and yes, it can fail for small data sets too). R will help you to examine your data so that you can have greater confidence in your analysis, and its reproducibility.
>
{: .comment}

## Import tabular data into R

As in the "R Basics" tutorial, we will continue in {% tool [RStudio](interactive_tool_rstudio) %}.

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

There are several ways to import data into R. For our purpose here, we will focus on using the tools every R installation comes with (so called "base" R) to import a comma-delimited file containing the results of our variant calling workflow. We will need to load the sheet using a function called `read.csv()`.

> <question-title>Review the arguments of the `read.csv()` function</question-title>
>
> Before using the `read.csv()` function, use R's help feature to answer the following questions.
>
> > <tip-title>Getting help in R</tip-title>
> > Entering `?` before the function name and then running that line will bring up the help documentation. Also, when reading this particular help be careful to pay attention to the `read.csv` expression under the **Usage** heading. Other answers will be in the **Arguments** heading.
> {: .tip}
>
> 1. What is the default parameter for `header` in the `read.csv()` function?
> 2. What argument would you have to change to read a file that was delimited by semicolons (`;`) rather than commas?
> 3. What argument would you have to change to read file in which numbers used commas for decimal separation (i.e. 1,00)?
> 4. What argument would you have to change to read in only the first 10,000 rows of a very large file?
>
> > <solution-title></solution-title>
> >
> > 1. The `read.csv() `function has the argument `header` set to `TRUE` by default, this means the function always assumes the first row is header information, (i.e. column names)
> > 2. The `read.csv()` function has the argument 'sep' set to ",". This means the function assumes commas are used as delimiters, as you would expect. Changing this parameter (e.g. `sep=";"`) would now interpret semicolons as delimiters.
> > 3. Although it is not listed in the `read.csv()` usage, `read.csv()` is a "version" of the function `read.table()` and accepts all its arguments. If you set `dec=","` you could change the decimal operator. We'd probably assume the delimiter is some other character.
> > 4. You can set `nrow` to a numeric value (e.g. `nrow=10000`) to choose how many rows of a file you read in. This may be useful for very large files where not all the data is needed to test some data cleaning steps you are applying.
> >
> > Hopefully, this exercise gets you thinking about using the provided help documentation in R. There are many arguments that exist, but which we wont have time to cover. Look here to get familiar with functions you use frequently, you may be surprised at what you find they can do.
> >
> {: .solution}
{: .question}

Now, let's read the file with the annotated differentially expressed genes that was produced in the "[Reference-based RNA-Seq data analysis]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %})" tutorial.

<!--
> <hands-on-title>Import dataset into Galaxy</hands-on-title>
> 1. In Galaxy, upload the following file from Zenodo:
>    ```
>    https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular
>    ```
> 2. Remember the dataset number (e.g. `2` in the example below)
>    ![todo](../../images/rna-seq-counts-to-viz-in-r/dataset_number.png)
>
{: .hands_on}
-->

> <hands-on-title>Read the annotated differentially expressed genes</hands-on-title>
> 1. Create a new script (if needed)
> 2. Read the tabular file in an object called `annotatedDEgenes`. We can import directly by URL:
>
>    ```R
>    ## read in a CSV file and save it as 'annotatedDEgenes'
>    annotatedDEgenes <- read.csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular")
>    ```
>
>    The first argument to pass to our `read.csv()` function is the file path for our data. The file path must be in quotes and now is a good time to remember to use tab autocompletion. If you use tab autocompletion you avoid typos and errors in file paths. Use it!
>
> 3. Inspect the **Environment** panel
>
>  > <tip-title>Upload files from a Galaxy history to RStudio</tip-title>
>  > Rstudio in Galaxy provides some special functions to import and export from your history.
>  >
>  > ```R
>  > gx_get(2) # will import dataset number 2 from your history
>  > ```
>  {: .tip}
{: .hands_on}

In the **Environment** panel, you should have the `annotatedDEgenes` object, listed as 130 obs. (observations/rows) of 1 variable (column) - so the command worked (sort of)!

> <hands-on-title>Inspect the annotatedDEgenes file</hands-on-title>
> 1. Double-click on the name of the object in the **Environment** panel
>
{: .hands_on}

It will open a view of the data in a new tab on the top-left panel.

As you can see, there is a problem with how the data has been loaded. The table should contain 130 observations of 13 variables.

> <question-title>Identify the issue with data</question-title>
>
> 1. What is wrong?
> 2. How should you adjust the parameters for `read.csv()` in order to produce the intended output?
>
> > <solution-title></solution-title>
> >
> > 1. The data file was not delimited by commas (`,`) which is the default expected delimiter for `read.csv()`. Instead it seems like it's delimited by "white space", i.e. spaces and/or tabs.
> > 2. We need to set the correct delimiter using the parameter `sep`. Given that we are not sure what the actual delimiter is, we could try both options, i.e. `sep=" "` (*space*) and `sep="\t"` (*tab*).
> >
> {: .solution}
{: .question}

> <hands-on-title>Read correctly tabular file</hands-on-title>
> 1. Read the file given it's a tabular file
>
>    ```R
>    annotatedDEgenes <- read.csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t")
>    ```
>
> 2. Inspect the object in the **Environment** panel
> 3. Open the object
>
>    You should have now a nice table
>
> 3. Check the column names
>
>    ```R
>    > colnames(annotatedDEgenes)
>    [1] "GeneID"     "Base.mean"  "log2.FC."   "StdErr"     "Wald.Stats"
>    [6] "P.value"    "P.adj"      "Chromosome" "Start"      "End"
>    [11] "Strand"     "Feature"    "Gene.name"
>    ```
>
{: .hands_on}

Congratulations! You've successfully loaded your data into RStudio!

## Summarize and determine the structure of a data frame

Tabular data are stored in R using **data frame**. A data frame could also be thought of as a collection of vectors, all of which have the same length.

> <hands-on-title>Learn more about our data frame</hands-on-title>
> 1. Get summary statistics of `annotatedDEgenes` using `summary` function
>
>    ```R
>    ## get summary statistics on a data frame
>    > summary(annotatedDEgenes)
>    GeneID            Base.mean          log2.FC.          StdErr            Wald.Stats
>    FBgn0000071:  1   Min.   :   19.15   Min.   :-4.1485   Min.   :0.08433   Min.   :-30.741
>    FBgn0000079:  1   1st Qu.:  100.29   1st Qu.:-1.3363   1st Qu.:0.12849   1st Qu.:-10.005
>    FBgn0000116:  1   Median :  237.99   Median :-1.0272   Median :0.16370   Median : -4.982
>    FBgn0000406:  1   Mean   : 1911.27   Mean   :-0.2074   Mean   :0.16432   Mean   : -1.901
>    FBgn0000567:  1   3rd Qu.:  948.66   3rd Qu.: 1.2203   3rd Qu.:0.19866   3rd Qu.:  7.692
>    FBgn0001137:  1   Max.   :65114.84   Max.   : 2.6999   Max.   :0.23292   Max.   : 27.566
>    (Other)    :124
>    P.value             P.adj               Seqname    Start              End                Strand
>    Min.   :0.000e+00   Min.   :0.000e+00   chr2L:24   Min.   :  127448   Min.   :  140340   -:58
>    1st Qu.:0.000e+00   1st Qu.:0.000e+00   chr2R:31   1st Qu.: 7277516   1st Qu.: 7279063   +:72
>    Median :0.000e+00   Median :0.000e+00   chr3L:27   Median :13161546   Median :13166253
>    Mean   :4.201e-07   Mean   :9.321e-06   chr3R:32   Mean   :13436843   Mean   :13446444
>    3rd Qu.:7.000e-12   3rd Qu.:3.700e-10   chrX :16   3rd Qu.:19250429   3rd Qu.:19284637
>    Max.   :9.418e-06   Max.   :1.951e-04              Max.   :31196915   Max.   :31203722
>
>    Feature              Gene.Name
>    lincRNA       :  3   Ama        :  1
>    protein_coding:126   Amy-p      :  1
>    pseudogene    :  1   Ant2       :  1
>                         Argk       :  1
>                         BM-40-SPARC:  1
>                         bou        :  1
>                         (Other)    :124
>    ```
>
> 2. Get the structure of `annotatedDEgenes` using `str` function
>
>    ```R
>    ## get the structure of a data frame
>    > str(annotatedDEgenes)
>    'data.frame':	130 obs. of  13 variables:
>     $ GeneID    : Factor w/ 130 levels "FBgn0000071",..: 87 12 28 26 31 96 65 62 125 1 ...
>     $ Base.mean : num  1087 6410 65115 2192 5430 ...
>     $ log2.FC.  : num  -4.15 -3 -2.38 2.7 -2.11 ...
>     $ StdErr    : num  0.1349 0.1043 0.0843 0.0979 0.0925 ...
>     $ Wald.Stats: num  -30.7 -28.7 -28.2 27.6 -22.7 ...
>     $ P.value   : num  1.62e-207 9.42e-182 2.85e-175 2.85e-167 1.57e-114 ...
>     $ P.adj     : num  1.39e-203 4.04e-178 8.15e-172 6.10e-164 2.70e-111 ...
>     $ Seqname   : Factor w/ 5 levels "chr2L","chr2R",..: 4 5 4 5 3 4 2 2 3 4 ...
>     $ Start     : int  24141394 10780892 26869237 10778953 13846053 31196915 24945138 22550093 820758 >    6762592 ...
>     $ End       : int  24147490 10786958 26871995 10786907 13860001 31203722 24946636 22552113 821512 >    6765261 ...
>     $ Strand    : Factor w/ 2 levels "-","+": 2 1 1 1 2 2 2 2 2 2 ...
>     $ Feature   : Factor w/ 3 levels "lincRNA","protein_coding",..: 2 2 2 2 2 2 2 2 1 2 ...
>     $ Gene.Name : Factor w/ 130 levels "Ama","Amy-p",..: 88 113 5 3 84 26 41 53 69 1 ...
>    ```
{: .hands_on}

Ok, thats a lot to unpack! Some things to notice:

- the object type `data.frame` is displayed in the first row along with its dimensions, in this case 130 observations (rows) and 13 variables (columns)
- Each variable (column) has a name (e.g. `GeneID`). This is followed by the object mode (e.g. `factor`, `int`, `num`, etc.). Notice that before each variable name there is a `$` - this will be important later.

So from both `summary` and `str`, we know that our data frame had 13 variables that summarize the data.

- `Base.mean`, `log2.FC.` and `P.value` variables (and several others) are numerical data

    We get summary statistics on the min and max values for these columns, as well as mean, median, and interquartile ranges

- Many of the other variables (e.g. `Strand`) are treated as categorical data

    This type has special treatment in R - more on this in a bit. The most frequent 6 different categories and the number of times they appear (e.g. the `Strand` called `-` appeared 58 times) are displayed. Another example is the `protein_coding` value for `Feature` which appeared in 126 observations.

## Factors

Factors, the final major data structure we will introduce, can be thought of as vectors which are specialized for categorical data. Given R's specialization for statistics, this make sense since categorial and continuous variables usually have different treatments. Sometimes you may want to have data treated as a factor, but in other cases, this may be undesirable.

Since some of the data in our data frame are factors, lets see how factors work.

> <hands-on-title>Inspect a factor object</hands-on-title>
> 1. Extract the `feature` column to a new object called `feature`
>
>    ```R
>    ## extract the "feature" column to a new object
>    feature <- annotatedDEgenes$Feature
>    ```
>
> 2. Inspect the first few items in our  using `head` function
>
>    ```R
>    > head(feature)
>    [1] protein_coding protein_coding protein_coding protein_coding
>    [5] protein_coding protein_coding
>    Levels: lincRNA protein_coding pseudogene
>    ```
{: .hands_on}

What we get back are the items in our factor, and also something called "Levels". **Levels are the different categories contained in a factor**. By default, R will organize the levels in a factor in alphabetical order. So the first level in this factor is "lincRNA".

> <hands-on-title>Get the structure of a factor object</hands-on-title>
> 1. Get the structure of the `feature` object
>
>    ```R
>    > str(feature)
>    Factor w/ 3 levels "lincRNA","protein_coding",..: 2 2 2 2 2 2 2 2 1 2 ...
>    ```
{: .hands_on}

For the sake of efficiency, R stores the content of a factor as a vector of integers, which an integer is assigned to each of the possible levels. Recall levels are assigned in alphabetical order. In this case, the first item in our "Feature" object is "protein_coding", which happens to be the 2nd level of our factor, ordered alphabeticaly. The 9th item in the list is "lincRNA", which is the 1st level of our factor.

One of the most common uses for factors is to plot categorical values. For example, suppose we want to know how many of our differentially expressed genes correspond to different features.

> <hands-on-title>Plot a factor object</hands-on-title>
> 1. Plot the number of levels in the `feature` object using `plot` function
>
>    ```R
>    plot(feature)
>    ```
{: .hands_on}

![feature plot](../../../introduction/images/rna-seq-counts-to-viz-in-r/featurePlot.png)

This isn't a particularly pretty example of a plot. We'll be learning much more about creating nice, publication-quality graphics later.

## Subset a data frame

Now we would like to know how to get specific values from data frames, and where necessary, change the mode of a column of values.

The first thing to remember is that a data frame is two-dimensional (rows and columns). Therefore, to select a specific value we will once again use `[]` (bracket) notation, but we will specify more than one value (except in some cases where we are taking a range).

> <question-title>Subsetting a data frame</question-title>
>
> Run the following indices and functions to figure out what they return
>
> 1. `annotatedDEgenes[1, 1]`
> 2. `annotatedDEgenes[2, 4]`
> 3. `annotatedDEgenes[130, 13]`
> 4. `annotatedDEgenes[2, ]`
> 5. `annotatedDEgenes[-1, ]`
> 6. `annotatedDEgenes[1:4, 1]`
> 7. `annotatedDEgenes[1:10, c("Feature", "Gene.Name")]`
> 8. `annotatedDEgenes[, c("Gene.Name")]`
> 9. `head(annotatedDEgenes)`
> 10. `tail(annotatedDEgenes)`
> 11. `annotatedDEgenes$GeneID`
> 12. `annotatedDEgenes[annotatedDEgenes$Feature == "pseudogene", ]`
>
> > <solution-title></solution-title>
> >
> > 1. `annotatedDEgenes[1, 1]`
> >
> >     ```R
> >     > annotatedDEgenes[1,1]
> >     [1] FBgn0039155
> >     130 Levels: FBgn0000071 FBgn0000079 FBgn0000116 FBgn0000406 FBgn0000567 FBgn0001137 ... FBgn0267635
> >     ```
> >
> > 2. `annotatedDEgenes[2, 4]`
> >
> >     ```R
> >     > annotatedDEgenes[2, 4]
> >     [1] 0.1043451
> >     ```
> >
> > 3. `annotatedDEgenes[130, 13]`
> >
> >     ```R
> >     > annotatedDEgenes[130,13]
> >     [1] CG10440
> >     130 Levels: Ama Amy-p Ant2 Argk BM-40-SPARC bou bves CAHbeta cbt CG10063 CG10365 CG10440 ... zye
> >     ```
> >
> > 4. `annotatedDEgenes[2, ]`
> >
> >     ```R
> >     > annotatedDEgenes[2, ]
> >            GeneID Base.mean   log2.FC.   StdErr Wald.Stats       P.value         P.adj Seqname    Start
> >     2 FBgn0003360  6409.577 -2.999777 0.1043451  -28.74863 9.422382e-182 4.040788e-178    chrX 10780892
> >            End Strand        Feature Gene.Name
> >     2 10786958      - protein_coding      sesB
> >     ```
> >
> > 5. `annotatedDEgenes[-1, ]`
> >
> >     ```R
> >     > annotatedDEgenes[-1, ]
> >             GeneID   Base.mean  log2.FC.     StdErr Wald.Stats       P.value         P.adj Seqname
> >     2  FBgn0003360  6409.57713 -2.999777 0.10434506 -28.748628 9.422382e-182 4.040788e-178    chrX
> >     3  FBgn0026562 65114.84056 -2.380164 0.08432692 -28.225437 2.850473e-175 8.149503e-172   chr3R
> >     4  FBgn0025111  2192.32237  2.699939 0.09794457  27.565988 2.846764e-167 6.104174e-164    chrX
> >     5  FBgn0029167  5430.06728 -2.105062 0.09254660 -22.745964 1.573283e-114 2.698810e-111   chr3L
> >     6  FBgn0039827   390.90178 -3.503014 0.16002962 -21.889786 3.250384e-106 4.646424e-103   chr3R
> >     7  FBgn0035085   928.26381 -2.414074 0.11518516 -20.958204  1.579343e-97  1.935147e-94   chr2R
> >     8  FBgn0034736   330.38302 -3.018179 0.15815418 -19.083774  3.444661e-81  3.693107e-78   chr2R
> >     9  FBgn0264475   955.45445 -2.334486 0.12423003 -18.791643  8.840041e-79  8.424559e-76   chr3L
> >     10 FBgn0000071   468.05793  2.360017 0.13564397  17.398615  8.452137e-68  7.249398e-65   chr3R
> >     # ... with 120 more rows
> >
> >           Start      End Strand        Feature   Gene.Name
> >     2  10780892 10786958      - protein_coding        sesB
> >     3  26869237 26871995      - protein_coding BM-40-SPARC
> >     4  10778953 10786907      - protein_coding        Ant2
> >     5  13846053 13860001      + protein_coding         Hml
> >     6  31196915 31203722      + protein_coding      CG1544
> >     7  24945138 24946636      + protein_coding      CG3770
> >     8  22550093 22552113      + protein_coding      CG6018
> >     9    820758   821512      +        lincRNA     CR43883
> >     10  6762592  6765261      + protein_coding         Ama
> >     # ... with 120 more rows
> >     ```
> >
> > 6. `annotatedDEgenes[1:4,1]`
> >
> >     ```R
> >     > annotatedDEgenes[1:4,1]
> >     [1] FBgn0039155 FBgn0003360 FBgn0026562 FBgn0025111
> >     130 Levels: FBgn0000071 FBgn0000079 FBgn0000116 FBgn0000406 FBgn0000567 FBgn0001137 ... FBgn0267635
> >     ```
> >
> > 7. annotatedDEgenes[1:10,c("Feature","Gene.Name")]
> >
> >     ```R
> >     > annotatedDEgenes[1:10,c("Feature","Gene.Name")]
> >               Feature   Gene.Name
> >     1  protein_coding        Kal1
> >     2  protein_coding        sesB
> >     3  protein_coding BM-40-SPARC
> >     4  protein_coding        Ant2
> >     5  protein_coding         Hml
> >     6  protein_coding      CG1544
> >     7  protein_coding      CG3770
> >     8  protein_coding      CG6018
> >     9         lincRNA     CR43883
> >     10 protein_coding         Ama
> >     ```
> >
> > 8. `annotatedDEgenes[,c("Gene.Name")]`
> >
> >     ```R
> >     > annotatedDEgenes[,c("Gene.Name")]
> >     [1] Kal1         sesB         BM-40-SPARC  Ant2         Hml          CG1544       CG3770
> >     [8] CG6018       CR43883      Ama          CG3168       CG15695      CG8500       Rgk1
> >     [15] Sesn         CG9119       Sema-2a      l(1)G0196    lectin-28C   Sox100B      Hsp27
> >     [22] CG34330      Ugt58Fa      Src64B       Tequila      CG32407      LpR2         CG12290
> >     [29] ps           CG8157       nemy         bves         CG8468       CR43855      CG13315
> >     [36] CG1124       Reg-2        CG5261       Treh         Hsp23        Gale         PPO1
> >     [43] Picot        RpS24        CG14946      CG10365      NimC4        CG42694      CG33307
> >     [50] CG6006       grk          CG42369      GILT1        CG5001       Hsp26        MtnA
> >     [57] CG6330       l(3)neo38    Cyt-b5-r     CG12116      PPO2         zye          tnc
> >     [64] CG14257      GstE2        olf413       CG31642      Nxf3         CG8292       CG5397
> >     [71] Mctp         CG7777       CG4019       ChLD3        Argk         bou          Mct1
> >     [78] CG9416       CG12512      CG32625      Npc2b        CG30463      Eip74EF      CAHbeta
> >     [85] CG14326      Cyp6w1       CG3457       wdp          cbt          CrebA        Cyp4d20
> >     [92] CR45973      Sr-CIV       SPH93        Cyp9h1       sug          CG14516      Ggamma30A
> >     [99] CG4297       CG9503       CG10063      CG8501       CG43336      stet         CG8852
> >     [106] CG15080      Amy-p        sn           CG9989       mlt          CG2444       CG17189
> >     [113] RhoGAP1A     CG13743      TwdlU        Galphaf      CG32335      E(spl)m3-HLH CG16749
> >     [120] CG34434      CG43799      Pde6         Sr-CIII      NimC2        CG5895       SP1173
> >     [127] CG1311       CG14856      CG6356       CG10440
> >     130 Levels: Ama Amy-p Ant2 Argk BM-40-SPARC bou bves CAHbeta cbt CG10063 CG10365 CG10440 ... zye
> >     ```
> >
> > 9. `head(annotatedDEgenes)`
> >
> >     ```R
> >     > head(annotatedDEgenes)
> >            GeneID  Base.mean  log2.FC.      StdErr Wald.Stats       P.value         P.adj Seqname    Start
> >     1 FBgn0039155  1086.9743 -4.148450 0.13494887  -30.74090 1.617921e-207 1.387691e-203   chr3R 24141394
> >     2 FBgn0003360  6409.5771 -2.999777 0.10434506  -28.74863 9.422382e-182 4.040788e-178    chrX 10780892
> >     3 FBgn0026562 65114.8406 -2.380164 0.08432692  -28.22544 2.850473e-175 8.149503e-172   chr3R 26869237
> >     4 FBgn0025111  2192.3224  2.699939 0.09794457   27.56599 2.846764e-167 6.104174e-164    chrX 10778953
> >     5 FBgn0029167  5430.0673 -2.105062 0.09254660  -22.74596 1.573283e-114 2.698810e-111   chr3L 13846053
> >     6 FBgn0039827   390.9018 -3.503014 0.16002962  -21.88979 3.250384e-106 4.646424e-103   chr3R 31196915
> >
> >     End Strand        Feature   Gene.Name
> >     1 24147490      + protein_coding        Kal1
> >     2 10786958      - protein_coding        sesB
> >     3 26871995      - protein_coding BM-40-SPARC
> >     4 10786907      - protein_coding        Ant2
> >     5 13860001      + protein_coding         Hml
> >     6 31203722      + protein_coding      CG1544
> >     ```
> >
> > 10. `tail(annotatedDEgenes)`
> >
> >      ```R
> >      > tail(annotatedDEgenes)
> >               GeneID Base.mean  log2.FC.    StdErr Wald.Stats      P.value        P.adj Seqname    Start
> >      125 FBgn0036560  27.71424  1.089000 0.2327527   4.678786 2.885785e-06 6.671531e-05   chr3L 16035484
> >      126 FBgn0035710  26.16177  1.048979 0.2329221   4.503559 6.682472e-06 1.436480e-04   chr3L  6689326
> >      127 FBgn0035523  70.99820  1.004819 0.2237625   4.490560 7.103600e-06 1.523189e-04   chr3L  4277961
> >      128 FBgn0038261  44.27058  1.006264 0.2241243   4.489756 7.130482e-06 1.525141e-04   chr3R 14798985
> >      129 FBgn0039178  23.55006  1.040917 0.2326264   4.474631 7.654326e-06 1.629061e-04   chr3R 24283954
> >      130 FBgn0034636  24.77052 -1.028531 0.2321678  -4.430119 9.418101e-06 1.951185e-04   chr2R 21560245
> >
> >      End Strand        Feature Gene.Name
> >      125 16037227      + protein_coding    CG5895
> >      126  6703521      - protein_coding    SP1173
> >      127  4281585      + protein_coding    CG1311
> >      128 14801163      + protein_coding   CG14856
> >      129 24288617      + protein_coding    CG6356
> >      130 21576035      - protein_coding   CG10440
> >      ```
> >
> > 11. `annotatedDEgenes$GeneID`
> >
> >      ```R
> >      > annotatedDEgenes$GeneID
> >      [1] FBgn0039155 FBgn0003360 FBgn0026562 FBgn0025111 FBgn0029167 FBgn0039827 FBgn0035085 FBgn0034736
> >      [9] FBgn0264475 FBgn0000071 FBgn0029896 FBgn0038832 FBgn0037754 FBgn0264753 FBgn0034897 FBgn0035189
> >      [17] FBgn0011260 FBgn0027279 FBgn0040099 FBgn0024288 FBgn0001226 FBgn0085359 FBgn0040091 FBgn0262733
> >      [25] FBgn0023479 FBgn0052407 FBgn0051092 FBgn0039419 FBgn0261552 FBgn0034010 FBgn0261673 FBgn0031150
> >      [33] FBgn0033913 FBgn0264437 FBgn0040827 FBgn0037290 FBgn0016715 FBgn0031912 FBgn0003748 FBgn0001224
> >      [41] FBgn0035147 FBgn0261362 FBgn0024315 FBgn0261596 FBgn0032405 FBgn0039109 FBgn0260011 FBgn0261584
> >      [49] FBgn0053307 FBgn0063649 FBgn0001137 FBgn0259715 FBgn0038149 FBgn0031322 FBgn0001225 FBgn0002868
> >      [57] FBgn0039464 FBgn0265276 FBgn0000406 FBgn0030041 FBgn0033367 FBgn0036985 FBgn0039257 FBgn0039479
> >      [65] FBgn0063498 FBgn0037153 FBgn0051642 FBgn0263232 FBgn0032004 FBgn0031327 FBgn0034389 FBgn0033635
> >      [73] FBgn0034885 FBgn0032598 FBgn0000116 FBgn0261284 FBgn0023549 FBgn0034438 FBgn0031703 FBgn0052625
> >      [81] FBgn0038198 FBgn0050463 FBgn0000567 FBgn0037646 FBgn0038528 FBgn0033065 FBgn0024984 FBgn0034718
> >      [89] FBgn0043364 FBgn0004396 FBgn0035344 FBgn0267635 FBgn0031547 FBgn0032638 FBgn0033775 FBgn0033782
> >      [97] FBgn0039640 FBgn0267252 FBgn0031258 FBgn0030598 FBgn0035727 FBgn0033724 FBgn0263041 FBgn0020248
> >      [105] FBgn0031548 FBgn0034391 FBgn0000079 FBgn0003447 FBgn0039593 FBgn0265512 FBgn0030326 FBgn0039485
> >      [113] FBgn0025836 FBgn0033368 FBgn0037223 FBgn0010223 FBgn0063667 FBgn0002609 FBgn0037678 FBgn0250904
> >      [121] FBgn0264343 FBgn0038237 FBgn0020376 FBgn0028939 FBgn0036560 FBgn0035710 FBgn0035523 FBgn0038261
> >      [129] FBgn0039178 FBgn0034636
> >      130 Levels: FBgn0000071 FBgn0000079 FBgn0000116 FBgn0000406 FBgn0000567 FBgn0001137 ... FBgn0267635
> >      ```
> >
> > 12. `annotatedDEgenes[annotatedDEgenes$Feature == "pseudogene",]`
> >
> >     ```R
> >     > annotatedDEgenes[annotatedDEgenes$Feature == "pseudogene",]
> >     GeneID Base.mean   log2.FC.    StdErr Wald.Stats      P.value        P.adj Seqname   Start
> >     115 FBgn0037223  37.00783 -1.161706 0.2297393  -5.056627 4.267378e-07 1.192225e-05   chr3R 4243605
> >     End Strand    Feature Gene.Name
> >     115 4245511      + pseudogene     TwdlU
> >     ```
> >
> {: .solution}
{: .question}

The subsetting notation is very similar to what we learned for vectors. The key differences include:
- 2 values, separated by commas: `data.frame[row, column]`
- Use of a colon between the numbers (`start:stop, inclusive`) for a continuous range of numbers
- Passing a vector using `c()` for a non continuous set of numbers, pass a vector
- Index using the name of a column(s) by passing them as vectors using `c()`

> <hands-on-title>Create a new data frame by subsetting an existing one</hands-on-title>
> 1. Create a new data frame containing only observations from the "+" strand
>
>    ```R
>    # create a new data frame containing only observations from the "+" strand
>    annotatedDEgenes_plusStrand <- annotatedDEgenes[annotatedDEgenes$Strand == "+",]
>    ```
>
> 2. Check the dimension of the new data frame
>
>    ```R
>    # check the dimension of the data frame
>    > dim(annotatedDEgenes_plusStrand)
>    [1] 72 13
>    ```
>
> 3. Get a summary of the data frame
>
>    ```R
>    # get a summary of the data frame
>    > summary(annotatedDEgenes_plusStrand)
>    GeneID           Base.mean           log2.FC.         StdErr            Wald.Stats
>    FBgn0000071: 1   Min.   :   23.55   Min.   :-4.1485   Min.   :0.08603   Min.   :-30.741
>    FBgn0001224: 1   1st Qu.:   96.04   1st Qu.:-1.3664   1st Qu.:0.12523   1st Qu.:-11.223
>    FBgn0001226: 1   Median :  275.91   Median :-1.0739   Median :0.15648   Median : -5.290
>    FBgn0002609: 1   Mean   : 1365.34   Mean   :-0.3133   Mean   :0.16087   Mean   : -2.729
>    FBgn0003447: 1   3rd Qu.:  988.33   3rd Qu.: 1.1856   3rd Qu.:0.19743   3rd Qu.:  7.592
>    FBgn0003748: 1   Max.   :28915.90   Max.   : 2.3600   Max.   :0.23275   Max.   : 17.399
>    (Other)    :66
>
>    P.value             P.adj              Seqname    Start              End               Strand
>    Min.   :0.000e+00   Min.   :0.000e+00   chr2L:12   Min.   :  127448   Min.   :  140340   -: 0
>    1st Qu.:0.000e+00   1st Qu.:0.000e+00   chr2R:14   1st Qu.: 7377082   1st Qu.: 7379907   +:72
>    Median :0.000e+00   Median :0.000e+00   chr3L:15   Median :14068548   Median :14070644
>    Mean   :4.556e-07   Mean   :1.013e-05   chr3R:24   Mean   :14422245   Mean   :14431849
>    3rd Qu.:6.000e-12   3rd Qu.:3.200e-10   chrX : 7   3rd Qu.:21339535   3rd Qu.:21378634
>    Max.   :7.654e-06   Max.   :1.629e-04              Max.   :31196915   Max.   :31203722
>
>    Feature             Gene.Name
>    lincRNA       : 2   Ama    : 1
>    protein_coding:69   bou    : 1
>    pseudogene    : 1   bves   : 1
>                        CAHbeta: 1
>                        CG10365: 1
>                        CG1124 : 1
>                        (Other):66
>    ```
{: .hands_on}

## Change type of columns

Sometimes, it is possible that R will misinterpret the type of data represented in a data frame, or store that data in a mode which prevents you from operating on the data the way you wish. For example, a long list of gene names isn't usually thought of as a categorical variable, the way that your experimental condition (e.g. control, treatment) might be. More importantly, some R packages you use to analyze your data may expect characters as input, not factors. At other times (such as plotting or some statistical analyses) a factor may be more appropriate. Ultimately, you should know how to change the mode of an object.

First, its very important to recognize that coercion happens in R all the time. This can be a good thing when R gets it right, or a bad thing when the result is not what you expect.

> <hands-on-title>Format gene names</hands-on-title>
> 1. Check the structure of the `Gene.name` column
>
>    ```R
>    > str(annotatedDEgenes$Gene.name)
>    Factor w/ 130 levels "Ama","Amy-p",..: 88 113 5 3 84 26 41 53 69 1 ...
>    ```
>
> 2. Try to coerce the column into numeric
>
>    ```R
>    > as.numeric(annotatedDEgenes$Gene.name)
>    [1]  88 113   5   3  84  26  41  53  69   1  33  27  60 109 114  63 112
>    [18]  89  91 116  87  38 128 121 124  35  92  15 107  57  97   7  59  68
>    [35]  18  13 108  49 126  85  78 105 104 111  24  11  99  44  37  52  82
>    ...
>    ```
>
>    Strangely, it works! Almost. Instead of giving an error message, R returns numeric values, which in this case are the integers assigned to the levels in this factor. This kind of behavior can lead to hard-to-find bugs, for example when we do have numbers in a factor, and we get numbers from a coercion. If we don’t look carefully, we may not notice a problem.
>
> 3. Coerce the column into character (which makes sense for gene names)
>
>    ```R
>    annotatedDEgenes$Gene.name = as.character(annotatedDEgenes$Gene.name)
>    ```
>
{: .hands_on}

R will sometimes coerce without you asking for it (implicit coercion). You can explicitly change it (explicit coercion).

> <tip-title>Avoiding needless coercion</tip-title>
> One way to avoid needless coercion when importing a data frame using functions such as `read.csv()` is to set the argument `StringsAsFactors` to `FALSE`. By default, this argument is `TRUE`. Setting it to `FALSE` will treat any non-numeric column to a character type. Going through the `read.csv() `documentation, you will also see you can explicitly type your columns using the `colClasses` argument. Other R packages (such as the Tidyverse `readr`) don't have this particular conversion issue, but many packages will still try to guess a data type.
{: .tip}

## Inspect a numerical column using math, sorting, renaming

Here are a few operations that don't need much explanation, but which are good to know. There are lots of arithmetic functions you may want to apply to your data frame. For example, you can use functions like `mean()`, `min()`, `max()` on an individual column.

Let's look at the `Base.mean` column, i.e. the mean of normalized counts of all samples, normalizing for sequencing depth and library composition.

> <hands-on-title>Inspect the `Base.mean` column and sort</hands-on-title>
> 1. Get the maximum value of `Base.mean` column
>
>    ```R
>    max(annotatedDEgenes$Base.mean)
>    [1] 65114.84
>    ```
>
> 2. Sort the data frame by values in the `Base.mean` column
>
>    ```R
>    > sorted_by_BaseMean <- annotatedDEgenes[order(annotatedDEgenes$Base.mean), ]
>    > head(sorted_by_BaseMean$Base.mean)
>    [1] 19.15076 23.55006 24.77052 26.16177 27.71424 31.58769
>    ```
>
>    > <question-title>Changing the order</question-title>
>    > The `order()` function lists values in increasing order by default. Look at the documentation for this function and change `sorted_by_BaseMean` to start with differentially expressed means with the greatest base mean.
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > ```R
>    > > > sorted_by_BaseMean <- annotatedDEgenes[order(annotatedDEgenes$Base.mean, decreasing = TRUE), ]
>    > > > head(sorted_by_BaseMean$Base.mean)
>    > > [1] 65114.841 28915.903 24584.306 22435.886  8903.630  8573.815
>    > > ```
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Coerce the column into character (which makes sense for gene names)
>
>    ```R
>    annotatedDEgenes$Gene.name = as.character(annotatedDEgenes$Gene.name)
>    ```
>
{: .hands_on}

## Save the data frame to a file

We can save data frame as table in our Galaxy history to use it as input for other tools.

> <hands-on-title>Save data frame in Galaxy history</hands-on-title>
>
> 1. Save the `annotatedDEgenes_plusStrand` object to a `.csv` file using the `write.csv()` function:
>
>    ```R
>    > write.csv(annotatedDEgenes_plusStrand, file = "annotatedDEgenes_plusStrand.csv")
>    ```
>
> 2. ***TODO***: *Description on how to save a file from RStudio to Galaxy*
{: .hands_on}

The `write.csv()` function has some additional arguments listed in the help, but at a minimum you need to tell it what data frame to write to file, and give a path to a file name in quotes (if you only provide a file name, the file will be written in the current working directory).

# Aggregate and Analyze with dplyr

Selecting columns and rows using bracket subsetting is handy, but it can be cumbersome and difficult to read, especially for complicated operations.

The functions we've been using so far, like `str()`, come built into R. By using packages, we can access more functions. Here, we would like to use [`dplyr`](https://cran.r-project.org/package=dplyr) package provides a number of very useful functions for manipulating data frames in a way that will reduce repetition, reduce the probability of making errors, and probably even save you some typing. As an added bonus, you might even find the `dplyr` grammar easier to read.

> <comment-title>What is dplyr?</comment-title>
>
> The package `dplyr` is a fairly new (2014) package that tries to provide easy tools for the most common data manipulation tasks. It is built to work directly with data frames. The thinking behind it was largely inspired by the package `plyr` which has been in use for some time but suffered from being slow in some cases.` dplyr` addresses this by porting much of the computation to C++. An additional feature is the ability to work with data stored directly in an external database. The benefits of doing this are that the data can be managed natively in a relational database, queries can be conducted on that database, and only the results of the query returned.
>
> This addresses a common problem with R in that all operations are conducted in memory and thus the amount of data you can work with is limited by available memory. The database connections essentially remove that limitation in that you can have a database of many 100s GB, conduct queries on it directly and pull back just what you need for analysis in R.
{: .comment}

To use `dplyr`, we need first to install the package and then load it to be able to use it.

> <hands-on-title>Install and load dyplr</hands-on-title>
>
> 1. Load dyplr
>
>    ```R
>    ## load dyplr package
>    library("dplyr")
>    ```
>
>    > <tip-title>Didn't work?</tip-title>
>    > You might need to install dyplr first
>    >
>    > ```R
>    > > install.packages("dplyr")
>    > ```
>    >
>    > You might get asked to choose a CRAN mirror -- this is asking you to choose a site to download the package from. The choice doesn't matter too much.
>    {: .tip}
>
{: .hands_on}

We only need to install a package once per computer, but we need to load it every time we open a new R session and want to use that package.

## Select columns and filter rows

To select columns, instead with bracket, we could use the `select` function of `dplyr`

> <hands-on-title>Select column</hands-on-title>
>
> 1. Select `GeneID`, `Start`, `End`, `Strand` columns in `annotatedDEgenes`
>
>    ```R
>    > select(annotatedDEgenes, GeneID, Start, End, Strand)
>             GeneID    Start      End Strand
>    1   FBgn0039155 24141394 24147490      +
>    2   FBgn0003360 10780892 10786958      -
>    3   FBgn0026562 26869237 26871995      -
>    4   FBgn0025111 10778953 10786907      -
>    5   FBgn0029167 13846053 13860001      +
>    6   FBgn0039827 31196915 31203722      +
>    7   FBgn0035085 24945138 24946636      +
>    8   FBgn0034736 22550093 22552113      +
>    9   FBgn0264475   820758   821512      +
>    10  FBgn0000071  6762592  6765261      +
>    # ... with 120 more rows
>    ```
>
>    The first argument to this function is the data frame (`annotatedDEgenes`), and the subsequent arguments are the columns to keep.
>
> 2. Select all columns *except* certain ones
>
>    ```R
>    > select(annotatedDEgenes, -Chromosome)
>             GeneID   Base.mean  log2.FC.     StdErr Wald.Stats       P.value         P.adj    Start End Strand    Feature    Gene.Name
>    1   FBgn0039155  1086.97430 -4.148450 0.13494887 -30.740902 1.617921e-207 1.387691e-203 24141394  24147490      +  protein_coding         Kal1
>    2   FBgn0003360  6409.57713 -2.999777 0.10434506 -28.748628 9.422382e-182 4.040788e-178 10780892  10786958      -  protein_coding         sesB
>    3   FBgn0026562 65114.84056 -2.380164 0.08432692 -28.225437 2.850473e-175 8.149503e-172 26869237  26871995      -  protein_coding  BM-40-SPARC
>    4   FBgn0025111  2192.32237  2.699939 0.09794457  27.565988 2.846764e-167 6.104174e-164 10778953  10786907      -  protein_coding         Ant2
>    5   FBgn0029167  5430.06728 -2.105062 0.09254660 -22.745964 1.573283e-114 2.698810e-111 13846053  13860001      +  protein_coding          Hml
>    6   FBgn0039827   390.90178 -3.503014 0.16002962 -21.889786 3.250384e-106 4.646424e-103 31196915  31203722      +  protein_coding       CG1544
>    7   FBgn0035085   928.26381 -2.414074 0.11518516 -20.958204  1.579343e-97  1.935147e-94 24945138   24946636      + protein_coding       CG3770
>    8   FBgn0034736   330.38302 -3.018179 0.15815418 -19.083774  3.444661e-81  3.693107e-78 22550093   22552113      + protein_coding       CG6018
>    9   FBgn0264475   955.45445 -2.334486 0.12423003 -18.791643  8.840041e-79  8.424559e-76   820758   821512      +   lincRNA      CR43883
>    10  FBgn0000071   468.05793  2.360017 0.13564397  17.398615  8.452137e-68  7.249398e-65  6762592   6765261      +  protein_coding          Ama
>    # ... with 120 more rows
>    ```
{: .hands_on}

`dplyr` also provides useful functions to select columns based on their names. For instance, `starts_with()` allows you to select columns that starts with specific letters.

> <hands-on-title>Select columns that start with the letter "P"</hands-on-title>
>
> 1. Select columns that start with the letter "P"
>
>    ```R
>    > select(annotatedDEgenes, starts_with("P."))
>
>              P.value         P.adj
>    1   1.617921e-207 1.387691e-203
>    2   9.422382e-182 4.040788e-178
>    3   2.850473e-175 8.149503e-172
>    4   2.846764e-167 6.104174e-164
>    5   1.573283e-114 2.698810e-111
>    6   3.250384e-106 4.646424e-103
>    7    1.579343e-97  1.935147e-94
>    8    3.444661e-81  3.693107e-78
>    9    8.840041e-79  8.424559e-76
>    10   8.452137e-68  7.249398e-65
>    # ... with 120 more rows
>    ```
{: .hands_on}

> <question-title>Selecting on multiple conditions</question-title>
>
> Create a table that contains all the columns with the letter "s" in their name except for the column "Wald.Stats", and that contains the column "End".
>
> Hint: look at the help for the function `starts_with()` we've just covered.
>
> > <solution-title></solution-title>
> >
> > ```R
> > select(annotatedDEgenes, contains("s"), -Wald.Stats, End)
> > ```
>  {: .solution}
{: .question}

Similarly to choose rows, we can use the `filter()` function.

> <hands-on-title>Filter rows</hands-on-title>
>
> 1. Filter for rows on the forward strand
>
>    ```R
>    > filter(annotatedDEgenes, Strand == "+")
>            GeneID   Base.mean  log2.FC.     StdErr Wald.Stats       P.value         P.adj Chromosome Start      End Strand        Feature    Gene.Name
>    1  FBgn0039155  1086.97430 -4.148450 0.13494887 -30.740902 1.617921e-207 1.387691e-203      chr3R 24141394 24147490      + protein_coding         Kal1
>    2  FBgn0029167  5430.06728 -2.105062 0.09254660 -22.745964 1.573283e-114 2.698810e-111      chr3L 13846053 13860001      + protein_coding          Hml
>    3  FBgn0039827   390.90178 -3.503014 0.16002962 -21.889786 3.250384e-106 4.646424e-103      chr3R 31196915 31203722      + protein_coding       CG1544
>    4  FBgn0035085   928.26381 -2.414074 0.11518516 -20.958204  1.579343e-97  1.935147e-94      chr2R 24945138 24946636      + protein_coding       CG3770
>    5  FBgn0034736   330.38302 -3.018179 0.15815418 -19.083774  3.444661e-81  3.693107e-78      chr2R 22550093 22552113      + protein_coding       CG6018
>    6  FBgn0264475   955.45445 -2.334486 0.12423003 -18.791643  8.840041e-79  8.424559e-76      chr3L 820758   821512      +        lincRNA      CR43883
>    7  FBgn0000071   468.05793  2.360017 0.13564397  17.398615  8.452137e-68  7.249398e-65      chr3R 6762592  6765261      + protein_coding          Ama
>    8  FBgn0038832   429.85033 -2.340711 0.15483651 -15.117308  1.245307e-51  8.900828e-49      chr3R 20842139 20844981      + protein_coding      CG15695
>    9  FBgn0037754   394.46410 -2.071994 0.14494069 -14.295459  2.335613e-46  1.540966e-43      chr3R 9784652  9789323      + protein_coding       CG8500
>    10 FBgn0034897  1340.88978 -1.795107 0.12629203 -14.213934  7.508655e-46  4.293449e-43      chr2R 23713899 23734846      + protein_coding         Sesn
>    # ... with 120 more rows
>    ```
>
>    > <comment-title>Equivalent in base R code</comment-title>
>    > Note that this is equivalent to the base R code below, but is easier to read!
>    >
>    > ```R
>    > > annotatedDEgenes[annotatedDEgenes$Strand == "+",]
>    > ```
>    {: .comment}
>
>    `filter()` keeps all the rows that match the conditions that are provided
>
> 2. Filter for rows corresponding to genes in Chromosome X or 2R
>
>    ```R
>    ## rows for genes in Chromosome X or 2R
>    filter(annotatedDEgenes, Chromosome %in% c("chrX", "chr2R"))
>    ```
>
> 3. Filter for rows where the log2 fold change is greater than 2
>
>    ```R
>    ## rows where the log2 fold change is greater than 2
>    filter(annotatedDEgenes, log2.FC. >= 2)
>    ```
{: .hands_on}

`filter()` allows to combine multiple conditions. They need to be separated using a `,` as arguments to the function, they will be combined using the `&` (AND) logical operator. If you need to use the `|` (OR) logical operator, you can specify it explicitly:

> <hands-on-title>Filter rows for multiple conditions</hands-on-title>
>
> 1. Filter for genes on chrX and with adjusted p-value <= 1e-100
>
>    ```R
>    ## both are equivalent
>    filter(annotatedDEgenes, Chromosome == "chrX" & P.adj <= 1e-100)
>    filter(annotatedDEgenes, Chromosome == "chrX", P.adj <= 1e-100)
>    ```
>
> 2. Filter for genes on chrX and with log2 Fold Change smaller than -2 or larger than 2
>
>    ```R
>    filter(annotatedDEgenes, Chromosome == "chrX", (log2.FC. <= -2 | log2.FC. >= 2))
>    ```
{: .hands_on}

> <question-title>Practising with conditionals</question-title>
>
> Select all the rows for genes that:
> - start after position $$1e^6$$ (one million) and before position $$2e^6$$ (included) in their chromosome
> - have a log2 fold change greater than 1 or an adjusted p-value less than $$1e^{-75}$$.
>
> > <solution-title></solution-title>
> >
> > ```R
> > filter(annotatedDEgenes, Start >= 1e6 & End <= 2e6, (log2.FC. > 1 | P.adj < 1e-75)
> > ```
> {: .solution}
{: .question}

But what if you wanted to select and filter?

We can do this with **pipes**. Pipes, are a fairly recent addition to R. They let you take the output of one function and send it directly to the next, which is useful when you need to do many things to the same data set. It was possible to do this before pipes were added to R, but it was much messier and more difficult. Pipes in R look like `%>%` and are made available via the `magrittr` package, which is installed as part of `dplyr`.

> <comment-title>Pipes on RStudio</comment-title>
>
> With RStudio, you can type the pipe with
> - <kbd>Ctrl</kbd> + <kbd>Shift</kbd> + <kbd>M</kbd> if you're using a PC
> - <kbd>Cmd</kbd> + <kbd>Shift</kbd> + <kbd>M</kbd> if you're using a Mac
{: .comment}

> <hands-on-title>Select and filter</hands-on-title>
>
> 1. Filter for genes on forward strand, select `GeneID`, `Start`, `End`, `Chromosome` and display the first results using pipes
>
>    ```R
>    > annotatedDEgenes %>%
>       filter(Strand == "+") %>%
>       select(GeneID, Start, End, Chromosome) %>%
>       head()
>
>           GeneID    Start      End Chromosome
>    1 FBgn0039155 24141394 24147490      chr3R
>    2 FBgn0029167 13846053 13860001      chr3L
>    3 FBgn0039827 31196915 31203722      chr3R
>    4 FBgn0035085 24945138 24946636      chr2R
>    5 FBgn0034736 22550093 22552113      chr2R
>    6 FBgn0264475   820758   821512      chr3L
>    ```
>
{: .hands_on}

In the above code, we use the pipe to
1. Send the `annotatedDEgenes` dataset through `filter()`
2. Keep rows for genes on the '+' strand of their chromosome
3. Select through `select()` to keep only the `GeneID`, `Start`, `End`, and `Chromosome` columns
4. Show only the first 6 rows with `head`

> <comment-title>No need to give 1st argument to the functions</comment-title>
>
> Since `%>%` takes the object on its left and passes it as the first argument to the function on its right, we don't need to explicitly include the data frame as an argument to the `filter()` and `select()` functions any more.
>
{: .comment}

Some may find it helpful to read the pipe like the word "then". For instance, in the above example, we took the data frame `annotatedDEgenes`, *then* we `filter`ed for rows where `Strand` was +, *then* we `select`ed the `GeneID`, `Start`, `End`, and `Chromosome` columns, *then* we showed only the first six rows.

The `dplyr` functions by themselves are somewhat simple, but by combining them into linear workflows with the pipe, we can accomplish complex manipulations of data frames.

If we want to create a new object with this smaller version of the data we can do so by assigning it to a new object.

> <hands-on-title>Select and filter to create new object</hands-on-title>
>
> 1. Create an `plus_strand_genes` from `annotatedDEgenes` after filter for genes on forward strand and selection of `GeneID`, `Start`, `End`, `Chromosome`
>
>    ```R
>    plus_strand_genes <- annotatedDEgenes %>%
>     filter(Strand == "+") %>%
>     select(GeneID, Start,


topic: data-science
tutorial: python-typing
title: Python - Type annotations
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: b1dbbfdf875a261388a5cacd67de8e68
last_modified: 2026-05-15T22:28:11.109264
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-typing/tutorial.html

content:
In some languages type annotations are a core part of the language and types are checked at compile time, to ensure your code can never use the incorrect type of object. Python, and a few other dynamic languages, instead use ["Duck Typing"](https://en.wikipedia.org/wiki/Duck_typing) wherein the type of the object is less important than whether or not the correct methods or attributes are available.

However, we can provide type hints as we write python which will allow our editor to type check code as we go, even if it is not typically enforced at any point.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Types

Types used for annotations can be any of the base types:

```python
str
int
float
bool
None
...
```

or they can be relabeling of existing types, letting you create new types as needed to represent your internal data structures

```python
from typing import NewType

NameType = NewType("NameType", str)
Point2D = NewType("Point2D", tuple[float, float])
```

> <tip-title>Does `tuple` cause an issue?</tip-title>
> You might be on a python earlier than 3.9. Please update, or rewrite these as Tuple and List which must be imported.
{: .tip}

## But why?

Imagine for a minute you have a situation like the following, take a minute to read and understand the code:

```python
# Fetch the user and history list
(history_id, user_id) = GetUserAndCurrentHistory("hexylena")

# And make sure all of the permissions are correct
history = History.fetch(history_id)
history.share_with(user_id)
history.save()
```

> <question-title></question-title>
> 1. Can you be sure the `history_id` and `user_id` are in the correct order? It
>    seems like potentially not, given the ordering of "user" and "history" in the
>    function name, but without inspecting the definition of that function we
>    won't know.
> 2. What happens if `history_id` and `user_id` are swapped?
>
> > <solution-title></solution-title>
> > 1. This is unanswerable without the code.
> > 2. Depending on the magnitude of `history_id` and `user_id`, those may be within allowable ranges. Take for example
> > 
> >    User | History Id
> >    ---- | ----------
> >    1    | 1
> >    1    | 2
> >    2    | 3
> >    2    | 4
> >    
> >    Given `user_id=1` and `history_id=2` we may intend that the second row in our tables, history #2 owned by user #1, is shared with that user, as they're the owner. But if those are backwards, we'll get a situation where history #1 is actually associated with user #1, but instead we're sharing with user #2. We've created a situation where we've accidentally shared the wrong history with the wrong user! This could be a GDPR violation for our system and cause a lot of trouble.
> {: .solution}
{: .question}

However, if we have type definitions for the `UserId` and `HistoryId` that declare them as their own types:


```python
from typing import NewType

UserId = NewType("UserId", int)
HistoryId = NewType("HistoryId", int)
```

And then defined on our function, e.g.

```python
def GetUserAndCurrentHistory(username: str) -> tuple[UserId, HistoryId]:
    x = UserId(1) # Pretend this is fetching from the database
    y = HistoryId(2) # Likewise
    return (x, y)
```

we would be able to catch that, even if we call the variable `user_id`, it will still be typed checked. 

```python
history_id: HistoryId
user_id: UserId

(user_id, history_id) = GetUserAndCurrentHistory("hexylena")
(history_id, user_id) = GetUserAndCurrentHistory("hexylena")
```

If we're using a code editor with typing hints, e.g. VSCode with PyLance, we'll see something like:

![Screenshot of VSCode showing the functions from above. The version with history_id first has a bright red line under the function call of GetUserAndCurrentHistory. A popup tab shown on hovering over the function name shows that Expression of type UserId cannot be assigned to declared type HistoryId. UserId is incompatible with HistoryId](../../images/typing.png)

Here we see that we're not allowed to call this function this way, it's simply impossible.

> <question-title></question-title>
> What happens if you execute this code?
> > <solution-title></solution-title>
> > It executes happily. Types are **not enforced at runtime**. So this case where they're both custom types around an integer, Python sees that it expects an int in both versions of the function call, and that works fine for it. That is why we are repeatedly calling them "type hints", they're hints to your editor to show suggestions and help catch bugs, but they're not enforced.
> > If you modified the line `y = HistoryId(2)` to be something like `y = "test"`, the code will also execute fine. Python doesn't care that there's suddenly a string where you promised and asked for, an int. It simply does not matter.
> >
> > However, types *are* checked when you do operations involving them. Trying to get the `len()` of an integer? That will raise an `TypeError`, as integers don't support the `len()` call.
> {: .solution}
{: .question}

## Typing Variables

Adding types to variables is easy, you've seen a few examples already:

```python
a: str = "Hello"
b: int = 3
c: float = 3.14159
d: bool = True
```

### Complex Types

But you can go further than this with things like `tuple` and `list` types:

```python
e: list[int] = [1, 2, 3]
f: tuple[int, str] = (3, "Hi.")
g: list[tuple[int, int]] = [(1, 2), (3, 4)]
```

### Typing Functions

Likewise you've seen an example of adding type hints to a function:

```python
def reverse_list_of_ints(a: list[int]) -> list[int]:
    return a[::-1]
```

But this is a very specific function, right? We can reverse lists with more than just integers. For this, you can use `Any`:

```python
from typing import Any

def reverse_list(a: list[Any]) -> list[Any]:
    return a[::-1]
```

But this will lose the type information from the start of the function to the end. You said it was a `list[Any]` so your editor might not provide any type hints there, even though you could know, that calling it with a `list[int]` would always return the same type. Instead you can do

```python
from typing import TypeVar

T = TypeVar("T") # Implicitly any

def reverse_list(a: list[T]) -> list[T]:
    return a[::-1]
```

Now this will allow the function to accept a list of any type of value, int, float, etc. But it will also accept types you might not have intended:

```python
w: list[tuple[int, int]] = [(1, 2), (3, 4), (5, 8)]
reverse_list(w)
```

We can lock down what types we'll accept by using a `Union` instead of `Any`. With a `Union`, we can define that a type in that position might be any one of a few more specific types. Say your function can only accept strings, integers, or floats:

```python
from typing import Union

def reverse_list(a: list[Union[int, float, str]]) -> list[Union[int, float, str]]:
    return a[::-1]
```

Here we have used a `Union[A, B, ...]` to declare that it can only be one of these three types.

> <question-title></question-title>
> 1. Are both of these valid definitions?`
> 
>    ```python
>    q1: list[Union[int, float, str]] = [1, 2, 3]
>    q2: list[Union[int, float, str]] = [1, 2.3214, "asdf"]
>    ```
> 
> 2. If that wasn't what you expected, how would you define it so that it would be?
> > <solution-title></solution-title>
> > Yes, both are valid, but maybe you expected a homogeneous list. If you wanted that, you could instead do
> > 
> > ```python
> > q3: Union[list[int], list[float], list[str]] = [1, 2, 3]
> > q4: Union[list[int], list[float], list[str]] = [1, 2.3243, "asdf"] # Fails
> > ```
> {: .solution}
{: .question}


### Optional

Sometimes you have an argument to a function that is truly optional, maybe you have a different code path if it isn't there, or you simply process things differently but still correctly. You can explicitly declare this by defining it as `Optional`

```python
from typing import Optional

def pretty(lines: list[str], padding: Optional[str] = None) -> None:
    for line in lines:
        if padding:
            print(f"{padding} {line}")
        else:
            print(line)


lines = ["hello", "world", "你好", "世界"]

# Without the optional argument
pretty(lines)
# And with the optional
pretty(lines, "★")
```

While this superficially *looks* like a keyword argument with a default value, however it's subtly different. Here an explicit value of None is allowed, and we still know that it will either be a string, or it will be None. Not something that was possible with just a keyword argument.

## Testing for Types

You can use `mypy` to ensure that these type annotations are working in a project, this is a step you could add to your automated testing, if you have that. Using the `HistoryId`/`UserId` example from above, we can write that out into a script and test it out by running `mypy` on that file:

```console
$ mypy tmp.py
tmp.py:15: error: Incompatible types in assignment (expression has type "UserId", variable has type "HistoryId")
tmp.py:15: error: Incompatible types in assignment (expression has type "HistoryId", variable has type "UserId")
```

Here it reports the errors in the console, and you can use this to prevent bad code from being committed.

## Exercise

Here is an example module that would be stored in `corp/__init__.py`

```python
def repeat(x, n):
    """Return a list containing n references to x."""
    return [x]*n


def print_capitalized(x):
    """Print x capitalized, and return x."""
    print(x.capitalize())
    return x


def concatenate(x, y) :
    """Add two strings together."""
    return x + y
```

And here are some example invocations of that module, as found in `test.py`

```python
from corp import *

x = repeat("A", 3) # Should return ["A", "A", "A"]
y = print_capitalized("hElLo WorLd") # Should print Hello World
z = concatenate("Hi", "Bob") # HiBob
```

> <hands-on-title>Add type annotations</hands-on-title>
> 1. Add type annotations to each of those functions AND the variables `x`, `y`, `z`
> 2. How did you know which types were appropriate? 
> 3. Does `mypy` approve of your annotations? (Run `mypy test.py`, once you've written the above files out to their appropriate locations.)
{: .hands_on}

> <solution-title></solution-title>
> 1. The proper annotations: 
>
>    ```python
>    def repeat(x: str, n: int) -> list[str]:
>    # Or
>    from typing import TypeVar
>    T = TypeVar("T")
>    def repeat(x: T, n: int) -> list[T]:
> 
>    def print_capitalized(x: str) -> str:
> 
>    def concatenate(x: str, y:str) -> str:
>    ```
> 
>    and
> 
>    ```python
>    x: list[str] = ...
>    y: str = ...
>    z: str = ...
>    ```
> 
> 2. You might have discovered this by a combination of looking at the function definitions and their documentation, and perhaps also the sample invocations and what types were passed there.
> 3. We hope so!
>
{: .solution}

## Automation with MonkeyType

You can use MonkeyType to automatically apply type annotations to your code. Based on the execution of the code, it will make a best guess about what types are supported.

> <hands-on-title>Using MonkeyType to generate automatic annotations</hands-on-title>
> 
> 1. Create a folder for a module named `some`
> 2. Touch `some/__init__.py` to ensure it's importable as a python module
> 3. Create `some/module.py` and add the following contents:
> 
>    ```python
>    def add(a, b):
>        return a + B
>    ```
> 
> 4. Create a script that uses that module:
> 
>    ```
>    from some.module import add
>    
>    add(1, 2)
>    ```
> 
> 5. `pip install monkeytype`
> 6. Run MonkeyType to generate the annotations
> 
>    ```console
>    monkeytype run myscript.py
>    ```
> 
> 7. View the generated annotations
> 
>    ```console
>    monkeytype stub myscript.py
>    ```
{: .hands_on}

> <question-title></question-title>
> 1. What was the output of that command?
> 2. This function will accept strings as well, add a statement to exercise that in `myscript.py` and re-run `monkeytype run` and `monkeytype stub`. What is the new output?
>
> > <solution-title></solution-title>
> > 1. The expected output is:
> > 
> >    ```python
> >    def add(a: int, b: int) -> int: ...
> >    ```
> > 
> > 2. You can add a statement like `add("a", "b")` below `add(1, 2)` to see:
> > 
> >    ```python
> >    def add(a: Union[int, str], b: Union[int, str]) -> Union[int, str]: ...
> >    ```
> {: .solution}
{: .question}

> <question-title></question-title>
> Why is it different?
>
> > <solution-title></solution-title>
> > Because MonkeyType works by running the code provided (`myscript.py`) and annotating based on what executions it saw. In the first invocation it had not seen any calls to `add()` with strings, so it only reported `int` as acceptable types. However, the second time it saw `str`s as well. Can you think of another type that would be supported by this operation, that was not caught? (list!)
> {: .solution} 
{: .question}

> <question-title></question-title>
> 1. Does that type annotation make sense based on what you've learned today?
> 2. Can you write a better type annoation based on what you know?
>
> > <solution-title></solution-title>
> > 1. It works, but it's not a great type annotation. Here the description looks like it can accept two `int`s and return a `str` which isn't correct.
> > 2. Here is a better type annotation
> > 
> >    ```python
> >    from typing import TypeVar
> >    T = TypeVar("T", int, str, list)
> >    
> >    def add(a: T, b: T) -> T:
> >        return a + b
> >    ```
> {: .solution} 
{: .question}


topic: data-science
tutorial: bash-git
title: Version Control with Git
difficulty: intermediate
time_estimation: 65M
tools_json: ["RStudio", "interactive_tool_rstudio"]
requirements_json: []
tags_json: []
content_hash: 9f8a31eb712dcf823b823c8bca01e882
last_modified: 2026-05-15T22:28:10.893831
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/bash-git/tutorial.html

content:
Version control is a way of tracking the change history of a project, and `git` is one of the most popular systems for doing that! This tutorial will guide you through the basics of using git for version control.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Basics

We'll start by exploring how version control can be used
to keep track of what one person did and when.
Even if you aren't collaborating with other people,
automated version control is much better than this situation:

![Cartoon titled 'final'.doc, showing a grad student and their advisor going through multiple revisions. The first named final.doc, then final_rev.2.doc, final_rev.6.comments.doc, a long filename with the revision number 18, until a final filename, revision 22, with special characters indicating frustration where the file name includes the text 'why did I come to grad school'.](../../images/bash-git/phd101212s.png "Piled Higher and Deeper by Jorge Cham, http://www.phdcomics.com")

We've all been in this situation before: it seems unnecessary to have
multiple nearly-identical versions of the same document. Some word
processors let us deal with this a little better, such as Microsoft
Word's
[Track Changes](https://support.office.com/en-us/article/Track-changes-in-Word-197ba630-0f5f-4a8e-9a77-3712475e806a),
Google Docs' [version history](https://support.google.com/docs/answer/190843?hl=en), or
LibreOffice's [Recording and Displaying Changes](https://help.libreoffice.org/Common/Recording_and_Displaying_Changes).

Version control systems start with a base version of the document and
then record changes you make each step of the way. You can
think of it as a recording of your progress: you can rewind to start at the base
document and play back each change you made, eventually arriving at your
more recent version.

![Changes Are Saved Sequentially, graphic shows three documents with text being added in each new revision.](../../images/bash-git/play-changes.svg)

Once you think of changes as separate from the document itself, you
can then think about "playing back" different sets of changes on the base document, ultimately
resulting in different versions of that document. For example, two users can make independent
sets of changes on the same document.

![Different Versions Can be Saved, showing a document splitting into two, with different changes.](../../images/bash-git/versions.svg)

Unless multiple users make changes to the same section of the document - a conflict - you can
incorporate two sets of changes into the same base document.

![Multiple Versions Can be Merged, shows two documents with different changes merging into a final document with both changes.](../../images/bash-git/merge.svg)

A version control system is a tool that keeps track of these changes for us,
effectively creating different versions of our files. It allows us to decide
which changes will be made to the next version (each record of these changes is
called a [commit](https://git-scm.com/docs/gitglossary#def_commit), and keeps useful metadata
about them. The complete history of commits for a particular project and their
metadata make up a [repository](https://git-scm.com/docs/gitglossary#def_repository).
Repositories can be kept in sync across different computers, facilitating
collaboration among different people.

> <tip-title>The Long History of Version Control Systems</tip-title>
>
> Automated version control systems are nothing new.
> Tools like [RCS](https://en.wikipedia.org/wiki/Revision_Control_System), [CVS](https://en.wikipedia.org/wiki/Concurrent_Versions_System), or [Subversion](https://en.wikipedia.org/wiki/Apache_Subversion) have been around since the early 1980s and are used by
> many large companies.
> However, many of these are now considered legacy systems (i.e., outdated) due to various
> limitations in their capabilities.
> More modern systems, such as Git and [Mercurial](https://swcarpentry.github.io/hg-novice/),
> are *distributed*, meaning that they do not need a centralized server to host the repository.
> These modern systems also include powerful merging tools that make it possible for
> multiple authors to work on
> the same files concurrently.
{: .tip}

> <question-title>Paper Writing</question-title>
>
> *   Imagine you drafted an excellent paragraph for a paper you are writing, but later ruin
>     it. How would you retrieve the *excellent* version of your conclusion? Is it even possible?
>
> *   Imagine you have 5 co-authors. How would you manage the changes and comments
>     they make to your paper?  If you use LibreOffice Writer or Microsoft Word, what happens if
>     you accept changes made using the `Track Changes` option? Do you have a
>     history of those changes?
>
> > <solution-title></solution-title>
> >
> > *   Recovering the excellent version is only possible if you created a copy
> >     of the old version of the paper. The danger of losing good versions
> >     often leads to the problematic workflow illustrated in the PhD Comics
> >     cartoon at the top of this page.
> >
> > *   Collaborative writing with traditional word processors is cumbersome.
> >     Either every collaborator has to work on a document sequentially
> >     (slowing down the process of writing), or you have to send out a
> >     version to all collaborators and manually merge their comments into
> >     your document. The 'track changes' or 'record changes' option can
> >     highlight changes for you and simplifies merging, but as soon as you
> >     accept changes you will lose their history. You will then no longer
> >     know who suggested that change, why it was suggested, or when it was
> >     merged into the rest of the document. Even online word processors like
> >     Google Docs or Microsoft Office Online do not fully resolve these
> >     problems.
> {: .solution}
{: .question }


Before diving in the tutorial, we need to open {% tool [RStudio](interactive_tool_rstudio) %}. If you do not know how or never interacted with RStudio, please follow the [dedicated tutorial]({% link topics/galaxy-interface/tutorials/rstudio/tutorial.md %}).

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Installing git</hands-on-title>
> The R Console and other interactive tools like RStudio are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. This is one of those cases and, in order to do that, we will use the terminal provided by the RStudio itself.
> We go to "Tools" and pick the "Shell..." option and we are good to go. Our workspace is the left, terminal window that just opened.
>
> Fortunately, [miniconda](https://docs.conda.io/en/latest/miniconda.html) is already installed. Miniconda is a package manager that simplifies the installation processes. We can and will use it to install every essential package for our tutorial. However, it is of critical importance that we do that in an new environment within our existing base and install our packages in said environment.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ conda create -n name_of_your_env nano git
> > $ conda activate name_of_your_env
> > ```
> {: .code-in}
>
>
> | Software | Version | Manual | Available for | Description |
> | -------- | ------------ | ------ | ------------- | ----------- |
> | [git](https://git-scm.com/) | 2.35.3 | [git Manual](https://git-scm.com/book/en/v2) | Linux, MacOS | Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. |
> | [GNU nano](https://www.nano-editor.org/) | 2.9.8 | [Nano manual](https://www.nano-editor.org/dist/latest/nano.html) | Linux, MacOS | GNU nano is a small and friendly text editor. |
>
{: .hands_on}

# Setting up git

When we use Git on a new computer for the first time,
we need to configure a few things. Below are a few examples
of configurations we will set as we get started with Git:

*   our name and email address,
*   what our preferred text editor is,
*   and that we want to use these settings globally (i.e. for every project).

On a command line, Git commands are written as `git verb options`,
where `verb` is what we actually want to do and `options` is additional optional information which may be needed for the `verb`. So here is how
Sherlock sets up his new laptop:

> <code-in-title>Setting up with bash</code-in-title>
> ```bash
> $ git config --global user.name "Sherlock Holmes"
> $ git config --global user.email "sherlock@baker.street"
> ```
{: .code-in}

Please use your own name and email address instead of Sherlocks's. This user name and email will be associated with your subsequent Git activity,
which means that any changes pushed to
[GitHub](https://github.com/),
[BitBucket](https://bitbucket.org/),
[GitLab](https://gitlab.com/) or
another Git host server
after this lesson will include this information.

For this lesson, we will be interacting with [GitHub](https://github.com/) and so the email address used should be the same as the one used when setting up your GitHub account. If you are concerned about privacy, please review [GitHub's instructions for keeping your email address private][git-privacy].

> <tip-title>Keeping your email private</tip-title>
>
> If you choose to use a private email address with GitHub, then use that same email address for the `user.email` value, e.g. `username@users.noreply.github.com` replacing `username` with your GitHub one.
{: .tip}


> <tip-title>Line Endings</tip-title>
>
> As with other keys, when you hit <kbd>Enter</kbd> or <kbd></kbd> or on Macs, <kbd>Return</kbd>, on your keyboard,
> your computer encodes this input as a character.
> Different operating systems use different character(s) to represent the end of a line.
> (You may also hear these referred to as newlines or line breaks.)
> Because Git uses these characters to compare files,
> it may cause unexpected issues when editing a file on different machines.
> Though it is beyond the scope of this lesson, you can read more about this issue
> [in the Pro Git book](https://www.git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_core_autocrlf).
>
> You can change the way Git recognizes and encodes line endings
> using the `core.autocrlf` command to `git config`.
> The following settings are recommended:
>
> > <code-in-title>Settings with bash</code-in-title>
> > On macOS and Linux:
> >
> >```bash
> >$ git config --global core.autocrlf input
> >```
> >
> >And on Windows:
> >
> >```bash
> >$ git config --global core.autocrlf true
> >```
> {: .code-in}
{: .tip}

Sherlock also has to set his favorite text editor, following this table:

| Editor             | Configuration command                            |
|:-------------------|:-------------------------------------------------|
| Atom | `$ git config --global core.editor "atom --wait"`|
| nano               | `$ git config --global core.editor "nano -w"`    |
| BBEdit (Mac, with command line tools) | `$ git config --global core.editor "bbedit -w"`    |
| Sublime Text (Mac) | `$ git config --global core.editor "/Applications/Sublime\ Text.app/Contents/SharedSupport/bin/subl -n -w"` |
| Sublime Text (Win, 32-bit install) | `$ git config --global core.editor "'c:/program files (x86)/sublime text 3/sublime_text.exe' -w"` |
| Sublime Text (Win, 64-bit install) | `$ git config --global core.editor "'c:/program files/sublime text 3/sublime_text.exe' -w"` |
| Notepad (Win)    | `$ git config --global core.editor "c:/Windows/System32/notepad.exe"`|
| Notepad++ (Win, 32-bit install)    | `$ git config --global core.editor "'c:/program files (x86)/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"`|
| Notepad++ (Win, 64-bit install)    | `$ git config --global core.editor "'c:/program files/Notepad++/notepad++.exe' -multiInst -notabbar -nosession -noPlugin"`|
| Kate (Linux)       | `$ git config --global core.editor "kate"`       |
| Gedit (Linux)      | `$ git config --global core.editor "gedit --wait --new-window"`   |
| Scratch (Linux)       | `$ git config --global core.editor "scratch-text-editor"`  |
| Emacs              | `$ git config --global core.editor "emacs"`   |
| Vim                | `$ git config --global core.editor "vim"`   |
| VS Code                | `$ git config --global core.editor "code --wait"`   |

It is possible to reconfigure the text editor for Git whenever you want to change it.

> <tip-title>Exiting Vim</tip-title>
> Note that Vim is the default editor for many programs. If you haven't used Vim before and wish to exit a session without saving
> your changes, press <kbd>Esc</kbd> then type `:q!` and hit <kbd>Enter</kbd> or <kbd></kbd> or on Macs, <kbd>Return</kbd>.
> If you want to save your changes and quit, press <kbd>Esc</kbd> then type `:wq` and hit <kbd>Enter</kbd> or <kbd></kbd> or on Macs, <kbd>Return</kbd>.
{: .tip}

Git (2.28+) allows configuration of the name of the branch created when you
initialize any new repository.  Sherlock decides to use that feature to set it to `main` so
it matches the cloud service he will eventually use.

> <code-in-title>Configure the name of the created branch</code-in-title>
> ```bash
> $ git config --global init.defaultBranch main
> ```
{: .code-in}

> <tip-title>Default Git branch naming</tip-title>
>
> Source file changes are associated with a "branch."
> For new learners in this lesson, it's enough to know that branches exist, and this lesson uses one branch.
> By default, Git will create a branch called `master`
> when you create a new repository with `git init` (as explained in the next Episode). This term evokes
> the racist practice of human slavery and the
> [software development community](https://github.com/github/renaming)  has moved to adopt
> more inclusive language.
>
> In 2020, most Git code hosting services transitioned to using `main` as the default
> branch. As an example, any new repository that is opened in GitHub and GitLab default
> to `main`.  However, Git has not yet made the same change.  As a result, local repositories
> must be manually configured have the same main branch name as most cloud services.
>
> For versions of Git prior to 2.28, the change can be made on an individual repository level.  The
> command for this is in the next episode.  Note that if this value is unset in your local Git
> configuration, the `init.defaultBranch` value defaults to `master`.
>
{: .tip}

The five commands we just ran above only need to be run once: the flag `--global` tells Git
to use the settings for every project, in your user account, on this computer.

You can check your settings at any time:

> <code-in-title>Checking your settings</code-in-title>
> ```bash
> $ git config --list
> ```
{: .code-in}

You can change your configuration as many times as you want: use the
same commands to choose another editor or update your email address.

> <tip-title>Proxy</tip-title>
>
> In some networks you need to use a
> [proxy](https://en.wikipedia.org/wiki/Proxy_server). If this is the case, you
> may also need to tell Git about the proxy:
>
> > <code-in-title>Git and proxy</code-in-title>
> > ```bash
> > $ git config --global http.proxy proxy-url
> > $ git config --global https.proxy proxy-url
> > ```
> {: .code-in}
>
> > <code-in-title>To disable the proxy, use</code-in-title>
> >
> > ```bash
> > $ git config --global --unset http.proxy
> > $ git config --global --unset https.proxy
> > ```
> {: .code-in}
{: .tip}

> <tip-title>Git Help and Manual</tip-title>
>
> Always remember that if you forget the subcommands or options of a `git` command, you can access the relevant list of options typing `git <command> -h` or access the corresponding Git manual by typing
> `git <command> --help`, e.g.:
>
> > <code-in-title>Access help </code-in-title>
> >```bash
> >$ git config -h
> >$ git config --help
> >```
> {: .code-in}
>
> While viewing the manual, remember the `:` is a prompt waiting for commands and you can press <kbd>Q</kbd> to exit the manual.
>
> More generally, you can get the list of available `git` commands and further resources of the Git manual typing:
>
> > <code-in-title>Access available commands</code-in-title>
> > ```bash
> > $ git help
> > ```
> {: .code-in}
>
{: .tip}

[git-privacy]: https://help.github.com/articles/keeping-your-email-address-private/

# Create a Repo

Once Git is configured,
we can start using it.

We will continue with the story of Sherlock who is investigating a crime and is collecting information about suspects.

![Cartoon of sherlock examining the git logo with his magnifying glass.](../../images/bash-git/sherlock_git.png)


First, let's create a directory for our work and then move into that directory:

> <code-in-title>Create a workspace</code-in-title>
> ```bash
> $ mkdir suspects
> $ cd suspects
> ```
{: .code-in}

Then we tell Git to make `suspects` a [repository](https://git-scm.com/docs/gitglossary#def_repository)
-- a place where Git can store versions of our files:

> <code-in-title>Turn our workspace into directory</code-in-title>
> ```bash
> $ git init
> ```
{: .code-in}

It is important to note that `git init` will create a repository that
includes subdirectories and their files---there is no need to create
separate repositories nested within the `suspects` repository, whether
subdirectories are present from the beginning or added later. Also, note
that the creation of the `suspects` directory and its initialization as a
repository are completely separate processes.

If we use `ls` to show the directory's contents,
it appears that nothing has changed:

> <code-in-title>Show directory content</code-in-title>
> ```bash
> $ ls
> ```
{: .code-in}

But if we add the `-a` flag to show everything,
we can see that Git has created a hidden directory within `suspects` called `.git`:

> <code-in-title>Show everything in our directory</code-in-title>
> ```bash
> $ ls -a
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> .	..	.git
> ```
{: .code-out}

Git uses this special subdirectory to store all the information about the project,
including all files and sub-directories located within the project's directory.
If we ever delete the `.git` subdirectory, we will lose the project's history.

Next, we will change the default branch to be called `main`.
This might be the default branch depending on your settings and version
of git.

> <code-in-title>Rename the branch</code-in-title>
> ```bash
> $ git checkout -b main
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> Switched to a new branch 'main'
> ```
{: .code-out}


We can check that everything is set up correctly
by asking Git to tell us the status of our project:

> <code-in-title>Check</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
>
> No commits yet
>
> nothing to commit (create/copy files and use "git add" to track)
> ```
{: .code-out}

If you are using a different version of `git`, the exact
wording of the output might be slightly different.

## Places to Create Git Repositories

 Along with tracking information about suspects (the project we have already created), Sherlock would also like to track information specific clues. So, Sherlock creates a `clues` project inside his `suspects` project with the following sequence of commands:

> <code-in-title>Create a project within a project</code-in-title>
> ```bash
> $ cd ~/Desktop   # return to Desktop directory
> $ cd suspects     # go into suspects directory, which is already a Git repository
> $ ls -a          # ensure the .git subdirectory is still present in the suspects directory
> $ mkdir clues    # make a subdirectory suspects/clues
> $ cd clues       # go into clues subdirectory
> $ git init       # make the clues subdirectory a Git repository
> $ ls -a          # ensure the .git subdirectory is present indicating we have created a new Git repository
> ```
{: .code-in}

> <question-title>Tracking in a subdirectory</question-title>
> Is the `git init` command, run inside the `clues` subdirectory, required for tracking files stored in the `clues` subdirectory?
>
> > <solution-title></solution-title>
> >
> > No. Sherlock does not need to make the `clues` subdirectory a Git repository
> > because the `suspects` repository will track all files, sub-directories, and
> > subdirectory files under the `suspects` directory.  Thus, in order to track
> > all information about clues, Sherlock only needed to add the `clues` subdirectory
> > to the `suspects` directory.
> >
> {: .solution}
{: .question}

> <tip-title>"Nested" repositories</tip-title>
> Additionally, Git repositories can interfere with each other if they are "nested":
> the outer repository will try to version-control
> the inner repository. Therefore, it's best to create each new Git
> repository in a separate directory. To be sure that there is no conflicting
> repository in the directory, check the output of `git status`. If it looks
> like the following, you are good to go to create a new repository as shown
> above:
>
> > <code-in-title>Before creating a new repository</code-in-title>
> > ```bash
> > $ git status
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> >```
> >fatal: Not a git repository (or any of the parent directories): .git
> >```
> {: .code-out}
>
{: .tip}

## Correcting `git init` Mistakes
Dr. Watson explains to Sherlock how a nested repository is redundant and may cause confusion down the road. Sherlock would like to remove the nested repository. How can Sherlock undo his last `git init` in the `clues` subdirectory?

### USE `rm` WITH CAUTION!

Removing files from a Git repository needs to be done with caution. But we have not learned yet how to tell Git to track a particular file; we will learn this in the next episode. Files that are not tracked by Git can easily be removed like any other "ordinary" files with
> <code-in-title>Remove files </code-in-title>
> ```bash
> $ rm filename
> ```
{: .code-in}

Similarly a directory can be removed using `rm -r dirname` or `rm -rf dirname`. If the files or folder being removed in this fashion are tracked by Git, then their removal becomes another change that we will need to track, as we will see later.

Git keeps all of its files in the `.git` directory. To recover from this little mistake, Sherlock can just remove the `.git` folder in the clues subdirectory by running the following command from inside the `suspects` directory:

> <code-in-title>Remove the `.git` folder</code-in-title>
> ```bash
> $ rm -rf clues/.git
> ```
{: .code-in}

But be careful! Running this command in the wrong directory will remove the entire Git history of a project you might want to keep. Therefore, always check your current directory using the command `pwd`.

# Tracking Changes

First let's make sure we're still in the right directory.
You should be in the `suspects` directory.

> <code-in-title>Check directory</code-in-title>
> ```bash
> $ cd ~/suspects
> ```
{: .code-in}

Let's create a file called `colonel.txt` that contains some notes about the colonel Smith's connection to the case.
We'll use `nano` to edit the file;
you can use whatever editor you like.
In particular, this does not have to be the `core.editor` you set globally earlier. But remember, the bash command to create or edit a new file will depend on the editor you choose (it might not be `nano`). For a refresher on text editors, check out ["Which Editor?"](https://swcarpentry.github.io/shell-novice/03-create/) in [The Unix Shell](https://swcarpentry.github.io/shell-novice/) lesson.

> <code-in-title>Edit file with nano in bash</code-in-title>
> ```bash
> $ nano colonel.txt
> ```
{: .code-in}

Type the text below into the `colonel.txt` file:

```
No alibi for the night of murder.
```


Let's first verify that the file was properly created by running the list command (`ls`):

> <code-in-title>Check new file</code-in-title>
> ```bash
> $ ls
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> colonel.txt
> ```
{: .code-out}

`colonel.txt` contains a single line, which we can see by running:

> <code-in-title>Inspect the new file</code-in-title>
> ```bash
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> ```
{: .code-out}

If we check the status of our project again,
Git tells us that it's noticed the new file:

> <code-in-title>Check</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
>
> No commits yet
>
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
>
> colonel.txt
>
> nothing added to commit but untracked files present (use "git add" to track)
> ```
{: .code-out}

The "untracked files" message means that there's a file in the directory that Git isn't keeping track of. We can tell Git to track a file using `git add`:

> <code-in-title>Track file</code-in-title>
> ```bash
> $ git add colonel.txt
> ```
{: .code-in}

and then check that the right thing happened:

> <code-in-title>Check</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
>
> No commits yet
>
> Changes to be committed:
>   (use "git rm --cached <file>..." to unstage)
>
>	new file:   colonel.txt
>
> ```
{: .code-out}

Git now knows that it's supposed to keep track of `colonel.txt`, but it hasn't recorded these changes as a commit yet. To get it to do that, we need to run one more command:

> <code-in-title>Record changes as a commit with a descriptive title</code-in-title>
> ```bash
> $ git commit -m "Start notes for colonel Smith as a suspect"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
>[main (root-commit) f22b25e] Start notes on colonel Smith as a suspect
> 1 file changed, 1 insertion(+)
> create mode 100644 colonel.txt
> ```
{: .code-out}

When we run `git commit`,
Git takes everything we have told it to save by using `git add`
and stores a copy permanently inside the special `.git` directory.
This permanent copy is called a [commit](https://git-scm.com/docs/gitglossary#def_commit)
(or [revision](https://git-scm.com/docs/gitglossary#def_revision)) and its short identifier is `f22b25e`. Your commit may have another identifier.

We use the `-m` flag (for "message")
to record a short, descriptive, and specific comment that will help us remember later on what we did and why.
If we just run `git commit` without the `-m` option,
Git will launch `nano` (or whatever other editor we configured as `core.editor`)
so that we can write a longer message.

[Good commit messages][commit-messages] start with a brief <50 characters) statement about the
changes made in the commit. Generally, the message should complete the sentence "If applied, this commit will" <commit message here>.
If you want to go into more detail, add a blank line between the summary line and your additional notes. Use this additional space to explain why you made changes and/or what their impact will be.

> <code-in-title>If we run `git status` now:</code-in-title>
>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> nothing to commit, working directory clean
> ```
{: .code-out}

it tells us everything is up to date.
If we want to know what we've done recently,
we can ask Git to show us the project's history using `git log`:

> <code-in-title>Access project's history</code-in-title>
> ```bash
> $ git log
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> commit f22b25e3233b4645dabd0d81e651fe074bd8e73b
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:   Thu Aug 22 09:51:46 2013 -0400
>
>   Start notes on colonel as a suspect
> ```
{: .code-out}

`git log` lists all commits  made to a repository in reverse chronological order.
The listing for each commit includes
the commit's full identifier
(which starts with the same characters as
the short identifier printed by the `git commit` command earlier),
the commit's author,
when it was created,
and the log message Git was given when the commit was created.

> <tip-title>Where Are My Changes?</tip-title>
>
> If we run `ls` at this point, we will still see just one file called `colonel.txt`.
> That's because Git saves information about files' history
> in the special `.git` directory mentioned earlier
> so that our filesystem doesn't become cluttered
> (and so that we can't accidentally edit or delete an old version).
{: .tip}

Now suppose Sherlock adds more information to the file.
(Again, we'll edit with `nano` and then `cat` the file to show its contents;
you may use a different editor, and don't need to `cat`.)

> <code-in-title>Edit with nano</code-in-title>
> ```bash
> $ nano colonel.txt
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> No clear motive. Seems high unlikely.
> ```
{: .code-out}

When we run `git status` now,
it tells us that a file, it already knows about, has been modified:

> <code-in-title>Find about current status</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> Changes not staged for commit:
>  (use "git add <file>..." to update what will be committed)
>  (use "git checkout -- <file>..." to discard changes in working directory)
>
>	modified:   colonel.txt
>
> no changes added to commit (use "git add" and/or "git commit -a")
> ```
{: .code-out}

The last line is the key phrase:
"no changes added to commit".
We have changed this file,
but we haven't told Git we will want to save those changes
(which we do with `git add`)
nor have we saved them (which we do with `git commit`).
So let's do that now. It is good practice to always review
our changes before saving them. We do this using `git diff`.
This shows us the differences between the current state
of the file and the most recently saved version:

> <code-in-title>Review the changes </code-in-title>
> ```bash
> $ git diff
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index df0654a..315bf3a 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1 +1,2 @@
>  No alibi for the night of murder.
> +No clear motive. Seems high unlikely.
> ```
{: .code-out}

The output is cryptic because
it is actually a series of commands for tools like editors and `patch`
telling them how to reconstruct one file given the other.
If we break it down into pieces:

1.  The first line tells us that Git is producing output similar to the Unix `diff` command
    comparing the old and new versions of the file.
2.  The second line tells exactly which versions of the file
    Git is comparing;
    `df0654a` and `315bf3a` are unique computer-generated labels for those versions.
3.  The third and fourth lines once again show the name of the file being changed.
4.  The remaining lines are the most interesting, they show us the actual differences
    and the lines on which they occur.
    In particular,
    the `+` marker in the first column shows where we added a line.

After reviewing our change, it's time to commit it:

> <code-in-title>Commit after checking changes</code-in-title>
> ```bash
> $ git commit -m "Add concerns about existence of motive for colonel"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git checkout -- <file>..." to discard changes in working directory)
>
> 	modified:   colonel.txt
>
> no changes added to commit (use "git add" and/or "git commit -a")
> ```
{: .code-out}

Whoops:
Git won't commit because we didn't use `git add` first.
Let's fix that:

> <code-in-title>First `add` then `commit`</code-in-title>
> ```bash
> $ git add colonel.txt
> $ git commit -m "Add concerns about existence of motive for colonel"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> [main 34961b1] Add concerns about existence of motive for colonel
>  1 file changed, 1 insertion(+)
> ```
{: .code-out}

Git insists that we add files to the set we want to commit
before actually committing anything. This allows us to commit our
changes in stages and capture changes in logical portions rather than
only large batches.
For example,
suppose we're adding a few citations to relevant research to our thesis.
We might want to commit those additions,
and the corresponding bibliography entries,
but *not* commit some of our work drafting the conclusion
(which we haven't finished yet).

To allow for this,
Git has a special *staging area*
where it keeps track of things that have been added to
the current [changeset](https://git-scm.com/docs/gitglossary#def_changeset)
but not yet committed.

> <tip-title>Staging Area</tip-title>
>
> If you think of Git as taking snapshots of changes over the life of a project,
> `git add` specifies *what* will go in a snapshot
> (putting things in the staging area),
> and `git commit` then *actually takes* the snapshot, and
> makes a permanent record of it (as a commit).
> If you don't have anything staged when you type `git commit`,
> Git will prompt you to use `git commit -a` or `git commit --all`,
> which is kind of like gathering *everyone* to take a group photo!
> However, it's almost always better to
> explicitly add things to the staging area, because you might
> commit changes you forgot you made. (Going back to the group photo simile,
> you might get an extra with incomplete makeup walking on
> the stage for the picture because you used `-a`!)
> Try to stage things manually,
> or you might find yourself searching for "git undo commit" more
> than you would like!
{: .tip}

![The Git Staging Area cartoon, a document is shown going into the staging area via "git add", and then into the repository via "git commit".](../../images/bash-git/git-staging-area.svg)

Let's watch as our changes to a file move from our editor
to the staging area
and into long-term storage.
First,
we'll add another line to the file:

> <code-in-title>Add and review new line with `nano` and `cat`</code-in-title>
> ```bash
> $ nano colonel.txt
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> No clear motive. Seems high unlikely.
> Fingerprints on victims glasses.
> ```
{: .code-out}

> <code-in-title>Review changes</code-in-title>
> ```bash
> $ git diff
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index 315bf3a..b36abfd 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1,2 +1,3 @@
>  No alibi for the night of murder.
>  No clear motive. Seems high unlikely.
> +Fingerprints on victims glasses.
> ```
{: .code-out}

So far, so good:
we've added one line to the end of the file
(shown with a `+` in the first column).
Now let's put that change in the staging area
and see what `git diff` reports:

> <code-in-title>See changes</code-in-title>
> ```bash
> $ git add colonel.txt
> $ git diff
> ```
{: .code-in}

But, there is no output:
as far as Git can tell,
there's no difference between what it's been asked to save permanently
and what's currently in the directory.
However,
if we do this:

> <code-in-title>See what is in the staging area</code-in-title>
> ```bash
> $ git diff --staged
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index 315bf3a..b36abfd 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1,2 +1,3 @@
>  No alibi for the night of murder.
>  No clear motive. Seems high unlikely.
> +Fingerprints on victims glasses.
> ```
{: .code-out}

it shows us the difference between
the last committed change
and what's in the staging area.
Let's save our changes:

> <code-in-title>Save changes</code-in-title>
> ```bash
> $ git commit -m "Make notes about colonel's fingerprints"
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> [main 005937f] Make notes about colonel's fingerprints
>  1 file changed, 1 insertion(+)
> ```
{: .code-out}

> <code-in-title>Check new status</code-in-title>
> ```bash
> $ git status
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> On branch main
> nothing to commit, working directory clean
> ```
{: .code-out}

> <code-in-title>and look at the history of what we've done so far: </code-in-title>
> ```bash
> $ git log
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> commit 005937fbe2a98fb83f0ade869025dc2636b4dad5 (HEAD -> main)
> Author: Holmes Sherlock <Holmes@tran.sylvan.ia>
> Date:   Thu Aug 22 10:14:07 2013 -0400
>
>     Make notes about colonel's fingerprints
>
> commit 34961b159c27df3b475cfe4415d94a6d1fcd064d
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:   Thu Aug 22 10:07:21 2013 -0400
>
>     Add concerns about existence of motive for colonel
>
> commit f22b25e3233b4645dabd0d81e651fe074bd8e73b
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:   Thu Aug 22 09:51:46 2013 -0400
>
>     Start notes on colonel as a suspect
> ```
{: .code-out}

> <tip-title>Word-based diffing</tip-title>
>
> Sometimes, e.g. in the case of the text documents a line-wise
> diff is too coarse. That is where the `--color-words` option of
> `git diff` comes in very useful as it highlights the changed
> words using colors.
{: .tip}

> <tip-title>Paging the Log</tip-title>
>
> When the output of `git log` is too long to fit in your screen,
> `git` uses a program to split it into pages of the size of your screen.
> When this "pager" is called, you will notice that the last line in your
> screen is a `:`, instead of your usual prompt.
>
> *   To get out of the pager, press <kbd>Q</kbd>.
> *   To move to the next page, press <kbd>Spacebar</kbd>.
> *   To search for `some_word` in all pages,
>     press <kbd></kbd>
>     and type `some_word`.
>     Navigate through matches pressing <kbd>N</kbd>.
{: .tip}

> <tip-title>Limit Log Size</tip-title>
>
> To avoid having `git log` cover your entire terminal screen, you can limit the
> number of commits that Git lists by using `-N`, where `N` is the number of
> commits that you want to view. For example, if you only want information from
> the last commit you can use:
>
> > <code-in-title>See specific number of commits</code-in-title>
> > ```bash
> > $ git log -1
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > commit 005937fbe2a98fb83f0ade869025dc2636b4dad5 (HEAD -> main)
> > Author: Sherlock Holmes <sherlock@baker.street>
> > Date:   Thu Aug 22 10:14:07 2013 -0400
> >
> >    Make notes about colonel's fingerprints
> > ```
> {: .code-out}
>
> You can also reduce the quantity of information using the
> `--oneline` option:
>
> > <code-in-title>See only basic information</code-in-title>
> > ```
> > $ git log --oneline
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > 005937f (HEAD -> main) Make notes about colonel's fingerprints
> > 34961b1 Add concerns about existence of motive for colonel
> > f22b25e Start notes on colonel as a base
> > ```
> {: .code-out}
>
> You can also combine the `--oneline` option with others. One useful
> combination adds `--graph` to display the commit history as a text-based
> graph and to indicate which commits are associated with the
> current `HEAD`, the current branch `main`, or
> [other Git references][git-references]:
>
> > <code-in-title>Extra options</code-in-title>
> > ```bash
> > $ git log --oneline --graph
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > * 005937f (HEAD -> main) Make notes about colonel's fingerprints
> > * 34961b1 Add concerns about existence of motive for colonel
> > * f22b25e Start notes on colonel as a base
> > ```
> {: .code-out}
{: .tip}

> <tip-title>Directories </tip-title>
>
> Two important facts you should know about directories in Git.
>
> 1. Git does not track directories on their own, only files within them.
>    Try it for yourself:
>
>    > <code-in-title>Test directory tracking  </code-in-title>
>    > ```bash
>    > $ mkdir mysteries
>    > $ git status
>    > $ git add mysteries
>    > $ git status
>    > ```
>    {: .code-in}
>
>    Note, our newly created empty directory `mysteries` does not appear in
>    the list of untracked files even if we explicitly add it (_via_ `git add`) to our
>    repository. This is the reason why you will sometimes see `.gitkeep` files
>    in otherwise empty directories. Unlike `.gitignore`, these files are not special
>    and their sole purpose is to populate a directory so that Git adds it to
>    the repository. In fact, you can name such files anything you like.
>
> 2. If you create a directory in your Git repository and populate it with files,
>    you can add all files in the directory at once by:
>
>     ```
>     git add <directory-with-files>
>     ```
>
>    Try it for yourself:
>    > <code-in-title>Add multiple files</code-in-title>
>    > ```bash
>    > $ touch mysteries/belgravia mysteries/reichenbachfall
>    > $ git status
>    > $ git add mysteries
>    > $ git status
>    > ```
>    {: .code-in}
>
>    Before moving on, we will commit these changes.
>
>    > <code-in-title>Commit the new changes</code-in-title>
>    > ```
>    > $ git commit -m "Add some initial thoughts on mysteries"
>    > ```
>    {: .code-in}
>
{: .tip}

To recap, when we want to add changes to our repository,
we first need to add the changed files to the staging area
(`git add`) and then commit the staged changes to the
repository (`git commit`):

![The Git Commit Workflow](../../images/bash-git/git-committing.svg)

# Let's put us to the test

> <question-title>Choosing a Commit Message</question-title>
>
> Which of the following commit messages would be most appropriate for the
> last commit made to `colonel.txt`?
>
> 1. "Changes"
> 2. "Added line 'Fingerprints on victims glasses.' to colonel.txt"
> 3. "Make notes about colonel's fingerprints"
>
> > <solution-title></solution-title>
> > Answer 1 is not descriptive enough, and the purpose of the commit is unclear;
> > and answer 2 is redundant to using "git diff" to see what changed in this commit;
> > but answer 3 is good: short, descriptive, and imperative.
> {: .solution}
{: .question }

> <question-title>Committing Changes to Git</question-title>
>
> Which command(s) below would save the changes of `myfile.txt`
> to my local Git repository?
>
> 1. ```
>    $ git commit -m "my recent changes"
>    ```
>
> 2. ```
>    $ git init myfile.txt
>    $ git commit -m "my recent changes"
>    ```
> 3. ```
>    $ git add myfile.txt
>    $ git commit -m "my recent changes"
>    ```
> 4. ```
>    $ git commit -m myfile.txt "my recent changes"
>    ```
>
> > <solution-title></solution-title>
> >
> > 1. Would only create a commit if files have already been staged.
> > 2. Would try to create a new repository.
> > 3. Is correct: first add the file to the staging area, then commit.
> > 4. Would try to commit a file "my recent changes" with the message myfile.txt.
> {: .solution}
{: .question }

> <question-title>Committing Multiple Files</question-title>
>
> The staging area can hold changes from any number of files
> that you want to commit as a single snapshot.
>
> 1. Add some text to `colonel.txt` noting your suspicion on judge Brown.
> 2. Create a new file `judge.txt` with your initial thoughts
> about the judge as a suspect.
> 3. Add changes from both files to the staging area,
> and commit those changes.
>
> > <solution-title></solution-title>
> >
> > The output below from `cat colonel.txt` reflects only content added during
> > this exercise. Your output may vary.
> >
> > First we make our changes to the `colonel.txt` and `judge.txt` files:
> > > <code-in-title>Edit `colonel.txt`</code-in-title>
> > > ```bash
> > > $ nano colonel.txt
> > > $ cat colonel.txt
> > > ```
> > {: .code-in}
> > > <code-out-title></code-out-title>
> > > ```
> > > Maybe judge Brown should also be considerable as a suspect.
> > > ```
> > {: .code-out}
> > > <code-in-title>Create and edit `judge.txt`</code-in-title>
> > > ```bash
> > > $ nano judge.txt
> > > $ cat judge.txt
> > > ```
> > {: .code-in}
> > > <code-out-title></code-out-title>
> > > ```
> > > Judge seems like a nice guy, but has a shady past.
> > > ```
> > {: .code-out}
> >
> > Now you can add both files to the staging area. We can do that in one line:
> > > <code-in-title>Add both files</code-in-title>
> > > ```bash
> > > $ git add colonel.txt judge.txt
> > > ```
> > {: .code-in}
> >
> > > <code-in-title>Or with multiple commands: </code-in-title>
> > > ```bash
> > > $ git add colonel.txt
> > > $ git add judge.txt
> > > ```
> > {: .code-in}
> >
> > Now the files are ready to commit. You can check that using `git status`. If you are ready to commit use:
> > > <code-in-title>Commit changes</code-in-title>
> > > ```bash
> > > $ git commit -m "Write plans to start a base on judge"
> > > ```
> > {: .code-in}
> > > <code-out-title></code-out-title>
> > > ```
> > > [main cc127c2]
> > >  Write plans to start a base on judge
> > >  2 files changed, 2 insertions(+)
> > >  create mode 100644 judge.txt
> > > ```
> > {: .code-out}
> {: .solution}
{: .question }

> <question-title>`bio` Repository</question-title>
>
> 1. Create a new Git repository on your computer called `bio`.
> 2. Write a three-line biography for yourself in a file called `me.txt`,
> commit your changes
> 3. Modify one line, add a fourth line
> 4. Display the differences
> between its updated state and its original state.
>
> > <solution-title></solution-title>
> >
> > If needed, move out of the `suspects` folder:
> > > <code-in-title>Change directory</code-in-title>
> > > ```bash
> > > $ cd ..
> > > ```
> > {: .code-in}
> >
> > Create a new folder called `bio` and 'move' into it:
> > > ### Create folder
> > > ```bash
> > > $ mkdir bio
> > > $ cd bio
> > > ```
> > {: .code-in}
> >
> > > <code-in-title>Initialise git:</code-in-title>
> > > ```bash
> > > $ git init
> > > ```
> > {: .code-in}
> >
> > Create your biography file `me.txt` using `nano` or another text editor.
> > Once in place, add and commit it to the repository:
> >
> > > <code-in-title>Create file and edit it</code-in-title>
> > > ```bash
> > > $ git add me.txt
> > > $ git commit -m "Add biography file"
> > > ```
> > {: .code-in}
> >
> > Modify the file as described (modify one line, add a fourth line).
> > To display the differences
> > between its updated state and its original state, use `git diff`:
> >
> > > <code-in-title>Display the differences</code-in-title>
> > > ```bash
> > > $ git diff me.txt
> > > ```
> > {: .code-in}
> {: .solution}
{: .question }

[commit-messages]: https://chris.beams.io/posts/git-commit/
[git-references]: https://git-scm.com/book/en/v2/Git-Internals-Git-References

# History Exploring

As we saw in the previous episode, we can refer to commits by their
identifiers.  You can refer to the _most recent commit_ of the working
directory by using the identifier `HEAD`.

We've been adding one line at a time to `colonel.txt`, so it's easy to track our
progress by looking, so let's do that using our `HEAD`s.  Before we start,
let's make a change to `colonel.txt`, adding yet another line.

> <code-in-title>Edit `colonel.txt`</code-in-title>
> ```bash
> $ nano colonel.txt
> $ cat colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> No alibi for the night of murder.
> No clear motive. Seems high unlikely.
> Fingerprints on victims glasses.
> Maybe judge Brown should also be considerable as a suspect.
> ```
{: .code-out}

Now, let's see what we get.

> <code-in-title>Display the changes</code-in-title>
> ```bash
> $ git diff HEAD colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index b36abfd..0848c8d 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1,3 +1,4 @@
>  No alibi for the night of murder.
>  No clear motive. Seems high unlikely.
>  Fingerprints on victims glasses.
> +Maybe judge Brown should also be considerable as a suspect.
> ```
{: .code-out}

which is the same as what you would get if you leave out `HEAD` (try it).  The
real goodness in all this is when you can refer to previous commits.  We do
that by adding `~1`
(where "~" is "tilde", pronounced [**til**-d*uh*])
to refer to the commit one before `HEAD`.

> <code-in-title>Refer to previous commits</code-in-title>
> ```bash
> $ git diff HEAD~1 colonel.txt
> ```
{: .code-in}

If we want to see the differences between older commits we can use `git diff`
again, but with the notation `HEAD~1`, `HEAD~2`, and so on, to refer to them:

> <code-in-title>Refer to previous commits</code-in-title>
> ```bash
> $ git diff HEAD~3 colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> diff --git a/colonel.txt b/colonel.txt
> index df0654a..b36abfd 100644
> --- a/colonel.txt
> +++ b/colonel.txt
> @@ -1 +1,4 @@
>  No alibi for the night of murder.
> +No clear motive. Seems high unlikely.
> +Fingerprints on victims glasses.
> +Maybe judge Brown should also be considerable as a suspect.
> ```
{: .code-out}

We could also use `git show` which shows us what changes we made at an older commit as
well as the commit message, rather than the _differences_ between a commit and our
working directory that we see by using `git diff`.

> <code-in-title>`git show` and `git diff` differences </code-in-title>
> ```bash
> $ git show HEAD~3 colonel.txt
> ```
{: .code-in}

> <code-out-title></code-out-title>
> ```
> commit f22b25e3233b4645dabd0d81e651fe074bd8e73b
> Author: Sherlock Holmes <sherlock@baker.street>
> Date:


topic: data-science
tutorial: bash-variant-calling
title: Variant Calling Workflow
difficulty: intermediate
time_estimation: 50M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 3cb587173fe68d405598bccb6b92413d
last_modified: 2026-05-15T22:28:10.896056
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/bash-variant-calling/tutorial.html

content:
In this tutorial we are working with files from a long-term evolution study of an *E. coli* population (designated Ara-3). We will perform variant calling to see how the population changed over time. We care how this population changed relative to the original population, *E. coli* strain REL606. Therefore, we will align each of our samples to the *E. coli* REL606 reference genome, and see what differences exist in our reads versus the genome.

The whole proccess is designed to be run in RStudio as an Interactive Tool, directly in Galaxy.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/) lesson
>
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Software to use</hands-on-title>
>
> The R Console and other interactive tools like RStudio are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. This is one of those cases and, in order to do that, we will use the terminal provided by the RStudio itself.
> We go to "Tools" and pick the "Shell..." option and we are good to go. Our workspace is the left, terminal window that just opened.
>
> Fortunately, [miniconda](https://docs.conda.io/en/latest/miniconda.html) is already installed. Miniconda is a package manager that simplifies the installation processes. We can and will use it to install every essential package for our tutorial. However, it is of critical importance that we do that in an new environment within our existing base and install our packages in said environment.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ conda create -n name_of_your_env bwa samtools bcftools
> > $ conda activate name_of_your_env
> > ```
> {: .code-in}
>
>
> | Software | Version | Manual | Available for | Description |
> | -------- | ------------ | ------ | ------------- | ----------- |
> | [BWA](http://bio-bwa.sourceforge.net/) | 0.7.17 | [BWA Manual](http://bio-bwa.sourceforge.net/bwa.shtml) | Linux, MacOS | Mapping DNA sequences against reference genome. |
> | [SAMtools](http://samtools.sourceforge.net/) | 1.15.1 | [SAMtools Manual](http://www.htslib.org/doc/samtools.html) | Linux, MacOS | Utilities for manipulating alignments in the SAM format. |
> | [BCFtools](https://samtools.github.io/bcftools/) | 1.15.1 | [BCFtools manual](https://samtools.github.io/bcftools/bcftools.html) | Linux, MacOS | Utilities for variant calling and manipulating VCFs and BCFs. |
> | [IGV](http://software.broadinstitute.org/software/igv/home) | [IGV Download](https://software.broadinstitute.org/software/igv/download) | [IGV User Guide](https://software.broadinstitute.org/software/igv/UserGuide) | Linux, MacOS, Windows | Visualization and interactive exploration of large genomics datasets. |
>
{: .hands_on}

# Alignment to a reference genome

![Simplified workflow that starts with the raw sequence reads and ends up to genome alignment](../../images/bash-variant-calling/variant_calling_workflow_align.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

We perform read alignment or mapping to determine where in the genome our reads originated from. There are a number of tools to
choose from and, while there is no gold standard, there are some tools that are better suited for particular NGS analyses. We will be
using the [Burrows Wheeler Aligner (BWA)](http://bio-bwa.sourceforge.net/), which is a software package for mapping low-divergent
sequences against a large reference genome.

The alignment process consists of two steps:

1. Indexing the reference genome
2. Aligning the reads to the reference genome

# Setting up

> <hands-on-title>Download Genome</hands-on-title>
>
> First we download the reference genome for *E. coli* REL606. Although we could copy or move the file with `cp` or `mv`, most genomics workflows begin with a download step, so we will practice that here.
>
> > ###  {% icon code-in %} Input: Bash
> > ```bash
> > $ cd ~/dc_workshop
> > $ mkdir -p data/ref_genome
> > $ curl -L -o data/ref_genome/ecoli_rel606.fasta.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/017/985/GCA_000017985.1_ASM1798v1/GCA_000017985.1_ASM1798v1_genomic.fna.gz
> > $ gunzip data/ref_genome/ecoli_rel606.fasta.gz
> > ```
> {: .code-in}
>
> > <question-title></question-title>
> >
> > We saved this file as `data/ref_genome/ecoli_rel606.fasta.gz` and then decompressed it.
> > What is the real name of the genome?
> >
> > > <solution-title></solution-title>
> > >
> > > ```bash
> > > $ head data/ref_genome/ecoli_rel606.fasta
> > > ```
> > >
> > >
> > > The name of the sequence follows the `> ` character. The name is `CP000819.1 Escherichia coli B str. REL606, complete genome`.
> > > Keep this chromosome name (`CP000819.1`) in mind, as we will use it later in the lesson.
> > {: .solution}
> {: .question}
>
> We will also download a set of trimmed FASTQ files to work with that will enable us to run our variant calling workflow quite quickly.
>
> > <code-in-title>Getting easy-to-work FASTQ files</code-in-title>
> > ```bash
> > $ curl -L -o sub.tar.gz https://ndownloader.figshare.com/files/14418248
> > $ tar xvf sub.tar.gz
> > $ mv sub/ ~/dc_workshop/data/trimmed_fastq_small
> > ```
> {: .code-in}
>
> You will also need to create directories for the results that will be generated as part of this workflow. We can do this in a single line of code, because `mkdir` can accept multiple new directory names as input.
>
> > <code-in-title>Create result directories</code-in-title>
> > ```bash
> > $ mkdir -p results/sam results/bam results/bcf results/vcf
> > ```
> {: .code-in}
{: .hands_on}


> <hands-on-title>Index the reference genome</hands-on-title>
> Our first step is to index the reference genome for use by BWA. Indexing allows the aligner to quickly find potential alignment sites for query sequences in a genome, which saves time during alignment. Indexing the reference only has to be run once. The only reason you would want to create a new index is if you are working with a different reference genome or you are using a different tool for alignment.
>
> > <code-in-title>Indexing with `bwa`</code-in-title>
> > ```bash
> > $ bwa index data/ref_genome/ecoli_rel606.fasta
> > ```
> {: .code-in}
>
> While the index is created, you will see output that looks something like this:
>
> > <code-out-title></code-out-title>
> > ```
> > [bwa_index] Pack FASTA... 0.04 sec
> > [bwa_index] Construct BWT for the packed sequence...
> > [bwa_index] 1.05 seconds elapse.
> > [bwa_index] Update BWT... 0.03 sec
> > [bwa_index] Pack forward-only FASTA... 0.02 sec
> > [bwa_index] Construct SA from BWT and Occ... 0.57 sec
> > [main] Version: 0.7.17-r1188
> > [main] CMD: bwa index data/ref_genome/ecoli_rel606.fasta
> > [main] Real time: 1.765 sec; CPU: 1.715 sec
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Align reads to reference genome</hands-on-title>
>
> The alignment process consists of choosing an appropriate reference genome to map our reads against and then deciding on an
> aligner. We will use the BWA-MEM algorithm, which is the latest and is generally recommended for high-quality queries as it
> is faster and more accurate.
>
> An example of what a `bwa` command looks like is below. This command will not run, as we do not have the files `ref_genome.fa`, `input_file_R1.fastq`, or `input_file_R2.fastq`.
>
>
> ```
> $ bwa mem ref_genome.fasta input_file_R1.fastq input_file_R2.fastq > output.sam
> ```
>
>
> Have a look at the [bwa options page](http://bio-bwa.sourceforge.net/bwa.shtml). While we are running bwa with the default
> parameters here, your use case might require a change of parameters. *NOTE: Always read the manual page for any tool before using
> and make sure the options you use are appropriate for your data.*
>
> We are going to start by aligning the reads from just one of the
> samples in our dataset (`SRR2584866`). Later, we will be
> iterating this whole process on all of our sample files.
>
> > <code-in-title>Aligning with bwa</code-in-title>
> > ```bash
> > $ bwa mem data/ref_genome/ecoli_rel606.fasta data/trimmed_fastq_small/SRR2584866_1.trim.sub.fastq data/trimmed_fastq_small/SRR2584866_2.trim.sub.fastq > results/sam/SRR2584866.aligned.sam
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > [M::bwa_idx_load_from_disk] read 0 ALT contigs
> > [M::process] read 77446 sequences (10000033 bp)...
> > [M::process] read 77296 sequences (10000182 bp)...
> > [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (48, 36728, 21, 61)
> > [M::mem_pestat] analyzing insert size distribution for orientation FF...
> > [M::mem_pestat] (25, 50, 75) percentile: (420, 660, 1774)
> > [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 4482)
> > [M::mem_pestat] mean and std.dev: (784.68, 700.87)
> > [M::mem_pestat] low and high boundaries for proper pairs: (1, 5836)
> > [M::mem_pestat] analyzing insert size distribution for orientation FR...
> > ```
> {: .code-out}
{: .hands_on}

# SAM/BAM format

The [SAM file](https://genome.sph.umich.edu/wiki/SAM),
is a tab-delimited text file that contains information for each individual read and its alignment to the genome. While we do not
have time to go into detail about the features of the SAM format, the paper by
[Heng Li et al.](http://bioinformatics.oxfordjournals.org/content/25/16/2078.full) provides a lot more detail on the specification.

**The compressed binary version of SAM is called a BAM file.** We use this version to reduce size and to allow for *indexing*, which enables efficient random access of the data contained within the file.

The file begins with a **header**, which is optional. The header is used to describe the source of data, reference sequence, method of
alignment, etc., this will change depending on the aligner being used. Following the header is the **alignment section**. Each line
that follows corresponds to alignment information for a single read. Each alignment line has **11 mandatory fields** for essential
mapping information and a variable number of other fields for aligner specific information. An example entry from a SAM file is
displayed below with the different fields highlighted.

![sam file example with fields description, showing qname, flag, rname, pos, maqp, and cigar as the first fields](../../images/bash-variant-calling/sam_bam.png)

Image from [Data Wrangling and Processing for Genomics](https://datacarpentry.org/wrangling-genomics/)

![sam file example with fields description, showing seq, qual, mrnm, mpos, and size as the second set of fields.](../../images/bash-variant-calling/sam_bam3.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

> <hands-on-title>SAM to BAM</hands-on-title>
> > <code-in-title>bash</code-in-title>
> > We will convert the SAM file to BAM format using the `samtools` program with the `view` command and tell this command that the input is in SAM format (`-S`) and to output BAM format (`-b`):
> >
> > ```bash
> > $ samtools view -S -b results/sam/SRR2584866.aligned.sam > results/bam/SRR2584866.aligned.bam
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > [samopen] SAM header is present: 1 sequences.
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Sort BAM file by coordinates</hands-on-title>
>
> Next we sort the BAM file using the `sort` command from `samtools`. `-o` tells the command where to write the output.
> > <code-in-title>`sort` command</code-in-title>
> > ```bash
> > $ samtools sort -o results/bam/SRR2584866.aligned.sorted.bam results/bam/SRR2584866.aligned.bam
> > ```
> {: .code-in}
>
> Our files are pretty small, so we will not see this output. If you run the workflow with larger files, you will see something like this:
> > <code-out-title></code-out-title>
> >
> > ```
> > [bam_sort_core] merging from 2 files...
> > ```
> {: .code-out}
>
> SAM/BAM files can be sorted in multiple ways, e.g. by location of alignment on the chromosome, by read name, etc. It is important to be aware that different alignment tools will output differently sorted SAM/BAM, and different downstream tools require differently sorted alignment files as input.
>
> You can use samtools to learn more about this bam file as well.
>
> > <code-in-title>`flagstat` command</code-in-title>
> > ```bash
> > samtools flagstat results/bam/SRR2584866.aligned.sorted.bam
> > ```
> {: .code-in}
>
> This will give you the following statistics about your sorted bam file:
>
> > <code-out-title></code-out-title>
> > ```
> > 351169 + 0 in total (QC-passed reads + QC-failed reads)
> > 0 + 0 secondary
> > 1169 + 0 supplementary
> > 0 + 0 duplicates
> > 351103 + 0 mapped (99.98% : N/A)
> > 350000 + 0 paired in sequencing
> > 175000 + 0 read1
> > 175000 + 0 read2
> > 346688 + 0 properly paired (99.05% : N/A)
> > 349876 + 0 with itself and mate mapped
> > 58 + 0 singletons (0.02% : N/A)
> > 0 + 0 with mate mapped to a different chr
> > 0 + 0 with mate mapped to a different chr (mapQ> =5)
> > ```
> {: .code-out}
{: .hands_on}

# Variant calling

A variant call is a conclusion that there is a nucleotide difference vs. some reference at a given position in an individual genome
or transcriptome, often referred to as a Single Nucleotide Variant (SNV). The call is usually accompanied by an estimate of
variant frequency and some measure of confidence. Similar to other steps in this workflow, there are a number of tools available for
variant calling. In this workshop we will be using `bcftools`, but there are a few things we need to do before actually calling the
variants.

![Workflow that leads from sequence reads to "variant calling" step](../../images/bash-variant-calling/variant_calling_workflow.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

> <hands-on-title>Step 1: Calculate the read coverage of positions in the genome</hands-on-title>
>
> Do the first pass on variant calling by counting read coverage with
> [bcftools](https://samtools.github.io/bcftools/bcftools.html). We will
> use the command `mpileup`. The flag `-O b` tells bcftools to generate a
> bcf format output file, `-o` specifies where to write the output file, and `-f` flags the path to the reference genome:
> > <code-in-title>`mpileup` command</code-in-title>
> > ```bash
> > $ bcftools mpileup -O b -o results/bcf/SRR2584866_raw.bcf -f data/ref_genome/ecoli_rel606.fasta results/bam/SRR2584866.aligned.sorted.bam
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > [mpileup] 1 samples in 1 input files
> > ```
> {: .code-out}
>
> We have now generated a file with coverage information for every base.
>
{: .hands_on}

> <hands-on-title>Step 2: Detect the single nucleotide variants (SNVs</hands-on-title>
>
> Identify SNVs using bcftools `call`. We have to specify ploidy with the flag `--ploidy`, which is one for the haploid *E. coli*. `-m` allows for multiallelic and rare-variant calling, `-v` tells the program to output variant sites only (not every site in the genome), and `-o` specifies where to write the output file:
> > <code-in-title>`call` command</code-in-title>
> > ```bash
> > $ bcftools call --ploidy 1 -m -v -o results/vcf/SRR2584866_variants.vcf results/bcf/SRR2584866_raw.bcf
> > ```
> {: .code-in}
>
{: .hands_on}

> <hands-on-title>Step 3: Filter and report the SNV variants in variant calling format (VCF</hands-on-title>
>
> Filter the SNVs for the final output in VCF format, using `vcfutils.pl`:
> > <code-in-title>Filtering with `vcfutils.pl` command</code-in-title>
> > ```bash
> > $ vcfutils.pl varFilter results/vcf/SRR2584866_variants.vcf  > results/vcf/SRR2584866_final_variants.vcf
> > ```
> {: .code-in}
>
{: .hands_on}


> <hands-on-title>Explore the VCF format:</hands-on-title>
>
> > <code-in-title></code-in-title>
> > ```bash
> > $ less -S results/vcf/SRR2584866_final_variants.vcf
> > ```
> {: .code-in}
>
> You will see the header (which describes the format), the time and date the file was
> created, the version of bcftools that was used, the command line parameters used, and
> some additional information:
>
> > <code-out-title></code-out-title>
> > ```
> > ##fileformat=VCFv4.2
> > ##FILTER<ID=PASS,Description="All filters passed">
> > ##bcftoolsVersion=1.8+htslib-1.8
> > ##bcftoolsCommand=mpileup -O b -o results/bcf/SRR2584866_raw.bcf -f data/ref_genome/ecoli_rel606.fasta results/bam/SRR2584866.aligned.sorted.bam
> > ##reference=file://data/ref_genome/ecoli_rel606.fasta
> > ##contig<ID=CP000819.1,length=4629812>
> > ##ALT<ID=*,Description="Represents allele(s) other than observed.">
> > ##INFO<ID=INDEL,Number=0,Type=Flag,Description="Indicates that the variant is an INDEL.">
> > ##INFO<ID=IDV,Number=1,Type=Integer,Description="Maximum number of reads supporting an indel">
> > ##INFO<ID=IMF,Number=1,Type=Float,Description="Maximum fraction of reads supporting an indel">
> > ##INFO<ID=DP,Number=1,Type=Integer,Description="Raw read depth">
> > ##INFO<ID=VDB,Number=1,Type=Float,Description="Variant Distance Bias for filtering splice-site artefacts in RNA-seq data (bigger is better)",Version=
> > ##INFO<ID=RPB,Number=1,Type=Float,Description="Mann-Whitney U test of Read Position Bias (bigger is better)">
> > ##INFO<ID=MQB,Number=1,Type=Float,Description="Mann-Whitney U test of Mapping Quality Bias (bigger is better)">
> > ##INFO<ID=BQB,Number=1,Type=Float,Description="Mann-Whitney U test of Base Quality Bias (bigger is better)">
> > ##INFO<ID=MQSB,Number=1,Type=Float,Description="Mann-Whitney U test of Mapping Quality vs Strand Bias (bigger is better)">
> > ##INFO<ID=SGB,Number=1,Type=Float,Description="Segregation based metric.">
> > ##INFO<ID=MQ0F,Number=1,Type=Float,Description="Fraction of MQ0 reads (smaller is better)">
> > ##FORMAT<ID=PL,Number=G,Type=Integer,Description="List of Phred-scaled genotype likelihoods">
> > ##FORMAT<ID=GT,Number=1,Type=String,Description="Genotype">
> > ##INFO<ID=ICB,Number=1,Type=Float,Description="Inbreeding Coefficient Binomial test (bigger is better)">
> > ##INFO<ID=HOB,Number=1,Type=Float,Description="Bias in the number of HOMs number (smaller is better)">
> > ##INFO<ID=AC,Number=A,Type=Integer,Description="Allele count in genotypes for each ALT allele, in the same order as listed">
> > ##INFO<ID=AN,Number=1,Type=Integer,Description="Total number of alleles in called genotypes">
> > ##INFO<ID=DP4,Number=4,Type=Integer,Description="Number of high-quality ref-forward , ref-reverse, alt-forward and alt-reverse bases">
> > ##INFO<ID=MQ,Number=1,Type=Integer,Description="Average mapping quality">
> > ##bcftools_callVersion=1.8+htslib-1.8
> > ##bcftools_callCommand=call --ploidy 1 -m -v -o results/bcf/SRR2584866_variants.vcf results/bcf/SRR2584866_raw.bcf; Date=Tue Oct  9 18:48:10 2018
> > ```
> >
> > Followed by information on each of the variations observed:
> >
> > ```
> > #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  results/bam/SRR2584866.aligned.sorted.bam
> > CP000819.1      1521    .       C       T       207     .       DP=9;VDB=0.993024;SGB=-0.662043;MQSB=0.974597;MQ0F=0;AC=1;AN=1;DP4=0,0,4,5;MQ=60
> > CP000819.1      1612    .       A       G       225     .       DP=13;VDB=0.52194;SGB=-0.676189;MQSB=0.950952;MQ0F=0;AC=1;AN=1;DP4=0,0,6,5;MQ=60
> > CP000819.1      9092    .       A       G       225     .       DP=14;VDB=0.717543;SGB=-0.670168;MQSB=0.916482;MQ0F=0;AC=1;AN=1;DP4=0,0,7,3;MQ=60
> > CP000819.1      9972    .       T       G       214     .       DP=10;VDB=0.022095;SGB=-0.670168;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,2,8;MQ=60      GT:PL
> > CP000819.1      10563   .       G       A       225     .       DP=11;VDB=0.958658;SGB=-0.670168;MQSB=0.952347;MQ0F=0;AC=1;AN=1;DP4=0,0,5,5;MQ=60
> > CP000819.1      22257   .       C       T       127     .       DP=5;VDB=0.0765947;SGB=-0.590765;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,2,3;MQ=60      GT:PL
> > CP000819.1      38971   .       A       G       225     .       DP=14;VDB=0.872139;SGB=-0.680642;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,4,8;MQ=60      GT:PL
> > CP000819.1      42306   .       A       G       225     .       DP=15;VDB=0.969686;SGB=-0.686358;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,5,9;MQ=60      GT:PL
> > CP000819.1      45277   .       A       G       225     .       DP=15;VDB=0.470998;SGB=-0.680642;MQSB=0.95494;MQ0F=0;AC=1;AN=1;DP4=0,0,7,5;MQ=60
> > CP000819.1      56613   .       C       G       183     .       DP=12;VDB=0.879703;SGB=-0.676189;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,8,3;MQ=60      GT:PL
> > CP000819.1      62118   .       A       G       225     .       DP=19;VDB=0.414981;SGB=-0.691153;MQSB=0.906029;MQ0F=0;AC=1;AN=1;DP4=0,0,8,10;MQ=59
> > CP000819.1      64042   .       G       A       225     .       DP=18;VDB=0.451328;SGB=-0.689466;MQSB=1;MQ0F=0;AC=1;AN=1;DP4=0,0,7,9;MQ=60      GT:PL
> > ```
> {: .code-out}
>
{: .hands_on}

This is a lot of information, so let's take some time to make sure we understand our output.

The first few columns represent the information we have about a predicted variation.

| column | info |
| ------- | ---------- |
| CHROM | contig location where the variation occurs |
| POS | position within the contig where the variation occurs |
| ID | a `.` until we add annotation information |
| REF | reference genotype (forward strand) |
| ALT | sample genotype (forward strand) |
| QUAL | Phred-scaled probability that the observed variant exists at this site (higher is better) |
| FILTER | a `.` if no quality filters have been applied, PASS if a filter is passed, or the name of the filters this variant failed |

In an ideal world, the information in the `QUAL` column would be all we needed to filter out bad variant calls.
However, in reality we need to filter on multiple other metrics.

The last two columns contain the genotypes and can be tricky to decode.

| column | info |
| ------- | ---------- |
| FORMAT | lists in order the metrics presented in the final column |
| results | lists the values associated with those metrics in order |

For our file, the metrics presented are GT:PL:GQ.

| metric | definition |
| ------- | ---------- |
| AD, DP | the depth per allele by sample and coverage |
| GT | the genotype for the sample at this loci. For a diploid organism, the GT field indicates the two alleles carried by the sample, encoded by a 0 for the REF allele, 1 for the first ALT allele, 2 for the second ALT allele, etc. A 0/0 means homozygous reference, 0/1 is heterozygous, and 1/1 is homozygous for the alternate allele. |
| PL | the likelihoods of the given genotypes |
| GQ | the Phred-scaled confidence for the genotype |

The Broad Institute's [VCF guide](https://www.broadinstitute.org/gatk/guide/article?id=1268) is an excellent place
to learn more about the VCF file format.

> <question-title></question-title>
>
> Use the `grep` and `wc` commands you have learned to assess how many variants are in the vcf file.
>
> > <solution-title></solution-title>
> >
> > > {% icon code-in%}
> > > ```bash
> > > $ grep -v "#" results/vcf/SRR2584866_final_variants.vcf | wc -l
> > > ```
> > {: .code-in}
> > > {% icon code-out%}
> > > ```
> > > 766
> > > ```
> > {: .code-out}
> >
> > There are 766 variants in this file.
> {: .solution}
{: .question}

> <hands-on-title>Assess the alignment (visualization) - optional step</hands-on-title>
>
> It is often instructive to look at your data in a genome browser. Visualization will allow you to get a "feel" for
> the data, as well as detecting abnormalities and problems. Also, exploring the data in such a way may give you
> ideas for further analyses.  As such, visualization tools are useful for exploratory analysis. In this lesson we
> will describe two different tools for visualization: a light-weight command-line based one and the Broad
> Institute's Integrative Genomics Viewer (IGV) which requires
> software installation and transfer of files.
>
> > <code-in-title> In order for us to visualize the alignment files, we will need to index the BAM file using `samtools`:</code-in-title>
> >
> > ```bash
> > $ samtools index results/bam/SRR2584866.aligned.sorted.bam
> > ```
> {: .code-in}
>
> ## Viewing with `tview`
>
> [Samtools](http://www.htslib.org/) implements a very simple text alignment viewer based on the GNU
> `ncurses` library, called `tview`. This alignment viewer works with short indels and shows [MAQ](http://maq.sourceforge.net/) consensus.
> It uses different colors to display mapping quality or base quality, subjected to users' choice. Samtools viewer is known to work with a 130 GB alignment swiftly. Due to its text interface, displaying alignments over network is also very fast.
>
> In order to visualize our mapped reads, we use `tview`, giving it the sorted bam file and the reference file:
>
> > <code-in-title>Visualize with `tview`</code-in-title>
> >
> > ```bash
> > $ samtools tview results/bam/SRR2584866.aligned.sorted.bam data/ref_genome/ecoli_rel606.fasta
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > 1         11        21        31        41        51        61        71        81        91        101       111       121
> > AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACTGGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATAC
> > ..................................................................................................................................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ..................N................. ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,........................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ..................N................. ,,,,,,,,,,,,,,,,,,,,,,,,,,,.............................
> > ...................................,g,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ....................................   ................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,....................................   ....................................      ,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ....................................  ,,a,,,,,,,,,,,,,,,,,,,,,,,,,,,,,     .......
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, .............................  ,,,,,,,,,,,,,,,,,g,,,,,    ,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ...........................T.......   ,,,,,,,,,,,,,,,,,,,,,,,c,          ......
> > ......................... ................................   ,g,,,,,,,,,,,,,,,,,,,      ...........................
> > ,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,,,,,,,,,,,,,,,,,,,       ..........................
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,   ................................T..  ..............................   ,,,,,,
> > ...........................       ,,,,,,g,,,,,,,,,,,,,,,,,   ....................................         ,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,, ....................................  ...................................        ....
> > ....................................  ........................  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,      ....
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,   ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ........................            .................................. .............................     ....
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,   ....................................        ..........................
> > ...............................       ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ....................................
> > ...................................  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ..................................
> > .................................... ,,,,,,,,,,,,,,,,,,a,,,,,,,,,,,,,,,,,        ,,,,,,,,,,,,,,,,,,,,,,,,,
> > ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,  ............................ ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
> > ```
> {: .code-out}
>
> The first line of output shows the genome coordinates in our reference genome. The second line shows the reference
> genome sequence. The third line shows the consensus sequence determined from the sequence reads. A `.` indicates
> a match to the reference sequence, so we can see that the consensus from our sample matches the reference in most
> locations. That is good! If that was not the case, we should probably reconsider our choice of reference.
>
> Below the horizontal line, we can see all of the reads in our sample aligned with the reference genome. Only
> positions where the called base differs from the reference are shown. You can use the arrow keys on your keyboard
> to scroll or type `?` for a help menu. To navigate to a specific position, type `g`. A dialogue box will appear. In
> this box, type the name of the "chromosome" followed by a colon and the position of the variant you would like to view
> (e.g. for this sample, type `CP000819.1:50` to view the 50th base. Type `Ctrl^C` or `q` to exit `tview`.
>
> > <question-title></question-title>
> >
> > Visualize the alignment of the reads for our `SRR2584866` sample. What variant is present at
> > position 4377265? What is the canonical nucleotide in that position?
> >
> > > <solution-title></solution-title>
> > >
> > > ```bash
> > > $ samtools tview ~/dc_workshop/results/bam/SRR2584866.aligned.sorted.bam ~/dc_workshop/data/ref_genome/ecoli_rel606.fasta
> > > ```
> > >
> > > Then type `g`. In the dialogue box, type `CP000819.1:4377265`.
> > > `G` is the variant. `A` is canonical. This variant possibly changes the phenotype of this sample to hypermutable. It occurs
> > > in the gene *mutL*, which controls DNA mismatch repair.
> > {: .solution}
> {: .question}
>
> ## Viewing with IGV
>
> [IGV](http://www.broadinstitute.org/igv/) is a stand-alone browser, which has the advantage of being installed locally and providing fast access. Web-based genome browsers, like [Ensembl](http://www.ensembl.org/index.html) or the [UCSC browser](https://genome.ucsc.edu/), are slower, but provide more functionality. They not only allow for more polished and flexible visualization, but also provide easy access to a wealth of annotations and external data sources. This makes it straightforward to relate your data with information about repeat regions, known genes, epigenetic features or areas of cross-species conservation, to name just a few.
>
>In order to use IGV, we will need to download the files, that we are going to use as input, to our local machine. We can do this easily via the Galaxy RStudio interface, in which we were working all along. 
>We will work on the lower right panel in the RStudio, on the "Files" tab, wehre we simply need to select the desired files by checking the boxes left to their name.
>Then, we click on the option "Export", which we can find on the said panel in the "More" dropdown tab. After assigning a name for the file to be downloaded we hit "Download" and our selection are localy downloaded. 
>The files we are going to use are listed below:
>
>~~~
>~/dc_workshop/results/bam/SRR2584866.aligned.sorted.bam 
>~/dc_workshop/results/bam/SRR2584866.aligned.sorted.bam.bai 
>~/dc_workshop/data/ref_genome/ecoli_rel606.fasta 
>~/dc_workshop/results/vcf/SRR2584866_final_variants.vcf 
>~~~
>
> Next, we need to open the IGV software. If you have not done so already, you can download IGV from the [Broad Institute's software page](https://www.broadinstitute.org/software/igv/download), double-click the `.zip` file
> to unzip it, and then drag the program into your Applications folder.
>
> 1. Open IGV.
> 2. Load our reference genome file (`ecoli_rel606.fasta`) into IGV using the **"Load Genomes from File..."** option under the **"Genomes"** pull-down menu.
> 3. Load our BAM file (`SRR2584866.aligned.sorted.bam`) using the **"Load from File..."** option under the **"File"** pull-down menu.
> 4.  Do the same with our VCF file (`SRR2584866_final_variants.vcf`).
>
> Your IGV browser should look like the screenshot below:
>
> ![IGV screenshot](../../images/bash-variant-calling/igv-screenshot.png)
>
> There should be two tracks: one coresponding to our BAM file and the other for our VCF file.
>
> In the **VCF track**, each bar across the top of the plot shows the allele fraction for a single locus. The second bar shows
> the genotypes for each locus in each *sample*. We only have one sample called here, so we only see a single line. Dark blue =
> heterozygous, Cyan = homozygous variant, Grey = reference.  Filtered entries are transparent.
>
> Zoom in to inspect variants you see in your filtered VCF file to become more familiar with IGV. See how quality information
> corresponds to alignment information at those loci.
> Use [this website](http://software.broadinstitute.org/software/igv/AlignmentData) and the links therein to understand how IGV colors the alignments.
>
{: .hands_on}

> <tip-title>BWA alignment options</tip-title>
> BWA consists of three algorithms: BWA-backtrack, BWA-SW and BWA-MEM. The first algorithm is designed for Illumina sequence
> reads up to 100bp, while the other two are for sequences ranging from 70bp to 1Mbp. BWA-MEM and BWA-SW share similar features such
> as long-read support and split alignment, but BWA-MEM, which is the latest, is generally recommended for high-quality queries as it
> is faster and more accurate.
{: .tip}


topic: data-science
tutorial: online-resources-gene
title: Learning about one gene across biological resources and formats
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["deutsch", "espa\u00f1ol", "italiano"]
content_hash: 1cbe6da6453b2b59cfd38d3535134eca
last_modified: 2026-05-27T09:01:47.963643
zenodo_link: https://zenodo.org/record/8304465
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/online-resources-gene/tutorial.html

content:
When we do a bioinformatics analysis, e.g. RNA-seq, we might end up with a list of gene names. We then need to explore these genes. But how can we do that? What are the resources available for that? And how to navigate through them?

The aim of this tutorial is to familiarize ourselves with that, using Human opsins as an example.

Human opsins are found in the cells of your retina. Opsins catch light and begin the sequence of signals that result in vision. We will proceed by asking questions about opsins and opsin genes, and then using different bioinformatics databases and resources to answer them.

> <comment-title></comment-title>
> This tutorial is a bit atypical: we will not work in Galaxy but mostly outside of it, navigating databases and tools through their own web interfaces. The scope of this tutorial is to illustrate several sources of biological data in different file formats, and representing different information.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
{: .agenda}

# Searching Human Opsins

To seach Human Opsins, we will start by checking the [NCBI Genome Data Viewer](https://www.ncbi.nlm.nih.gov/genome/gdv). The NCBI Genome Data Viewer (GDV) ({% cite rangwala2021accessing %}) is a genome browser supporting the exploration and analysis of annotated eukaryotic genome assemblies. The GDV browser displays biological information mapped to a genome, including gene annotation, variation data, BLAST alignments, and experimental study data from the NCBI GEO and dbGaP databases. GDV release notes describe new features relating to this browser. 

> <hands-on-title>Open NCBI Genome Data Viewer</hands-on-title>
>
> 1. Open the NCBI Genome Data Viewer at [www.ncbi.nlm.nih.gov/genome/gdv](https://www.ncbi.nlm.nih.gov/genome/gdv/)
>
{: .hands-on}

The homepage includes a simple "tree of life" where the human node is highlighted because it is the default organism to search. We can change that in the *Search organisms* box but we will leave for now as we are interested in Human Opsins. 

![Genome Data Viewer home page screenshot, the word "opsin" is written in the search box and the result is previewed.](./images/GenomeDataViewerpage.png "Genome Data Viewer home page")

The panel on the right reports multiple assemblies of the genome of interest, and a map of the chromosomes in that genome. We can search for Opsins there.

> <hands-on-title>Search for OpsinsOpen NCBI Genome Data Viewer</hands-on-title>
>
> 1. Type `opsin` in the *Search in genome* box
> 2. Click on the magnifier icon or press <kbd>Enter<kbd>
>
{: .hands-on}

Below the box is now displayed a table with genes related to opsin together with their names and location, i.e. the chromosome number, as well ass the start and end position

In the list of genes related to the search term opsin, there are the rhodopsin gene (RHO), and three cone pigments, short-, medium-, and long-wavelength sensitive opsins (for blue, green, and red light detection). There are other entities, e.g. a -LCR (Locus Control region), putative genes and receptors. 

Multiple hits are on the X chromosome, one of the sex-determining chromosomes.

> <question-title></question-title>
>
> 1. How many genes have been found in Chromosome X?
> 2. How many are protein coding genes?
>
> > <solution-title></solution-title>
> >
> > 1. The hits in ChrX are: 
> > - OPSIN-LCR
> > - OPN1LW
> > - OP1MW
> > - OPN1MW2
> > - OPN1MW3
> >
> > 2. By hovering over each gene, a box open and we can click on *Details* to learn more about each gene. Then we learn that the first (OPSIN-LCR) is not protein coding but a gene regulatory region and the other are protein coding genes. So there are 4 protein coding genes related to opsins in Chromosome X. In particular, Chromosome X includes one red pigment gene (OPN1LW) and three green pigment genes (OPN1MW, OPN1MW2 and OPN1MW3 in the reference genome assembly).
> >
> {: .solution}
{: .question}

Let's now focus on one specific opsin, the gene OPN1LW. 

> <hands-on-title>Open Genome Browser for gene OPN1LW</hands-on-title>
>
> 1. Click on the blue arrow that appears in the results table when you hover your mouse on the OPN1LW row
>
{: .hands-on}

You should have landed in [this page](https://www.ncbi.nlm.nih.gov/genome/gdv/browser/genome/?id=GCF_000001405.40), that is the genome view of gene OPN1LW. 

![Genome Data Viewer of gene OPN1LW, screenshot of the two main panels of the viewer, with chromosomes on the left and the feature viewer on the right.](./images/GenomeDataViewerofgeneOPN1LW.png "Genome Data Viewer of gene OPN1LW")

There is a lot of information in this page, let's focus on one section at the time. 

1. The Genome Data Viewer, on the top, tells us that we are looking at the data from the organism `Homo sapiens`, assembly `GRCh38.p14` and in particular at `Chr X` (Chromosome X). Each of these information has a unique ID.
2. The entire Chromosome is represented directly below, and the positions along the short (`p`) and long (`q`) arms are highlihgted.
3. Below, a blue box highlights that we are now focusing on the Region corresponding to the Gene `OPN1LW`. 

    There are multiple ways to interact with the viewer below. Try for example to hover with the mouse on the dots representing exons in the blue box. 

4. In the graph below, the gene requence is a green line with the exons (protein coding fragments) represented by green rectangles. 

    Hover with the mouse on the green line corresponding to `NM_020061.6` (our gene of interest) to get more detailed information.

    > <question-title></question-title>
    >
    > 1. What is the location of the OPN1LW segment?
    > 2. What is the length of the OPN1LW segment?
    > 3. What are introns and exons?
    > 4. How many exons and introns are in the OPN1LW gene?
    > 5. What is the total length of the coding region?
    > 6. What is the distribution between coding and non coding regions? What does that mean in term of biology?
    > 7. What is the lenght of the protein in number of amino acids?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. From 154,144,243 to 154,159,032
    > > 2. 1,4790 nucleotides, found at *Span on 14790 nt, nucleotides)*
    > > 3. Eukaryotic genes are often interrupted by non-coding regions called intervening sequences or introns. The coding regions are called exons. 
    > > 4. From this diagram, you can see that the OPN1LW gene consists of 6 exons and 5 introns, and that the introns are far larger than the exons. 
    > > 5. The CDS length is 1,095 nucleotides. 
    > > 6. Of the 14790 nt in the gene, only 1095 nt code for protein, which means that less than 8% of the base pairs contain the code. When this gene is expressed in cells in the human retina, an RNA copy of the entire gene is synthesized. Then the intron regions are cut out, and the exon regions joined together to produce the mature mRNA (a process called splicing). which will be translated by ribosomes as they make the red opsin protein. In this case, 92% of the initial RNA transcript is tossed out, leaving the pure protein code. 
    > > 7. The length of the resulting protein is 364 aa, amino acids.
    > {: .solution}
    {: .question}

But what is the sequence of this gene? There are multiple ways to retrieve this information, we will go through what we think is one of the most intituitive. 

> <hands-on-title>Open Genome Browser for gene OPN1LW</hands-on-title>
>
> 1. Click on the {% icon tool %} *Tools* section on the top right of the box showing the gene
> 2. Click on *Sequence Text View*
{: .hands-on}

This panel reports the DNA sequence of the introns (in green), as well as the one of the exons (in pink, including the translated protein sequence below). 

![Screenshot of the sequence view of the NHI resource, text is highlighted in different colors.](./images/SequenceTextView.png "Sequence Text View")

This sequence box is not showing the entire gene at the moment, but a subsequence of it. You can move upstream and downstream the genetic code with the arrows *Prev Page* and *Next Page*, or start from a specific position with the *Go To Position* button. We suggest to start with the start of the coding part of the gene, which as we learned earlier is at position 154,144,243. 

> <hands-on-title>Go to a specific position in Sequence View</hands-on-title>
>
> 1. Click on *Go To Position*
> 2. Type on `154144243`
> 
>    We have to remove the commas to validate the value
{: .hands-on}

The sequence highlighted in purple here signals a regulatory region. 

> <question-title></question-title>
>
> 1. What is the first amino acid of the resulting protein product?
> 2. What is the last one?
> 3. Can you keep a note of the first three and last three AAs of this protein?
>
> > <solution-title></solution-title>
> >
> > 1. The correspondent protein starts with Methionine, M (they all do).
> > 2. The last AA of the last exon (found in the 2nd page) is Alanine (A). After that, the stop codon TGA comes, which is not translated into an AA.
> > 3. The first three AAs are: M,A,Q; the last three: S,P,A.
> >
> {: .solution}
{: .question}

We can now close the *Sequence View*. 

From this resource, we can also get files, in different format, describing the gene. They are available from the *Download* section. 

1. *Download FASTA* will allow us to download the simplest file format to represent the nucleotide sequence of all the visible range of the genome (longer than the gene only).
2. *Download GenBank flat file* will allow us to access the annotation avaible on this page (and beyond) in a flat text format.
3. *Download Track Data* allows us to inpect two of the file formats we presented in the slides: the GFF (GFF3) and BED formats. If you change the tracks, each one may or may not be available. 

# Finding more information about our gene

Let's now get an overview of the information we have (in the literature) about our gene, using the NCBI resources

> <hands-on-title>Go to a specific position in Sequence View</hands-on-title>
>
> 1. Open the NCBI search at [www.ncbi.nlm.nih.gov/search](https://www.ncbi.nlm.nih.gov/search/)
> 2. Type `OPN1LW` in the *Search NCBI* search box
> 
{: .hands-on}

![Screenshot of the NIH result page, with cards named Literature, Genes, Proteins, genomes, Clinical and PubChem](./images/NIHresults.png "Results when searching for `OPN1LW` on NCBI").

## Literature

Let's start with the literature and in particular *PubMed* or *PubMed Central* results

> <details-title>What's the difference between PubMed and PubMed Central? </details-title>
>
> PubMed is a biomedical literature database which contains the abstracts of publications in the database. 
>
> PubMed Central is a full text repository, which contains the full text of publications in the database.
>
> While the exact number of hits may vary in time from the screenshot above, any gene name should have more hits in PubMed Central (searched in the full texts of publications) than in PubMed (searched only in the abstracts).  
>
{: .details}

> <hands-on-title>Open PubMed</hands-on-title>
>
> 1. Click on *PubMed* in the *Literature* box
> 
{: .hands-on}

You have entered PubMed, a free database of scientific literature, to the results of a complete search for articles directly associated with this gene locus. 

By clicking on the title of each article, you can see abstracts of the article. If you are on a university campus where there is online access to specific journals, you might also see links to full articles. PubMed is your entry point to a wide variety of scientific literature in the life sciences. On the left side of any PubMed page, you will find links to a description of the database, help, and tutorials on searching. 

> <question-title></question-title>
>
> 1. Can you guess which type of conditions are associated to this gene?
>
> > <solution-title></solution-title>
> >
> > 1. We will answer this question later
> >
> {: .solution}
{: .question}

> <hands-on-title>Back to NCBI Search page</hands-on-title>
>
> 1. Go back the [NCBI Search page](https://www.ncbi.nlm.nih.gov/search/all/?term=OPN1LW)
> 
{: .hands-on}

## Clinical

Let's now focus on the *Clinical* box, and specially on *OMIM*. OMIM, the Online Mendeliam Inheritance in Man (and woman!), is a catalog of human genes and genetic disorders. 

> <hands-on-title>Open OMIM</hands-on-title>
>
> 1. Click on *OMIM* in the *Clinical* box
> 
{: .hands-on}

Each OMIM entry is a genetic disorder (here mostly types of colorblindness) associated with mutations in this gene. 

> <hands-on-title>Read as much as your interest dictates</hands-on-title>
>
> 1. Follow links to get more information about each entry
> 
{: .hands-on}

> <comment-title>Read as much as your interest dictates</comment-title>
>
> For more information about OMIM itself, click the OMIM logo at the top of the page. Through OMIM, a wealth of information is available for countless genes in the human genome, and all information is backed up by references to the latest research articles.
> 
{: .comment}

How do variations in the gene affect the protein product, and its functions? Let's go back to the NIH page and investigate access the list of Single Nucleotide Polymorphisms (SNPs) that were detected by genetics studies in the gene. 

> <hands-on-title>Open dbSNP</hands-on-title>
>
> 1. Go back the [NCBI Search page](https://www.ncbi.nlm.nih.gov/search/all/?term=OPN1LW)
> 2. Click on *dbSNP* in the *Clinical* box
> 
{: .hands-on}

![Screenshot of the dbSNPs page about gene OPN1LW. Three main panels, the one on the left to filter the search based on tags, the central showing results, the right for a more detailed and programmatic search.](./images/dbSNPs.png "dbSNP in OPN1LW")

> <question-title></question-title>
>
> 1. What is the clinical significance of the rs5986963 and rs5986964 (first 2 variants listed at the time of creation of this tutorial)?
> 2. What is the functional consequence of rs104894912?
> 3. What is the functional consequence of rs104894913?
>
> > <solution-title></solution-title>
> >
> > 1. The Clinical significance is `benign` so it seems that they have no effect on the final protein product
> > 2. rs104894912 mutation leads to a `stop_gained` variant, which truncates the resulting protein too early and is therefore `pathogenic`
> > 3. rs104894913 mutation leads to a `missense_variant`, also `pathogenic`.
> >
> {: .solution}
{: .question}

Let's investigate about more on the rs104894913 variant

> <hands-on-title>Learn more about a variant dbSNP</hands-on-title>
>
> 1. Click on `rs104894913` to open its [dedicated page](https://www.ncbi.nlm.nih.gov/snp/rs104894913)
> 2. Click on *Clinical Significance* 
>
>    > <question-title></question-title>
>    > 
>    > What type of condition is associated with the rs104894913 variant? 
>    > 
>    > > <solution-title></solution-title>
>    > >
>    > > The name of the associated disease is "Protan defect". A quick internet search with your search engine will clarify that this is a type of color blindness.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. Click on the *Variant details*
>
>    > <question-title></question-title>
>    > 
>    > 1. Which substitution is associated with this variant?
>    > 2. What is impact of this subtitution in term of codon and amino acid?
>    > 3. At which position of the protein is this substitution?
>    > 
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The substitution `NC_000023.10:g.153424319G>A` corresponds to change from a Guanine (G) to an Adenine (A)
>    > > 2. This substitution change the codon `GGG`, a Glycine, into `GAG`, a Glutathione
>    > > 3. `p.Gly338Glu` means that the substitution is at position 338 of the protein.
>    > {: .solution}
>    {: .question}
{: .hands-on}

What does this mean this substitution for the protein? Let's have a deeper look at this protein. 

## Protein

> <hands-on-title>Open Protein</hands-on-title>
>
> 1. Go back the [NCBI Search page](https://www.ncbi.nlm.nih.gov/search/all/?term=OPN1LW)
> 2. Click on *Protein* in the *Proteins* box
> 3. Click on `OPN1LW – opsin 1, long wave sensitive` in the box on top
> 
{: .hands-on}

![Screenshot of the Opsin 1 NIH protein page, two main panels. The one on the left reporting information about the gene, the one on the right is a table of content and a series of links to other resources](./images/Opsin1NIH.png "Opsin 1 NIH protein page")

This page presents once again some data that we are familiar with (e.g. distribution of the exons along the gene sequence). 

> <hands-on-title>Download the protein sequences</hands-on-title>
>
> 1. Click on *Download Datasets*
> 2. Select 
>    - `Gene Sequences (FASTA)`
>    - `Transcript sequences (FASTA)`
>    - `Protein sequences (FASTA)`
> 3. Click on *Download* button
> 4. Open the downloaded ZIP file
{: .hands-on}

> <question-title></question-title>
>
> 1. What does the folder contain? 
> 2. Do you think they implemented good data practices?
>
> > <solution-title></solution-title>
> >
> > 1. The folder includes
> >    - a folder `ncbi_datasets` with different subfolders in it leadig some data files (multiple formats),
> >    - a `README.md` (a Markdown file), which is designed to "travel" together with the data and explain how was the data retreived, what is the structure of the data containing subfolder, and where to find extensive documentation. 
> > 2. It is definitely a good data management practice to guide users (not only your collaborators, but also yourself in the not-so-far future, when you will forget where does that file in your Downloads folder come from) to the data source and the data structure.
> >
> {: .solution}
{: .question}

# Searching by sequence

What could we do with these sequences that we just downloaded? Let's assume that we just sequenced the transcripts that we isolated through an experiment - so we know the sequence of our entity of interest, but don't know what it is. What we need to do in this case is to search the entire database of sequences known to science and match our unknown entity with an entry that has some annotation. Let's do it. 

> <hands-on-title>Search the protein sequence against all protein sequences</hands-on-title>
>
> 1. Open (with the simplest text editor you have installed) the `protein.faa` file that you just downloaded.
> 2. Copy its contents
> 3. Open BLAST [blast.ncbi.nlm.nih.gov](https://blast.ncbi.nlm.nih.gov/Blast.cgi)
> 4. Click on the `Protein BLAST, protein > protein`
> 
>     We will indeed use a protein sequence to search against a database of proteins
>
> 5. Paste the protein sequence into the big text box
> 6. Check the rest of parameters
> 7. Click the blue button `BLAST`
{: .hands-on}

This phase will take some time, there is afterall some server somewhere that is comparing the entirety of known sequences to your target.  When the search is complete, the result should look similar to the one below: 

![Screenshot of BLAST results, one big header on top and the results listed as a table on bottom](./images/BLASTresults.png "BLAST results")

> <hands-on-title>Graphic Summary of the protein sequences</hands-on-title>
>
> 1. Click on the tab *Graphic Summary*
{: .hands-on}

We access a box containing lots of colored lines. Each line represents a hit from your blast search. If you click on a red line, the narrow box just above the box gives a brief description of the hit. 

> <hands-on-title>Descriptions of the protein sequences</hands-on-title>
>
> 1. Click on the tab *Descriptions*
{: .hands-on}

> <question-title></question-title>
>
> 1. What is the first hit? Is it expected?
> 2. What are the other hits? For which organisms?
>
> > <solution-title></solution-title>
> >
> > 1. The first hit is our red opsin. That's encouraging, because the best match should be to the query sequence itself, and you got this sequence from that gene entry. 
> > 2. Other hits are other opsins. They include entries from other primates (e.g. `Pan troglogytes`). 
> >
> >
> {: .solution}
{: .question}

The hits are for our red opsin in human but also other opsins in other primates. We could want that, for example if we wanted to use this data to build a phylogenetic tree. If we instead are pretty sure that our sequence of interest is human, we could also have filtered the search only in human sequences.

> <hands-on-title>Filter a BLAST Search</hands-on-title>
>
> 1. Click on *Edit Search*
> 2. Type `Homo sapiens` in *Organism* field
> 3. Click the blue button `BLAST`
{: .hands-on}

With this new search, we find the other opsins (green, blue, rod-cell pigment) in the list. Other hits have lower numbers of matching residues. If you click on any of the colored lines in the *Graphic Summary*, you'll open more information about that hit, and you can see how much similarity each one has to the red opsin, our original query sequence. As you go down the list, each succeeding sequence has less in common with red opsin. Each sequence is shown in comparison with red opsin in what is called a pairwise sequence alignment. Later, you'll make multiple sequence alignments from which you can discern relationships among genes.

> <details-title>More details on BLAST scores</details-title>
>
> The displays contain two prominent measures of the significance of the hit:
>
> 1. the BLAST Score - lableled Score (bits)
>
>    The BLAST Score indicates the quality of the best alignment between the query sequence and the found sequence (hit). The higher the score, the better the alignment. Scores are reduced by mismatches and gaps in the best alignment. Calculation of the score is complex, involving a substitution matrix, which is a table that assigns a score to each pair of residues aligned. The most widely used matrix for protein alignment is known as BLOSUM62.
>
> 2. the Expectation Value (labeled Expect or E)
>
>     The expectation value E of a hit tells whether the hit is likely be result from chance likeness between hit and query, or from common ancestry of hit and query. () 
>
>     > <comment-title>Filter a BLAST Search</comment-title>
>     > 
>     > If E is smaller than $$10\mathrm{e}{-100}$$, it is sometimes given as 0.0.
>     {: .comment}
>
>     The expectation value is the number of hits you would expect to occur purely by chance if you searched for your sequence in a random genome the size of the human genome. 
>
>      $$E = 25$$ means that you could expect to find 25 matches in a genome of this size, purely by chance. So a hit with $$E = 25$$ is probably a chance match, and does not imply that the hit sequence shares common ancestry with your search sequence. 
>
>      Expectation values of around 0.1 may or may not be biologically significant (other tests would be needed to decide).
>
>      But very small values of E mean that the hit is biologically significant. The correspondence between your search sequence and this hit must arise from common ancestry of the sequences, because the odds are are simply too low that the match could arise by chance. For example, $$E = 10\mathrm{e}{-18}$$ for a hit in the human genome means that you would expect only one chance match in one billion billion different genomes the same size of the human genome.
>
>      The reason we believe that we all come from common ancestors is that massive sequence similarity in all organisms is simply too unlikely to be a chance occurrence. Any family of similar sequences across many organisms must have evolved from a common sequence in a remote ancestor.
>
{: .details}

> <hands-on-title>Dowloading </hands-on-title>
>
> 1. Click on *Descriptions* tab
> 2. Click at any sequence hit
> 3. Click on *Download*
> 4. Select `FASTA (aligned sequences)`
{: .hands-on}

It will download a new, slightly different, type of file: an aligned FASTA. If you want, explore it before the next section. 

While in the previous sections of this tutorial we extensively used the web interfaces of the tools (genomic viewers, quick literature scanning, reading annotations, etc.), this BLAST search is an example of a step that you could fully automate with Galaxy. 

> <hands-on-title> Similarity search with BLAST in Galaxy </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. Import the protein sequence via link from [Zenodo]({{ page.zenodo_link }}) or Galaxy shared data libraries:
>
>    ```text
>    {{ page.zenodo_link }}/files/protein.faa
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 1. {% tool [NCBI BLAST+ blastp](toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2) %} with the following parameters:
>    - *"Protein query sequence(s)"*: `protein.faa`
>    - *"Subject database/sequences"*: `Locally installed BLAST database`
>    - *"Protein BLAST database"*: `SwissProt`
>
>      To search against only annotated sequences in UniProt, we need select the latest release of `SwissProt`
>
>    - *"Set expectation value cutoff"*: `0.001`
>    - *"Output format"*: `Tabular (extended 25 columns)` 
>
{: .hands_on}

> <question-title></question-title>
>
> Do you think we are looking at exactly the same results as our original search for `opsin` in [www.ncbi.nlm.nih.gov/genome/gdv](https://www.ncbi.nlm.nih.gov/genome/gdv/)? Why?
>
> > <solution-title></solution-title>
> >
> > The results might be similar, but there are definitely some differences. Indeed, not only a text search is different than a sequence search in terms of method, but also this second round we started from the sequence of one specific opsin, so one branch of the entire protein family tree. Some of the family members are more similar between each other, so this type of search looks at the whole family from a quite biased perspective. 
> >
> {: .solution}
{: .question}

# More information about our protein

So far, we explored this information about opsins:
- how to know which proteins of a certain type exist in a genome,
- how to know where they are along the genome,
- how to get more information about a gene of interest,
- how to download their sequences in different formats,
- how to use these files to perform a similarity search.

You might be curious about how to know more about the proteins they code for, now. We have already collected some information (e.g. diseases associated), but in the next steps we will cross it with data about the protein structure, localisation, interactors, functions, etc.

The portal to visit to obtain all information about a protein is [UniProt](https://www.uniprot.org/). We can search it using a text search, or the gene or protein name. Let's go for our usual `OPN1LW` keyword. 

> <hands-on-title>Searching on UniProt</hands-on-title>
>
> 1. Open  [UniProt](https://www.uniprot.org/)
> 2. Type `OPN1LW` in the search bar
> 3. Select the card view
{: .hands-on}

The first hit should be `P04000 · OPSR_HUMAN`. Before opening the page, two things to notice: 

1. The name of the protein `OPSR_HUMAN` is different than the gene name, as well as their IDs are. 
2. This entry has a golden star, which means that was manually annotated and curated. 

> <hands-on-title>Open a result on UniProt</hands-on-title>
>
> 1. Click on `P04000 · OPSR_HUMAN`
{: .hands-on}

![Screenshot of the UniProt entry page header](./images/UniProt.png "UniProt page")

This is a long page with a lot of information, we designed an [entire tutorial]({% link topics/data-science/tutorials/online-resources-protein/tutorial.md %}) to go through it.

tools_json:
["NCBI BLAST+ blastp", "toolshed.g2.bx.psu.edu/repos/devteam/ncbi_blast_plus/ncbi_blastp_wrapper/2.10.1+galaxy2"]


topic: data-science
tutorial: sql-r
title: SQL with R
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["type: \"internal\"", "type: \"internal\""]
tags_json: ["SQL", "R"]
content_hash: 9b4f97eaf5782a17f5f3c23b6e159b94
last_modified: 2026-05-15T22:28:11.122629
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-r/tutorial.html

content:
In this tutorial you'll learn to use SQL via R. Some R and SQL experience is a pre-requisite.

> <comment-title></comment-title>
>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Databases and SQL](https://github.com/swcarpentry/sql-novice-survey/) lesson, which is licensed CC-BY 4.0.
>
> Abigail Cabunoc and Sheldon McKay (eds): "Software Carpentry: Using Databases and SQL."  Version 2017.08, August 2017,
> [github.com/swcarpentry/sql-novice-survey](https://github.com/swcarpentry/sql-novice-survey), [https://doi.org/10.5281/zenodo.838776](https://doi.org/10.5281/zenodo.838776)
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

For this tutorial we need to download a database that we will use for the queries.

```r
download.file("http://swcarpentry.github.io/sql-novice-survey/files/survey.db", destfile="survey.db")
```

# Programming with Databases - R

Let's have a look at how to access a database from
a data analysis language like R.
Other languages use almost exactly the same model:
library and function names may differ,
but the concepts are the same.

Here's a short R program that selects latitudes and longitudes
from an SQLite database stored in a file called `survey.db`:

```r
library(RSQLite)
connection <- dbConnect(SQLite(), "survey.db")
results <- dbGetQuery(connection, "SELECT Site.lat, Site.long FROM Site;")
print(results)
dbDisconnect(connection)
```

The program starts by importing the `RSQLite` library.
If we were connecting to MySQL, DB2, or some other database,
we would import a different library,
but all of them provide the same functions,
so that the rest of our program does not have to change
(at least, not much)
if we switch from one database to another.

Line 2 establishes a connection to the database.
Since we're using SQLite,
all we need to specify is the name of the database file.
Other systems may require us to provide a username and password as well.

On line 3, we retrieve the results from an SQL query.
It's our job to make sure that SQL is properly formatted;
if it isn't,
or if something goes wrong when it is being executed,
the database will report an error.
This result is a dataframe with one row for each entry and one column for each column in the database.

Finally, the last line closes our connection,
since the database can only keep a limited number of these open at one time.
Since establishing a connection takes time,
though,
we shouldn't open a connection,
do one operation,
then close the connection,
only to reopen it a few microseconds later to do another operation.
Instead,
it's normal to create one connection that stays open for the lifetime of the program.

Queries in real applications will often depend on values provided by users.
For example,
this function takes a user's ID as a parameter and returns their name:

```r
library(RSQLite)

connection <- dbConnect(SQLite(), "survey.db")

getName <- function(personID) {
  query <- paste0("SELECT personal || ' ' || family FROM Person WHERE id =='",
                  personID, "';")
  return(dbGetQuery(connection, query))
}

print(paste("full name for dyer:", getName('dyer')))

dbDisconnect(connection)
```

We use string concatenation on the first line of this function
to construct a query containing the user ID we have been given.
This seems simple enough,
but what happens if someone gives us this string as input?

```
dyer'; DROP TABLE Survey; SELECT '
```

It looks like there's garbage after the user's ID,
but it is very carefully chosen garbage.
If we insert this string into our query,
the result is:

```
SELECT personal || ' ' || family FROM Person WHERE id='dyer'; DROP TABLE Survey; SELECT '';
```

If we execute this,
it will erase one of the tables in our database.

This is called an SQL injection attack,
and it has been used to attack thousands of programs over the years.
In particular,
many web sites that take data from users insert values directly into queries
without checking them carefully first.
A very [relevant XKCD](https://xkcd.com/327/) that explains the
dangers of using raw input in queries a little more succinctly:

![A 4 panel comic, in the first panel a person is shown answering the phone, hearing that their son's school has some computer trouble. In panel 2 they apologises asking if their child broke something. In panel 3, the unseen person on the other end of the phone call asks if they really named their son Robert'); Drop table students;--? They respond saying 'oh yes. little bobby tables we call him.' In the 4th panel the caller says 'well we have lost this years student records, I hope you're happy.' They respond 'And I hope you've learned to sanitize your database inputs'.](../../images/xkcd/exploits_of_a_mom.png)

Since an unscrupulous parent might try to smuggle commands into our queries in many different ways,
the safest way to deal with this threat is
to replace characters like quotes with their escaped equivalents,
so that we can safely put whatever the user gives us inside a string.
We can do this by using a prepared statement
instead of formatting our statements as strings.
Here's what our example program looks like if we do this:

```r
library(RSQLite)
connection <- dbConnect(SQLite(), "survey.db")

getName <- function(personID) {
  query <- "SELECT personal || ' ' || family FROM Person WHERE id == ?"
  return(dbGetPreparedQuery(connection, query, data.frame(personID)))
}

print(paste("full name for dyer:", getName('dyer')))

dbDisconnect(connection)
```

The key changes are in the query string and the `dbGetQuery` call (we use dbGetPreparedQuery instead).
Instead of formatting the query ourselves,
we put question marks in the query template where we want to insert values.
When we call `dbGetPreparedQuery`,
we provide a dataframe
that contains as many values as there are question marks in the query.
The library matches values to question marks in order,
and translates any special characters in the values
into their escaped equivalents
so that they are safe to use.

> <question-title>Filling a Table vs. Printing Values</question-title>
>
> Write an R program that creates a new database in a file called
> `original.db` containing a single table called `Pressure`, with a
> single field called `reading`, and inserts 100,000 random numbers
> between 10.0 and 25.0.  How long does it take this program to run?
> How long does it take to run a program that simply writes those
> random numbers to a file?
{: .question}

> <question-title>Filtering in SQL vs. Filtering in R</question-title>
>
> Write an R program that creates a new database called
> `backup.db` with the same structure as `original.db` and copies all
> the values greater than 20.0 from `original.db` to `backup.db`.
> Which is faster: filtering values in the query, or reading
> everything into memory and filtering in R?
{: .question}

## Database helper functions in R

R's database interface packages (like `RSQLite`) all share
a common set of helper functions useful for exploring databases and
reading/writing entire tables at once.

To view all tables in a database, we can use `dbListTables()`:

```r
connection <- dbConnect(SQLite(), "survey.db")
dbListTables(connection)
```


To view all column names of a table, use `dbListFields()`:

```r
dbListFields(connection, "Survey")
```


To read an entire table as a dataframe, use `dbReadTable()`:

```r
dbReadTable(connection, "Person")
```


Finally to write an entire table to a database, you can use `dbWriteTable()`.
Note that we will always want to use the `row.names = FALSE` argument or R
will write the row names as a separate column.
In this example we will write R's built-in `iris` dataset as a table in `survey.db`.

```r
dbWriteTable(connection, "iris", iris, row.names = FALSE)
head(dbReadTable(connection, "iris"))
```

And as always, remember to close the database connection when done!

```r
dbDisconnect(connection)
```


topic: data-science
tutorial: python-plotting
title: Plotting in Python
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: a9990919fdf3f448ccd5b1095029fce0
last_modified: 2026-05-15T22:28:11.107291
zenodo_link: https://zenodo.org/record/3477564
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-plotting/tutorial.html

content:
In this lesson, we will be using Python 3 with some of its most popular scientific libraries. This tutorial assumes that the reader is familiar with the fundamentals of data analysis using the Python programming language, as well as, how to run Python programs using Galaxy. Otherwise, it is advised to follow the "Introduction to Python" and "Advanced Python" tutorials available in the same platform. We will be using JupyterNotebook, a Python interpreter that comes with everything we need for the lesson.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which is licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Plot data using matplotlib

For the purposes of this tutorial, we will use a file with the annotated differentially expressed genes that was produced in the [Reference-based RNA-Seq data analysis]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial.

Firstly, we read the file with the data.

```python
data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
print(data)
```

We can now use the `DataFrame.info()` method to find out more about a dataframe.

```python
data.info()
```

We learn that this is a DataFrame. It consists of 130 rows and 12 columns. None of the columns contains any missing values. 6 columns contain 64-bit floating point `float64` values, 2 contain 64-bit integer `int64` values and 4 contain character `object` values. It uses 13.2KB of memory.

We now have a basic understanding of the dataset and we can move on to creating a few plots and further explore the data. `matplotlib` is the most widely used scientific plotting library in Python, especially the `matplotlib.pyplot` module.

```python
import matplotlib.pyplot as plt
```
Simple plots are then (fairly) simple to create. You can use the `plot()` method and simply specify the data to be displayed in the x and y axis, by passing the data as the first and second argument. In the following example, we select a subset of the dataset and plot the P-value of each gene, using a lineplot.

```python
subset = data.iloc[121:, :]

x = subset['P-value']
y = subset['Gene name']

plt.plot(x, y)
plt.xlabel('P-value')
plt.ylabel('Gene name')
```

![A line chart is shown with a y axis of gene name and an x axis of p-value. The specific content of the graph is not important other than that it is produced with the expected x and y labels.](../../images/python-plotting/Figure9_Lineplot.png)

We use Jupyter Notebook and so running the cell generates the figure directly below the code. The figure is also included in the Notebook document for future viewing. However, other Python environments like an interactive Python session started from a terminal or a Python script executed via the command line require an additional command to display the figure.

Instruct matplotlib to show a figure:

```python
plt.show()
```
This command can also be used within a Notebook - for instance, to display multiple figures if several are created by a single cell.

If you want to save and download the image to your local machine, you can use the `plt.savefig()` command with the name of the file (png, pdf etc) as the argument. The file is saved in the Jupyter Notebook session and then you can download it. For example:

```python
plt.tight_layout()
plt.savefig('foo.png')
```

`plt.tight_layout()` is used to make sure that no part of the image is cut off during saving.

When using dataframes, data is often generated and plotted to screen in one line, and `plt.savefig()` seems not to be a possible approach. One possibility to save the figure to file is then to save a reference to the current figure in a local variable (with `plt.gcf()`) and then call the savefig class method from that variable. For example, the previous plot:

```python
subset = data.iloc[121:, :]

x = subset['P-value']
y = subset['Gene name']

fig = plt.gcf()
plt.plot(x, y)
fig.savefig('my_figure.png')
```



## More about plots

You can use the `plot()` method directly on a dataframe. You can plot multiple lines in the same plot. Just specify more columns in the x or y axis argument. For example:

```python
new_subset = data.iloc[0:10, :]
new_subset.loc[:, ['P-value', 'P-adj']].plot()
plt.xticks(range(0,len(new_subset.index)), new_subset['Gene name'], rotation=60)
plt.xlabel('Gene name')
```

![The graph is similar to the last image, now the x axis is the gene name, and the y axis runs from 0 to 7 with an annotation of 1e-65 above. Now there are two lines, one in red labelled p-adj, and one in blue labelled p-value.](../../images/python-plotting/Figure10_Multiple_lines_plot.png)

In this example, we select a new subset of the dataset, but plot only the two columns `P-value` and `P-adj`. Then we use the `plt.xticks()` method to change the text and the rotation of the x axis.

Another useful plot type is the barplot. In the following example we plot the number of genes that belong to the different chromosomes of the dataset.

```python
bar_data = data.groupby('Chromosome').size()
bar_data.plot(kind='bar')
plt.xticks(rotation=60)
plt.ylabel('N')
```
![It is now a bar plot with chromosome as an X axis and some values of N for the y axis for a few chromosomes.](../../images/python-plotting/Figure11_Barplot.png)

`matplotlib` supports also different plot styles from ather popular plotting libraries such as ggplot and seaborn. For example, the previous plot in ggplot style.

```python
plt.style.use('ggplot')
bar_data = data.groupby('Chromosome').size()
bar_data.plot(kind='bar')
plt.xticks(rotation=60)
plt.ylabel('N')
```
![The same graph as the previous, but with a different aethetic, the background is now light grey instead of white and the bars are red instead of blue to be a bit more like ggplot2 outputs.](../../images/python-plotting/Figure12_ggplot_Barplot.png)

You can also change different parameters and customize the plot.

```python
plt.style.use('default')
bar_data = data.groupby('Chromosome').size()
bar_data.plot(kind='bar', color = 'red', edgecolor = 'black')
plt.xticks(rotation=60)
plt.ylabel('N')
```

![The same graph again, but now the bars are red with a black border.](../../images/python-plotting/Figure13_Barplot_2.png)

Another useful type of plot is a scatter plot. In the following example we plot the Base mean of a subset of genes.

```python
scatter_data = data[['Base mean', 'Gene name']].head(n = 15)

plt.scatter(scatter_data['Gene name'], scatter_data['Base mean'])
plt.xticks(rotation = 60)
plt.ylabel('Base mean')
plt.xlabel('Gene name')
```

![A scatterplot is shown comparing gene name to base mean.](../../images/python-plotting/Figure14_Scatterplot.png)

> <question-title>Plotting</question-title>
>
>  Using the same dataset, create a scatterplot of the average P-value for every chromosome for the "+" and the "-" strand.
>
> > <solution-title></solution-title>
> >
> > First find the data and save it in a new dataframe. Then create the scatterplot. You can even go one step further and assign different colors for the different strands.Note the use of the `map` method that assigns the different colors using a dictionary as an input.
> >
> > ```
> > exercise_data = data.groupby(['Chromosome', 'Strand']).agg(mean_pvalue = ('P-value', 'mean')).reset_index()
> >
> > colors = {'+':'red', '-':'blue'}
> > plt.scatter(x = exercise_data['Chromosome'], y = exercise_data['mean_pvalue'], c = exercise_data['Strand'].map(colors))
> > plt.ylabel('Average P-value')
> > plt.xlabel('Chromosome')
> > ```
> > ![Another scatterplot showing chromosome vs average p-value, but every column has both a blue and red point, presumably showing the values for different strands.](../../images/python-plotting/Figure15_Exercise_plot.png)
> >
> {: .solution}
{: .question}


## Making your plots accessible
Whenever you are generating plots to go into a paper or a presentation, there are a few things you can do to make sure that everyone can understand your plots.

Always make sure your text is large enough to read. Use the `fontsize` parameter in `xlabel`, `ylabel`, `title`, and `legend`, and `tick_params` with `labelsize` to increase the text size of the numbers on your axes.
Similarly, you should make your graph elements easy to see. Use `s` to increase the size of your scatterplot markers and `linewidth` to increase the sizes of your plot lines.
Using `color` (and nothing else) to distinguish between different plot elements will make your plots unreadable to anyone who is colorblind, or who happens to have a black-and-white office printer. For lines, the `linestyle` parameter lets you use different types of lines. For scatterplots, `marker` lets you change the shape of your points.


topic: data-science
tutorial: python-types
title: Python - Basic Types & Type Conversion
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: eb3a45e44672bce5a29d089878c1bac6
last_modified: 2026-05-15T22:28:11.108911
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-types/tutorial.html

content:
Python is a typed language, data has a type, and different types of data cannot always be connected immediately and might need some conversion step before they can be used together. For instance if you add a number to a number, what should happen? If you add a number to a message, what do you expect will happen?

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Types

Every value in a program has a specific type.

Name                  | Python Code | Represents
---                   | ---         | ---
Integer               | `int`       | represents positive or negative whole numbers like 3 or -512.
Floating point number | `float`     | represents real numbers like 3.14159 or -2.5.
Character string      | `str`       | text, written with either `'` or `"` quotes (they must match)

### Checking the Type

Use the built-in function `type` to find out what type a value has. This works on values as well as variables. But remember: the *value* has the type --- the *variable* is just a label.

Check the type of values with the [`type()`](https://docs.python.org/3.8/library/functions.html#type) function:

```python
print(type(52))
print(type(3.14159))
```

You can also check the types of variables

```python
fitness = 'average'
print(type(fitness))
```

### Methods

A value's type determines what the program can do to it. Some operations may work

```python
print(5 - 3)
```

And some operations may not work:

```python
print('hello' - 'h')
```

For instance, you can use the `+` and `*` operators on strings.

```python
full_name = 'Ahmed' + ' ' + 'Walsh'
print(full_name)
separator = '=' * 10
print(separator)
```

Some methods only accept specific types, or only work on specific types.

The built-in function `len` returns the length of your data. Which of the following would you expect to work? `len(string)`? `len(int)`?

```python
print(len(full_name))
print(len(52))
```

### Matching Types

Not all types support all operations, adding an integer to a string doesn't make much sense:

```python
print(1 + '2')
```

This does not work because it's ambiguous: should `1 + '2'` be `3` (a number) or `'12'` (a string)? Some types can be converted to other types by using the type name as a function.

```python
print(1 + int('2'))
print(str(1) + '2')
```

### Operation Support

Here is a quick chart showing which operations are allowed for each pair:

Left\Right | `int`  | `float` | `str`
---------- | ---    | -----   | ---
`int`      | `+-*/` | `+-*/`  | `*`
`float`    | `+-*/` | `+-*/`  | ``
`str`      | `*`    | ``      | `+`

As you can see you can do `3 * "test"` and `"test" * 3`, but it doesn't work with floats.

## Can mix integers and floats freely in operations.

Integers and floating-point numbers can be mixed in arithmetic. Python 3 (which we use) automatically converts integers to floats as needed.

```python
print(f'half is {1 / 2.0}')
print(f'three squared is {3.0 ** 2}')
```


## Variables only change value when something is assigned to them.

If we make one cell in a spreadsheet depend on another, and update the latter,
the former updates automatically. However, this does **not** happen in programming languages.

```python
variable_one = 1
variable_two = 5 * variable_one
variable_one = 2
print(f'first is {variable_one} and second is {variable_two}')
```

The computer reads the value of `first` when doing the multiplication, creates
a new value, and assigns it to `second`. After that, `second` does not remember
where it came from. Every computation happens line-by-line.

> <question-title>Fractions</question-title>
>
> What type of value is 3.14159?
> How can you find out?
>
> > <solution-title></solution-title>
> >
> > It is a floating-point number (often abbreviated "float").
> > It is possible to find out by using the built-in function `type()`.
> >
> > ```
> > print(type(3.14159))
> > <class 'float'>
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!

```

> <question-title>Automatic Type Conversion</question-title>
>
> What type of value is the result of (3.25 + 4)?
>
> > <solution-title></solution-title>
> >
> > It is a float:
> > integers are automatically converted to floats as necessary.
> >
> > ```
> > result = 3.25 + 4
> > print(f'result is {type(result)}')
> > ```
> >
> > ```
> > 7.25 is <class 'float'>
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!

```

> <question-title>Choose a Type</question-title>
>
> What type of value (integer, floating point number, or character string)
> would you use to represent each of the following?  Try to come up with more than one good answer for each problem.  For example, in  # 1, when would counting days with a floating point variable make more sense than using an integer?
>
> 1. Number of days since the start of the year.
> 2. Time elapsed from the start of the year until now in days.
> 3. Serial number of a piece of lab equipment.
> 4. A lab specimen's age
> 5. Current population of a city.
> 6. Average population of a city over time.
>
> > <solution-title></solution-title>
> >
> > The answers to the questions are:
> > 1. Integer, since the number of days would lie between 1 and 365.
> > 2. Floating point, since fractional days are required
> > 3. Character string if serial number contains letters and numbers, otherwise integer if the serial number consists only of numerals
> > 4. This will vary! How do you define a specimen's age? whole days since collection (integer)? date and time (string)?
> > 5. Choose floating point to represent population as large aggregates (eg millions), or integer to represent population in units of individuals.
> > 6. Floating point number, since an average is likely to have a fractional part.
> {: .solution}
{: .question}

> <question-title>Division Types</question-title>
>
> In Python 3, the `//` operator performs integer (whole-number) floor division, the `/` operator performs floating-point
> division, and the `%` (or *modulo*) operator calculates and returns the remainder from integer division:
>
> ```python
> print(f'5 // 3: {5 // 3}')
> print(f'5 / 3: {5 / 3}')
> print(f'5 % 3: {5 % 3}')
> ```
>
> ```
> 5 // 3: 1
> 5 / 3: 1.6666666666666667
> 5 % 3: 2
> ```
>
> If `num_subjects` is the number of subjects taking part in a study,
> and `num_per_survey` is the number that can take part in a single survey,
> write an expression that calculates the number of surveys needed
> to reach everyone once.
>
> > <solution-title></solution-title>
> > We want the minimum number of surveys that reaches everyone once, which is
> > the rounded up value of `num_subjects/ num_per_survey`. This is
> > equivalent to performing a floor division with `//` and adding 1. Before
> > the division we need to subtract 1 from the number of subjects to deal with
> > the case where `num_subjects` is evenly divisible by `num_per_survey`.
> > ```
> > num_subjects = 600
> > num_per_survey = 42
> > num_surveys = (num_subjects - 1) // num_per_survey + 1
> >
> > print(num_subjects, 'subjects,', num_per_survey, 'per survey:', num_surveys)
> > ```
> >
> > ```
> > 600 subjects, 42 per survey: 15
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!
```

> <question-title>Strings to Numbers</question-title>
>
> Where reasonable, `float()` will convert a string to a floating point number,
> and `int()` will convert a floating point number to an integer:
>
> ```
> print("string to float:", float("3.4"))
> print("float to int:", int(3.4))
> ```
>
> ```
> string to float: 3.4
> float to int: 3
> ```
>
> If the conversion doesn't make sense, however, an error message will occur.
>
> > <code-in-title>Python</code-in-title>
> > ```
> > print("string to float:", float("Hello world!"))
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: could not convert string to float: 'Hello world!'
> > ```
> {: .code-out}
>
>
> Given this information, what do you expect the following program to do?
>
> What does it actually do?
>
> Why do you think it does that?
>
> ```
> print("fractional string to int:", int("3.4"))
> ```
>
> > <solution-title></solution-title>
> > What do you expect this program to do? It would not be so unreasonable to expect the Python 3 `int` command to
> > convert the string "3.4" to 3.4 and an additional type conversion to 3. After all, Python 3 performs a lot of other
> > magic - isn't that part of its charm?
> >
> > ```
> > int("3.4")
> > ```
> > However, Python 3 throws an error.
> > ```
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> > ValueError: invalid literal for int() with base 10: '3.4'
> > ```
> >
> > Why? To be consistent, possibly. If you ask Python to perform two consecutive
> > typecasts, you must convert it explicitly in code.
> >
> > ```
> > int(float("3.4"))
> > ```
> >
> > ```
> > 3
> > ```
> {: .solution}
{: .question}

```python
# Test out solutions here!
```

> <question-title>Arithmetic with Different Types</question-title>
>
> Which of the following will return the floating point number `2.0`?
> Note: there may be more than one right answer.
>
> ```
> first = 1.0
> second = "1"
> third = "1.1"
> ```
>
> 1. `first + float(second)`
> 2. `float(second) + float(third)`
> 3. `first + int(third)`
> 4. `first + int(float(third))`
> 5. `int(first) + int(float(third))`
> 6. `2.0 * second`
>
> > <solution-title></solution-title>
> >
> > Answer: 1 and 4 give exactly 2.0.
> > Answer 5 gives the value `2` which may be considered equivalent, but is not returning a float specifically.
> {: .solution}
{: .question}

```python
# Test out solutions here!
```


topic: data-science
tutorial: gnmx-lecture7
title: Versioning your code and data with git
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: baf2c61f1a15b82981842d5757111d23
last_modified: 2026-05-15T22:28:11.071889
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture7/tutorial.html

content:
[![XKCD1597](https://imgs.xkcd.com/comics/git.png)](https://xkcd.com/1597/)

# Setup and Introduction

## Lecture setup

1. Start [JupyterLab](https://mybinder.org/v2/gh/jupyterlab/jupyterlab-demo/try.jupyter.org?urlpath=lab)
2. Start terminal within JupyterLab instance

## Git resources

- [The Git Book](https://git-scm.com/book/en/v2)
- Troubleshooting [PG13](https://dangitgit.com/en) [R](https://ohshitgit.com/)

## Version control

- Manages changes over time
- Enables collaboration
- Provides complete history

## Git versus GitHub

- Git - version control system
- GitHub - hosting service

## Initial fundamentals

## Logic

Why do we need version control? Well ... to control versions and avoid mess:

![Why version control?](https://i.imgur.com/vVGsuDQ.png "From tutorials by <a href='https://www.kylebradbury.org/index.html'>Kyle Bradbury</a>")

## Main commands

Here are some of the most fundamental commands in `git` repertoire:

![Main commands](https://i.imgur.com/Xyhswin.png "From tutorials by <a href='https://www.kylebradbury.org/index.html'>Kyle Bradbury</a>")

## Branches

A repository may have multiple branches:

![Branch flow](https://i.imgur.com/vtXzgGO.png "From <a href='https://gitbetter.substack.com/p/how-to-work-in-multiple-git-branches'>GitBetter</a>")

# Let's do it!

## Clone a repo

Let's clone a [sample repo](https://github.com/nekrut/git_foo_bar) from GitHub

```bash
git clone https://github.com/nekrut/git_foo_bar.git
```

## Check history of this repo using `git log`

```bash
$git log
commit ee99d64890f3b1cf0637a14269ab8738357e8dd8 (HEAD -> main, origin/main, origin/HEAD)
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Mon Mar 11 17:38:42 2024 -0400

    Update README.md

commit de89f51d8e124665713f6fd94cd46447d172033b
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 08:02:50 2023 -0500

    Create file2.txt

commit bc3e5c4a9d54203739bb90f29d92e20082b9e5d4
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 08:02:25 2023 -0500

    Create file1.txt

commit 99dd43783be37b08c1ca80cdc881eae537526396
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 08:00:34 2023 -0500

    Updated readme

commit 18ebdabfe6f92f33ea626994ce6c9998cbe63522
Author: Anton Nekrutenko <anekrut@gmail.com>
Date:   Tue Feb 21 07:59:46 2023 -0500

    Initial commit
```

## Check status of the repo

```bash
$git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
```

## Let's change and stage a file

Use editor to modify `file1.txt`. Once it is saved, we can see what is happening by using `git status` again:

```bash
$git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")
```

You can see, that the file is modified, but it is *not staged*. To stage it for commit you need to explicitly add it to staging:

```bash
git add file1.txt 
```

If you run `git status` now you will get:

```bash
$git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/
```

## Oh &#128169; `get reset`

Running `git reset` will restore the *status quo* as it was prior to `git add`:

```bash
$git reset
Unstaged changes after reset:
M       file1.txt
```

and `git status` will look as it did prior to `git add`:

```bash
git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")
```

## Let's stage and commit

If you are indeed ready to go ahead let's `add`:

```bash
git add file1.txt
```

and `commit`

```bash
$git commit -m 'modified file1'
[main 476adb7] modified file1
 1 file changed, 1 insertion(+), 1 deletion(-)
```

If run `git log` you will this addition commit:

```bash
$git log
commit 476adb7b64a9330199fea13ca2e7523f7fe90189 (HEAD -> main)
Author: nekrut <anekrut@gmail.com>
Date:   Tue Feb 21 13:23:24 2023 +0000

    modified file1

```

or alternatively you can use `--oneline` flag:

```bash
$git log --oneline
476adb7 (HEAD -> main) modified file1
de89f51 (origin/main, origin/HEAD) Create file2.txt
bc3e5c4 Create file1.txt
99dd437 Updated readme
18ebdab Initial commit
```

## Oh &#128169; `get revert`

To roll everything back you can do this:

```bash
$git revert 476adb7
[main 1f13d5f] Revert "modified file1"
 1 file changed, 1 insertion(+), 1 deletion(-)
```

This will bring `vim` editor with a pre-filled rollback message:

![vim image](https://i.imgur.com/sEZNPN8.png)

To save this and get out:

- press <kbd>ESC</kbd>
- type `:wq`
- hit <kbd>Enter</kbd>

Now, let's look at the log (showing just two lat commits):

```bash
 git log
commit 1f13d5fcf38057db9a90066b99aa475a6eeb1bae (HEAD -> main)
Author: nekrut <anekrut@gmail.com>
Date:   Tue Feb 21 13:31:24 2023 +0000

    Revert "modified file1"
    
    This reverts commit 476adb7b64a9330199fea13ca2e7523f7fe90189.

commit 476adb7b64a9330199fea13ca2e7523f7fe90189
Author: nekrut <anekrut@gmail.com>
Date:   Tue Feb 21 13:23:24 2023 +0000

    modified file1
```

## Let's actually do make changes and highlight them with diff

First, let's modify, say, `file2.txt` by adding a line. In this example I modified it from:

```
This
is
another
file
I've
made
```

to 

```
This
is
file2
I've
made
```

Now `add` and `commit`:

```bash
$ git status
On branch main
Your branch is ahead of 'origin/main' by 2 commits.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file2.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")

$ git add file2.txt 

$ git commit -m 'changed file2'
[main a947686] changed file2
 1 file changed, 1 insertion(+), 2 deletions(-)
```

Let's now compare the changes between two commits:

```bash
$ git diff a947686 1f13d5f
diff --git a/file2.txt b/file2.txt
index 1f383ac..e13e461 100644
--- a/file2.txt
+++ b/file2.txt
@@ -1,5 +1,6 @@
 This
 is
-file2
+another
+file
 I've
 made
```
 
 here:
 
 - `a` and `b` = tags of files being compared
 - `---` and `+++` marked to indiciate differences between `a` and `b`
 - `@@ -1,5 +1,6 @@` file chunk header which has the following format:
     - `@@ [file a range][file b range] @@`
     - File ranges are: `<start line><number of lines>`

## Branches

![git branches](https://i.imgur.com/cLCQ650.png)

> From [GitBetter](https://gitbetter.substack.com/p/how-to-work-in-multiple-git-branches)

To enable collaborations and to give the ability to develop major features without disrupting production (`master` or `main`) branch Git allows creation of multiple branches.  

### Create a new branch and switch to it

Let's create branch dev:

```bash
$ git branch dev
```

to see existing branches:

```bash
$ git branch
  dev
* main
```

The `main` branch is active (tagged with `*`). To actually switch branches:

```bash
$ git checkout dev
Switched to branch 'dev'
```

### Make changes

Let's make some changes, say, to `file1.txt` from this:

```
This
is
a 
file
```

to this:

```
This
is
a 
file1
```

and then get `status`, `add`, and `commit`:

```bash
$ git status
On branch dev
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file1.txt

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .ipynb_checkpoints/

no changes added to commit (use "git add" and/or "git commit -a")

$ git add file1.txt 

$ git commit -m 'changes to file1.txt'
[dev 05ce64d] changes to file1.txt
 1 file changed, 1 insertion(+), 1 deletion(-)
 ```

### Switch to another branch and look at the modified file

If we switch back to `main` and look at the content of `file1.txt` we will see the "old" content:

```
This
is
a 
file
```

### Merge branches

To incorporate changes from `dev` to `master` we need to do a merge:

```bash
$ git merge dev main
Updating a947686..05ce64d
Fast-forward
 file1.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
```

and if we look at `file1.txt` again we will see the change:

$ more file1.txt 
This
is
a 
file1

We can now delete the branch:

```bash
$ git branch -d dev
Deleted branch dev (was 05ce64d).
```

### Merging with conflicts

To simulate a merging conflict we need to make different changes to the same line of a file in different branches. 

#### Modify `file1.txt` in `main`

First, let's checkout `main` and make the following change to `file1.txt`, say, from 

```
This
is
a 
file1
```

to 

```
This
is
a 
file1 - the first file we created
```

now we add and commit:

```bash
$ git add file1.txt 

$ git commit -m 'main changes to file1'
[main dfd0598] main changes to file1
 1 file changed, 1 insertion(+), 1 deletion(-)
```

> <tip-title>Showing differences between branches</tip-title>
>
>You can use `git diff` to see the differences between branches such as, for example:
>
> ```bash
> $ git diff main dev
> diff --git a/file1.txt b/file1.txt
> index f8e8191..e82ee85 100644
> --- a/file1.txt
> +++ b/file1.txt
> @@ -1,4 +1,4 @@
>  This
>  is
>  a 
> -file1 - the first file we created
> +file1
> ```
{: .tip}

#### Modify `file1.txt` in `dev`

Checkout `dev`:

```bash
$ git checkout dev
Switched to branch 'dev'
```

Change `file1.txt`, say, from:


```
This
is
a 
file1
```

to

```
This
is
a 
file1 - the first file
```

now we add and commit:

```bash
$ git add file1.txt 

$ git commit -m 'dev changes to file1'
[dev b2b8b9e] dev changes to file1
 1 file changed, 1 insertion(+), 1 deletion(-)
 ```

#### Checkout `main` and try to `merge`


```bash
 $ git merge main dev
 Auto-merging file1.txt
 CONFLICT (content): Merge conflict in file1.txt
 Automatic merge failed; fix conflicts and then commit the result.
```

If then actually look at the content of the file, you will see this:

![file1](https://i.imgur.com/s1zuCN6.png)


so here you need to decide which version you will keep and edit the file correspondingly.  For example, I edited it to this form and saved:

![file1_edited](https://i.imgur.com/1IbIQmD.png)

Now I need to `add` and `commit`:

```bash
 $ git add file1.txt
 jovyan@jupyter-jupyterlab-2djupyterlab-2ddemo-2dryet2dsd:~/bmmb554_foo_bar$ git commit -m 'merged dev into main'
```


topic: data-science
tutorial: python-subprocess
title: Python - Subprocess
difficulty: intermediate
time_estimation: 45M
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: ef5c726c7ee961fdbbbc85b7eb26f45b
last_modified: 2026-05-15T22:28:11.108222
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-subprocess/tutorial.html

content:
Sometimes you need to run other tools in Python, like maybe you want to
Here we'll give a quick tutorial on how to read and write files within Python.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Subprocesses

Programs can run other programs, and in Python we do this via the `subprocess` module. It lets you run any other command on the system, just like you could at the terminal.

The first step is importing the module

```python
import subprocess
```

You'll primarily use two functions:

```python
help(subprocess.check_call)
```

Which executes a command and checks if it was successful (or it raises an exception), and

```python
help(subprocess.check_output)
```


# Check Call: Downloading Files

Which executes a command returns the output of that command. This is really useful if you're running a subprocess that writes something to stdout, like a report you need to parse. We'll learn how to use these by running two gene callers, augustus and glimmer. You can install both from Conda if you do not have them already.

```
conda create -n subprocess augustus glimmer3
```

Additionally you'll need two files, you *generally should not do this*, but you can use a subprocess to download the file! We'll use `subprocess.check_call` for this which simply executes the program, and continues on. If there is an error in the execution, it will raise an exception and stop execution.

```python
url = "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/836/945/GCF_000836945.1_ViralProj14044/GCF_000836945.1_ViralProj14044_genomic.fna.gz"
genome = 'Escherichia virus T4.fna.gz'
subprocess.check_call(['wget', url, '-O', genome])
subprocess.check_call(['gzip', '-d', genome])
```

> <tip-title>What do these commands look like on the CLI?</tip-title>
> ```
> wget https://ftp.ncbi.nlm.nih.gov/.... -O "Escherichia virus T4.fna.gz"
> gzip -d "Escherichia virus T4.fna.gz"
> ```
{: .tip}

The above segment
- sets a url variable
- sets an output filename, `Escherichia virus T4.fna.gz`
- runs `check_call` with a single argument: a list
    - `wget` a tool we use to download files
    - the URL
    - `-O` indicating the next argument will be the 'output name'
    - what we want the output filename to be called
- runs `check_call` with a single argument: a list
    - `gzip` a tool to decompress files
    -`-d` indicating we want to decompress
    - and the filename.

This list is especially important. When you run commands on the command line, normally you just type in a really bit of text by yourself. It's one big string, and you're responsible for making sure quotation marks appear in the right place. For instance, if you have spaces in your filenames, you have to quote the filename. Python requires you specify a list of arguments, and then handles the quoting for you! Which, honestly, is easier and safer.

> > <code-in-title>Terminal</code-in-title>
> > Here we manually quote the argument
> > ```
> > glimmer3 "bow genome.txt"
> > ```
> {: .code-in}
> > <code-out-title>Python</code-out-title>
> > Here python handles that for us
> > ```
> > subprocess.check_call(['glimmer3', 'bow genome.txt'])
> > ```
> {: .code-out}
{: .code-2col}

> <tip-title>Exploitation!</tip-title>
> This is one of the major reasons we don't use `os.system` or older Python interfaces for running commands.
> If you're processing files, and a user supplies a file with a space, if your program isn't expecting that space in that filename, then it could do something dangerous!
> Like exploit your system!
>
> So, **always** use `subprocess` if you run to commands, never any other module, *despite what you see on the internet!*
{: .tip}


There are more functions in the module, but the vast majority of the time, those are sufficient.

# Check Output: Gene Calling with Augustus

```python
gff3 = subprocess.check_output([
    'augustus', # Our command
    '--species=E_coli_K12', # the first argument, the species, we're using a phage so we call genes  based on it's host organism
    'Escherichia virus T4.fna', # The path to our genome file, without the .gz because we decompressed it.
    '--gff3=on' # We would like gff3 formatted output (it's easier to parse!)
])

gff3 = gff3.decode('utf-8')
gff3 = gff3.split('\n')
```

> <tip-title>What does this commands look like on the CLI?</tip-title>
> ```
> augustus --species=E_coli_K12 'Escherichia virus T4.fna' --gff3=on
> ```
{: .tip}

If you're using `subprocess.check_output()` python doesn't return plain text `str` to you, instead it returns a `bytes` object. We can decode that into text with `.decode('utf-8')`, a phrase you should memorise as going next to `check_output()`, for 99% of use cases.

Let's look at the results!

```python
print(gff3[0:20])
```

It's a lot of comment lines, starting with `#`. Let's remove those

```python
cleaned_gff3 = []
for line in gff3:
    if line.startswith('#'):
        continue
    cleaned_gff3.append(line)

print(cleaned_gff3[0:20])
```

And now you've got a set of gff3 formatted gene calls! You can use all of your loop processing skills to slice and dice this data into something great!

# Aside: `stdin`, `stderr`, `stdout`

All unix processes have three default file handles that are available to them:

- `stdin`, where data is passed to the program via a pipe. E.g. `generate-data | my-program`, there the program would read the output of `generate-data` from the pipe.
- `stdout`, the default place where things are written. E.g. if you `print()` in python, it goes to `stdout`. People often redirect `stdout` to a file, like `my-program > output.txt` to save the output.
- `stderr`, *generally* if your program produces output on `stdout`, you might still want to log messages (errors, % done, etc.) If you write to `stdout`, it might get mixed in with the user's outputs, so we write to `stderr`, which also gets printed to the screen, and looks identical as any print statement, but it's coming from a separate pipe.

# Pipes

One of the more complicated cases, however, is when you need pipes.

```python
url = "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/001/721/125/GCF_001721125.1_ASM172112v1/GCF_001721125.1_ASM172112v1_cds_from_genomic.fna.gz"
cds = 'E. Coli CDSs.fna.gz'
subprocess.check_call(['wget', url, '-O', cds])
subprocess.check_call(['gzip', '-d', cds])
```

With subprocesses, you can control the stdin, and stdout of the process by using file handles.

> > <code-in-title>Terminal</code-in-title>
> > Here we pipe a file to a process named `build-icm` which takes one argument, the output name. It reads sequences from stdin.
> > ```
> > cat seq.fa | build-icm test.icm
> > # OR
> > build-icm test.icm < seq.fa
> > ```
> {: .code-in}
> > <code-out-title>Python</code-out-title>
> > Here we need to do a bit more.
> > 1. Open a file handle
> > 2. Pass that file handle to `check_call` or `check_output`. This determines where stdin comes from.
> > ```
> > with open('seq.fa', 'r') as handle:
> >     subprocess.check_call(['build-icm', 'test.icm'], stdin=handle)
> > ```
> {: .code-out}
{: .code-2col}

We'll do that now:

```python
with open('E. Coli CDSs.fna', 'r') as handle:
    subprocess.check_call(['build-icm', 'test.icm'], stdin=handle)
```

> <tip-title>What does this commands look like on the CLI?</tip-title>
> ```
> build-icm test.icm < 'E. Coli CDSs.fna'
> ```
{: .tip}

Here we build a model, based on the sequences of *E. Coli* K-12, that Glimmer3 can use.

```python
output = subprocess.check_output([
    'glimmer3', # Our program
    'Escherichia virus T4.fna', # The input genome
    'test.icm', # The model we just built
    't4-genes'  # The base name for output files. It'll produce t4-genes.detail and t4-genes.predict.
]).decode('utf-8') # And of course we decode as utf-8

print(output)
```

> <tip-title>What does this commands look like on the CLI?</tip-title>
> ```
> glimmer3 'Escherichia virus T4.fna' test.icm t4-genes
> ```
{: .tip}

*What happened here?* The output of the program was written to `stderr`, not `stdout`, so Python may print that out to your screen, but `output` will be empty. To solve this common problem we can re-run the program and collect both `stdout` and `stderr`.

```python
output = subprocess.check_output([
    'glimmer3', # Our program
    'Escherichia virus T4.fna', # The input genome
    'test.icm', # The model we just built
    't4-genes'  # The base name for output files. It'll produce t4-genes.detail and t4-genes.predict.
], stderr=subprocess.STDOUT).decode('utf-8') # And of course we decode as utf-8

print(output)
```

Here we've re-directed the `stderr` to `stdout` and mixed both of them together. This isn't always what we want, but here the program produces no output, and we can do that safely, and now we can parse it or do any other computations we need with it! Our Glimmer3 gene calls are in `t4-genes.detail` and `t4-genes.predict` if we want to open and process those as well.


topic: data-science
tutorial: r-basics
title: R basics in Galaxy
difficulty: introductory
time_estimation: 3H
tools_json: ["RStudio", "interactive_tool_rstudio"]
requirements_json: []
tags_json: ["interactive-tools", "R"]
content_hash: f0fc10dbca7947fe1d0d7399bde1d4a1
last_modified: 2026-05-15T22:28:11.114188
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/r-basics/tutorial.html

content:
This tutorial will introduce R basics, using an RStudio Interactive Tool in Galaxy

{% include topics/data-science/tutorials/r-basics/tutorial_origin.md %}
{% include topics/data-science/tutorials/r-basics/r_introduction.md %}
{% include topics/data-science/tutorials/r-basics/r_history.md %}
{% include topics/data-science/tutorials/r-basics/r_advantages.md %}
{% include topics/data-science/tutorials/r-basics/disclaimer.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Before diving in the tutorial, we need to open {% tool [RStudio](interactive_tool_rstudio) %}. If you do not know how or never interacted with RStudio, please follow the [dedicated tutorial]({% link topics/galaxy-interface/tutorials/rstudio/tutorial.md %}).

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

# Creating objects in R

> <comment-title>Reminder</comment-title>
> At this point you should be coding along in the `r_basics.R` script we created in the last episode. Writing your commands in the script(and commenting it) will make it easier to record what you did and why.
{: .comment}

What might be called a variable in many languages is called an **object** in R.

To create an object you need:

- a name (e.g. `a`)
- a value (e.g. `1`)
- the assignment operator (`<-`)

> <hands-on-title>Create a first object</hands-on-title>
>
> 1. Assign `1` to the object `a` using the R assignment operator `<-` in your script
> 2. Write a comment in the line above
>
>    ```R
>    # this line creates the object 'a' and assigns it the value '1'
>    a <- 1
>    ```
>
> 3. Select the lines
> 4. Execute them
>
>    > <tip-title>Execute from a script</tip-title>
>    > - Click on the **Run the current line or selection**
>    > - Type <kbd>CTRL</kbd>+<kbd>Enter</kbd> (or <kbd>CMD</kbd>+<kbd>Enter</kbd>)
>    {: .tip}
>
> 5. Check the **Console** and **Environment** panels
>
{: .hands_on}

The **Console** displays the lines of code run from the script and any outputs or status/warning/error messages (usually in red).

In the **Environment**, we have now a table:

Values | |
--- | ---
 a | 1

This **Environment** window allows you to keep track of the objects you have created in R.

> <question-title>Exercise: Create some objects in R</question-title>
>
> Create the following objects, with an appropriate name (your best guess at what name to use is fine):
>
> 1. Create an object that has the value of number of pairs of human chromosomes
> 2. Create an object that has a value of your favorite gene name
> 3. Create an object that has this URL as its value (`ftp://ftp.ensemblgenomes.org/pub/bacteria/release-39/fasta/bacteria_5_collection/escherichia_coli_b_str_rel606/`)
> 4. Create an object that has the value of the number of chromosomes in a diploid human cell
>
> > <solution-title></solution-title>
> >
> > Here as some possible answers to the challenge:
> >
> > 1. `human_chr_number <- 23`
> > 2. `gene_name <- 'pten'`
> > 3. `ensemble_url <- 'ftp://ftp.ensemblgenomes.org/pub/bacteria/release-39/fasta/bacteria_5_collection/escherichia_coli_b_str_rel606/'`
> > 4. `human_diploid_chr_num <- 46`
> >
> {: .solution}
{: .question}

#  Naming objects in R

Here are some important details about naming objects in R.

- **Avoid spaces and special characters**

    Object names cannot contain spaces or the minus sign (`-`). You can use `_` to make names more readable. You should avoid using special characters in your object name (e.g. `!`, `@`, `#`, `.` , etc.). Also, object names cannot begin with a number.

- **Use short, easy-to-understand names**

    You should avoid naming your objects using single letters (e.g. `n`, `p`, etc.). This is mostly to encourage you to use names that would make sense to anyone reading your code (a colleague, or even yourself a year from now). Also, avoiding excessively long names will make your code more readable.

- **Avoid commonly used names**

    There are several names that may already have a definition in the R language (e.g. `mean`, `min`, `max`). One clue that a name already has meaning is that if you start typing a name in RStudio and it gets a colored highlight or RStudio gives you a suggested autocompletion you have chosen a name that has a reserved meaning.

- **Use the recommended assignment operator**

    In R, we use `<-` as the preferred assignment operator. `=` works too, but is most commonly used in passing arguments to functions (more on functions later). There is a shortcut for the R assignment operator:
    - Windows execution shortcut: <kbd>Alt</kbd>+<kbd>-</kbd>
    - Mac execution shortcut: <kbd>Option</kbd>+<kbd>-</kbd>

There are a few more suggestions about naming and style you may want to learn more about as you write more R code. There are several "style guides" that have advice, and one to start with is the [tidyverse R style guide](http://style.tidyverse.org/index.html).

> <comment-title>Pay attention to warnings in the script console</comment-title>
>
> If you enter a line of code in your script that contains an error, RStudio may give you an error message and underline this mistake. Sometimes these messages are easy to understand, but often the messages may need some figuring out. Paying attention to these warnings will help you avoid mistakes.
>
> In the example below, the object name has a space, which is not allowed in R. The error message does not say this directly, but R is "not sure" about how to assign the name to `human_ chr_number` when the object name we want is `human_chr_number`.
>
> ![rstudio script warning](../../images/r-basics/rstudio_script_warning.png)
{: .comment}

# Reassigning object names or deleting objects

Once an object has a value, you can change that value by overwriting it. R will not give you a warning or error if you overwriting an object, which may or may not be a good thing depending on how you look at it.

> <hands-on-title>Overwrite an object</hands-on-title>
>
> 1. Overwrite the `gene_name` with the `tp53`
>
>    ```R
>    # gene_name has the value 'pten' or whatever value you used in the challenge.
>    # We will now assign the new value 'tp53'
>    gene_name <- 'tp53'
>    ```
>
> 2. Check the new value in the **Environment** panel
> 3. Type `gene_name` in the **Script** panel
> 4. Check in the **Console** panel
>
>    ```R
>    > gene_name
>    [1] "tp53"
>    ```
>
>    If you run a line of code that has only an object name, R will normally display the contents of that object.
{: .hands_on}

You can also remove an object from R's memory entirely

> <hands-on-title>Remove an object</hands-on-title>
>
> 1. Delete the `gene_name` object
>
>    ```R
>    # delete the object 'gene_name'
>    rm(gene_name)
>    ```
>
> 2. Check that `gene_name` is not displayed in the **Environment** panel
> 3. Type `gene_name` only in **Script** and check the **Console**
>
>    ```R
>    > gene_name
>    Error: object 'gene_name' not found
>    ```
>
>    R tells us that the object no longer exists.
{: .hands_on}

# Understanding object data types or modes

In R, **every object has two properties**:

- **Length**: how many distinct values are held in that object
- **Mode**: what is the classification (type) of that object.

    The **"mode" property** corresponds to the **type of data an object represents**. The most common modes you will encounter in R are:

    - **Numeric (num)**: numbers such floating point/decimals (1.0, 0.5, 3.14)

        > <comment-title>More specific types</comment-title>
        > There are also more specific numeric types (dbl - Double, int - Integer). These differences are not relevant for most beginners and pertain to how these values are stored in memory
        {: .comment}

    - **Character (chr)**: a sequence of letters/numbers in single `''` or double `" "` quotes
    - **Logical**: boolean values, `TRUE` or `FALSE`

    There are a few other modes (i.e. "complex", "raw" etc.) but these are the three we will work with in this lesson.

Data types are familiar in many programming languages, but also in natural language where we refer to them as the parts of speech, e.g. nouns, verbs, adverbs, etc. Once you know if a word - perhaps an unfamiliar one - is a noun, you can probably guess you can count it and make it plural if there is more than one (e.g. 1 [Tuatara](https://en.wikipedia.org/wiki/Tuatara), or 2 Tuataras). If something is a adjective, you can usually change it into an adverb by adding "-ly" (e.g. [jejune](https://www.merriam-webster.com/dictionary/jejune) vs. jejunely). Depending on the context, you may need to decide if a word is in one category or another (e.g "cut" may be a noun when it's on your finger, or a verb when you are preparing vegetables). These concepts have important analogies when working with R objects.

> <hands-on-title>Create an object and check its mode</hands-on-title>
>
> 1. Assign `'chr02'` to a `chromosome_name` object
>
>    ```R
>    chromosome_name <- 'chr02'
>    ```
>
> 2. Check the mode of the object
>
>    ```R
>    mode(chromosome_name)
>    ```
>
> 3. Check the result in the console
>
{: .hands_on}

The created object seems to a character object.

> <question-title>Create objects and check their modes</question-title>
>
> 1. Create the following objects in R
>    1. `od_600_value` with value `0.47`
>    2. `chr_position` with value `'1001701'`
>    3. `spock` with value `TRUE`
>    4. `pilot` with value `Earhart`
> 2. Guess their mode
> 3. Check them using `mode()`
>
> > <solution-title></solution-title>
> >
> > 1. Object creation
> >
> >    ```R
> >    > od_600_value <- 0.47
> >    > chr_position <- '1001701'
> >    > spock <- TRUE
> >    > pilot <- Earhart
> >    [1] Error in eval(expr, envir, enclos): object 'Earhart' not found
> >    ```
> >
> >    We cannot take a string of alphanumeric characters (e.g. `Earhart`) and assign as a value for an object. In this case, R looks for an object named `Earhart` but since there is no object, no assignment can be made.
> >
> > 2. Modes
> >    1. `od_600_value`: numeric
> >    2. `chr_position`: character
> >
> >       If a series of numbers are given as a value R will consider them to be in the "character" mode if they are enclosed as single or double quotes
> >
> >    3. `spock`: logical
> >    4. `pilot`
> >
> >       ```R
> >       > mode(pilot)
> >       Error in mode(pilot) : object 'pilot' not found
> >       ```
> >
> >       If `Earhart` did exist, then the mode of `pilot` would be whatever the mode of `Earhart` was originally.
> >
> {: .solution}
{: .question}

# Mathematical and functional operations on objects

Once an object exists (which by definition also means it has a mode), R can appropriately manipulate that object. For example, objects of the numeric modes can be added, multiplied, divided, etc. R provides several mathematical (arithmetic) operators including:

- `+`: addition
- `-`: subtraction
- `*`: multiplication
- `/`: division
- `^` or `**`: exponentiation
- `a%%b`: modulus (the remainder after division)

> <hands-on-title>Execute mathematical operations</hands-on-title>
>
> 1. Execute `(1 + (5 ** 0.5))/2`
>
>    ```R
>    > (1 + (5 ** 0.5))/2
>    [1] 1.618034
>    ```
>
> 2. Multiply the object `human_chr_number` by 2
>
>    ```R
>    > human_chr_number <- 23
>    # multiply the object 'human_chr_number' by 2
>    > human_chr_number * 2
>    [1] 46
>    ```
>
{: .hands_on}

> <question-title>Exercise: Compute the golden ratio</question-title>
>
> One approximation of the golden ratio ($$\varphi$$) is
>
> $$\frac{1 + \sqrt{5}}{2}$$
>
> Compute the golden ratio to 3 digits of precision using the `sqrt()` and `round()` functions.
>
> Hint: remember the `round()` function can take 2 arguments.
>
> > <solution-title></solution-title>
> >
> > ```R
> > > round((1 + sqrt(5))/2, digits = 3)
> > [1] 1.618
> > ```
> >
> > Notice that you can place one function inside of another.
> >
> {: .solution}
{: .question}

# Vectors

Vectors are probably the most used commonly used object type in R. **A vector is a collection of values that are all of the same type (numbers, characters, etc.)**.

One of the most common ways to create a vector is to use the `c()` function - the "concatenate" or "combine" function. Inside the function you may enter one or more values, separated by a comma.

> <hands-on-title>Create a vector</hands-on-title>
>
> 1. Create a `snp_genes` object containing "OXTR", "ACTN3", "AR", "OPRM1"
>
>    ```R
>    # Create the SNP gene name vector
>    > snp_genes <- c("OXTR", "ACTN3", "AR", "OPRM1")
>    ```
>
> 2. Check how this object is stored in the **Environment** panel
{: .hands_on}

Vectors always have a **mode** and a **length**. In the **Environment** panel, we could already have an insight on these properties: `chr [1:4]` may indicate character and 4 values.

> <hands-on-title>Check vector properties</hands-on-title>
>
> 1. Check the mode of `snp_genes` object
>
>    ```R
>    # Check the mode of 'snp_genes'
>    > mode(snp_genes)
>    [1] "character"
>    ```
>
> 2. Check the mode of `snp_genes` length
>
>    ```R
>    # Check the mode of 'snp_genes'
>    > length(snp_genes)
>    [1] "4"
>    ```
>
> 3. Check both properties using `str` function
>
>    ```R
>    # Check the structure of 'snp_genes'
>    > str(snp_genes)
>    [1] chr [1:4] "OXTR" "ACTN3" "AR" "OPRM1"
>    ```
{: .hands_on}

The `str()` (structure) function is giving the same information as the **Environmnent** panel.

Vectors are quite important in R. Another data type that we will not work in this tutorial but in extra tutorial: data frames, are collections of vectors. What we learn here about vectors will pay off even more when we start working with data frames.

## Creating and subsetting vectors

Once we have vectors, one thing we may want to do is specifically retrieve one or more values from our vector. To do so, we use **bracket notation**. We type the name of the vector followed by square brackets. In those square brackets we place the index (e.g. a number) in that bracket.

> <hands-on-title>Get values from vectors</hands-on-title>
>
> 1. Create several vectors
>    - `snps` object with "rs53576", "rs1815739", "rs6152", "rs1799971"
>    - `snp_chromosomes` object with "3", "11", "X", "6"
>    - `snp_positions` object with 8762685, 66560624, 67545785, 154039662
>
>    ```R
>    # Some interesting human SNPs
>    # while accuracy is important, typos in the data won't hurt you here
>    snps <- c('rs53576', 'rs1815739', 'rs6152', 'rs1799971')
>    snp_chromosomes <- c('3', '11', 'X', '6')
>    snp_positions <- c(8762685, 66560624, 67545785, 154039662)
>    ```
>
> 2. Get the 3rd value in the `snp_genes` vector
>
>    ```R
>    # get the 3rd value in the snp_genes vector
>    > snp_genes[3]
>    [1] "AR"
>    ```
{: .hands_on}

In R, every item your vector is indexed, starting from the first item (1) through to the final number of items in your vector.

You can also retrieve a range of numbers:

> <hands-on-title>Retrieve a range of values from vectors</hands-on-title>
> 1. Get the 1st through 3rd value in the `snp_genes` vector
>
>    ```R
>    # get the 1st through 3rd value in the snp_genes vector
>    > snp_genes[1:3]
>    [1] "OXTR"  "ACTN3" "AR"
>    ```
>
> 2. Get the 1st, 3rd, and 4th value in the `snp_genes` vector
>
>    ```R
>    # get the 1st, 3rd, and 4th value in the snp_genes vector
>    > snp_genes[c(1, 3, 4)]
>    [1] "OXTR"  "AR"    "OPRM1"
>    ```
{: .hands_on}

To retrieve several (but not necessarily sequential) items from a vector, you pass a **vector of indices**, a vector that has the numbered positions you wish to retrieve.

There are additional (and perhaps less commonly used) ways of subsetting a vector (see [these
examples](https://thomasleeper.com/Rcourse/Tutorials/vectorindexing.html)). Also, several of these subsetting expressions can be combined.

> <hands-on-title>Retrieve a complex range of values from vectors</hands-on-title>
> 1. Get the 1st through the 3rd value, and 4th value in the `snp_genes` vector
>
>    ```R
>    # get the 1st through the 3rd value, and 4th value in the snp_genes vector
>    # yes, this is a little silly in a vector of only 4 values.
>    > snp_genes[c(1:3,4)]
>    [1] "OXTR"  "ACTN3" "AR"    "OPRM1"
>    ```
{: .hands_on}

## Adding to, removing, or replacing values in existing vectors

Once you have an existing vector, you may want to add a new item to it. To do so, you can use the `c()` function again to add your new value.

> <hands-on-title>Add values to vectors</hands-on-title>
> 1. Add "CYP1A1", "APOA5" to `snp_genes` vector
>
>    ```R
>    # add the gene 'CYP1A1' and 'APOA5' to our list of snp genes
>    # this overwrites our existing vector
>    snp_genes <- c(snp_genes, "CYP1A1", "APOA5")
>    ```
>
> 2. Check the content of `snp_genes`
>
>    ```R
>    > snp_genes
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1" "APOA5"
>    ```
{: .hands_on}

To remove a value from a vection, we can use a negative index that will return a version a vector with that index's value removed.

> <hands-on-title>Remove values to vectors</hands-on-title>
> 1. Check value corresponding to `-6` in `snp_genes`
>
>    ```R
>    > snp_genes[-6]
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1"
>    ```
>
> 2. Remove the 6th value of `snp_genes`
>
>    ```R
>    > snp_genes <- snp_genes[-6]
>    > snp_genes
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1"
>    ```
{: .hands_on}

We can also explicitly rename or add a value to our index using double bracket notation.

> <hands-on-title>Rename values in vectors</hands-on-title>
> 1. Rename the 7th value to "APOA5"
>
>    ```R
>    > snp_genes[[7]]<- "APOA5"
>    > snp_genes
>    [1] "OXTR"   "ACTN3"  "AR"     "OPRM1"  "CYP1A1" NA       "APOA5"
>    ```
{: .hands_on}

Notice in the operation above that R inserts an `NA` value to extend our vector so that the gene "APOA5" is an index 7. This may be a good or not-so-good thing depending on how you use this.

> <question-title>Exercise: Examining and subsetting vectors</question-title>
>
> Which of the following are true of vectors in R?
> 1. All vectors have a mode **or** a length
> 2. All vectors have a mode **and** a length
> 3. Vectors may have different lengths
> 4. Items within a vector may be of different modes
> 5. You can use the `c()` to one or more items to an existing vector
> 6. You can use the `c()` to add a vector to an exiting vector
>
> > <solution-title></solution-title>
> >
> > 1. False: vectors have both of these properties
> > 2. True
> > 3. True
> > 4. False: vectors have only one mode (e.g. numeric, character) and all items in a vector must be of this mode.
> > 5. True
> > 6. True
> >
> {: .solution}
{: .question}

## Logical Subsetting

There is one last set of cool subsetting capabilities we want to introduce. It is possible within R to retrieve items in a vector based on a logical evaluation or numerical comparison.

For example, let's say we wanted get

> <hands-on-title>Subset logically vectors</hands-on-title>
> 1. Extract all of the SNPs in our vector of SNP positions (`snp_positions`) that were greater than 100,000,000
>
>    ```R
>    > snp_positions[snp_positions > 100000000]
>    [1] 154039662
>    ```
{: .hands_on}

In the square brackets you place the name of the vector followed by the comparison operator and (in this case) a numeric value. Some of the most common logical operators you will use in R are:

- `<`: less than
- `<=`: less than or equal to
- `>`: greater than
- `>=`: greater than or equal to
- `==`: exactly equal to
- `!=`: not equal to
- `!x`: not x
- `a | b`: a or b
- `a & b`: a and b

The reason why the expression `snp_positions[snp_positions > 100000000]` works can be better understood if you examine what the expression `snp_positions > 100000000` evaluates to:

```R
> snp_positions > 100000000
[1] FALSE FALSE FALSE  TRUE
```

The output above is a logical vector, the 4th element of which is `TRUE`. When you pass a logical vector as an index, R will return the true values:

```R
> snp_positions[c(FALSE, FALSE, FALSE, TRUE)]
[1] 154039662
```

If you have never coded before, this type of situation starts to expose the "magic" of programming. We mentioned before that in the bracket notation you take your named vector followed by brackets which contain an index: `named_vector[index]`. The "magic" is that the index needs to *evaluate to* a number. So, even if it does not appear to be an integer (e.g. 1, 2, 3), as long as R can evaluate it, we will get a result. That our expression `snp_positions[snp_positions > 100000000]` evaluates to a number can be seen in the following situation.

How to know which **index** (1, 2, 3, or 4) in our vector of SNP positions was the one that was greater than 100,000,000?

> <hands-on-title>Getting which indices of any item that evaluates as TRUE</hands-on-title>
> 1. Return the indices in our vector of SNP positions (`snp_positions`) that were greater than 100,000,000
>
>    ```R
>    > which(snp_positions > 100000000)
>    [1] 4
>    ```
{: .hands_on}

**Why this is important?** Often in programming we will not know what inputs and values will be used when our code is executed. Rather than put in a pre-determined value (e.g 100000000) we can use an object that can take on whatever value we need.

> <hands-on-title>Subset logically vectors</hands-on-title>
> 1. Create a `snp_marker_cutoff` containing `100000000`
> 1. Extract all of the SNPs in `snp_positions` that were greater than `snp_marker_cutoff`
>
> ```R
> > snp_marker_cutoff <- 100000000
> > snp_positions[snp_positions > snp_marker_cutoff]
> [1] 154039662
> ```
>
{: .hands_on}

Ultimately, it's putting together flexible, reusable code like this that gets at the "magic" of programming!

## A few final vector tricks

Finally, there are a few other common retrieve or replace operations you may want to know about. First, you can check to see if any of the values of your vector are missing (i.e. are `NA`). Missing data will get a more detailed treatment later, but the `is.NA()` function will return a logical vector, with TRUE for any NA value:

> <hands-on-title>Check for missing values</hands-on-title>
> 1. Check what are the missing values in `snp_genes`
>
>    ```R
>    # current value of 'snp_genes':
>    # chr [1:7] "OXTR" "ACTN3" "AR" "OPRM1" "CYP1A1" NA "APOA5"
>    > is.na(snp_genes)
>    [1] FALSE FALSE FALSE FALSE FALSE  TRUE FALSE
>    ```
>
{: .hands_on}

Sometimes, you may wish to find out if a specific value (or several values) is present a vector. You can do this using the comparison operator `%in%`, which will return TRUE for any value in your collection that is in the vector you are searching.

> <hands-on-title>Check for presence of values</hands-on-title>
> 1. Check if "ACTN3" and "APOA5" are in `snp_genes`
>
>    ```R
>    # current value of 'snp_genes':
>    # chr [1:7] "OXTR" "ACTN3" "AR" "OPRM1" "CYP1A1" NA "APOA5"
>    # test to see if "ACTN3" or "APO5A" is in the snp_genes vector
>    # if you are looking for more than one value, you must pass this as a vector
>    > c("ACTN3","APOA5") %in% snp_genes
>    [1] TRUE TRUE
>    ```
>
{: .hands_on}

> <question-title></question-title>
>
> 1. What data types/modes are the following vectors?
>    1. `snps`
>    2. `snp_chromosomes`
>    3. `snp_positions`
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > mode(snps)
>    > [1] "character"
>    > > mode(snp_chromosomes)
>    > [1] "character"
>    > > mode(snp_positions)
>    > [1] "double"
>    > ```
>    {: .solution}
>
> 2. Add the following values to the specified vectors:
>    1. Add "rs662799" to the `snps` vector
>    2. Add 11 to the `snp_chromosomes` vector
>    3. Add 116792991 to the `snp_positions` vector
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > snps <- c(snps, 'rs662799')
>    > > snps
>    > [1] "rs53576"   "rs1815739" "rs6152"    "rs1799971" "rs662799"
>    > > snp_chromosomes <- c(snp_chromosomes, "11") # did you use quotes?
>    > > snp_chromosomes
>    > [1] "3"  "11" "X"  "6"  "11"
>    > > snp_positions <- c(snp_positions, 116792991)
>    > > snp_positions
>    > [1]   8762685  66560624  67545785 154039662 116792991
>    > ```
>    {: .solution}
>
> 3. Make the following change to the `snp_genes` vector:
>    1. Create a new version of `snp_genes` that does not contain CYP1A1
>    2. Add 2 NA values to the end of `snp_genes`
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > snp_genes <- snp_genes[-5]
>    > > snp_genes <- c(snp_genes, NA, NA)
>    > > snp_genes
>    > [1] "OXTR"  "ACTN3" "AR"    "OPRM1" NA      "APOA5" NA      NA
>    > ```
>    >
>    {: .solution}
>
> 4. Using indexing, create a new vector named `combined` that contains:
>    - 1st value in `snp_genes`
>    - 1st value in `snps`
>    - 1st value in `snp_chromosomes`
>    - 1st value in `snp_positions`
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > combined <- c(snp_genes[1], snps[1], snp_chromosomes[1], snp_positions[1])
>    > > combined
>    > [1] "OXTR"    "rs53576" "3"       "8762685"
>    > ```
>    >
>    {: .solution}
>
> 5. What type of data is `combined`?
>
>    > <solution-title></solution-title>
>    >
>    > ```R
>    > > typeof(combined)
>    > [1] "character"
>    > ```
>    >
>    {: .solution}
{: .question}

## Coercing values

Sometimes, it is possible that R will misinterpret the type of data represented in a vector, or store that data in a mode which prevents you from operating on the data the way you wish. For example, a long list of gene names isn't usually thought of as a categorical variable, the way that your experimental condition (e.g. control, treatment) might be. More importantly, some R packages you use to analyze your data may expect characters as input, not factors. At other times (such as plotting or some statistical analyses) a factor may be more appropriate. Ultimately, you should know how to change the mode of an object.

First, its very important to recognize that coercion happens in R all the time. This can be a good thing when R gets it right, or a bad thing when the result is not what you expect. Consider:

```R
> chromosomes <- c('3', '11', 'X', '6')
> mode(chromosomes)
[1] "character"
```

Although there are several numbers in our vector, they are all in quotes, so we have explicitly told R to consider them as characters. However, even if we removed the quotes from the numbers, R would coerce everything into a character:

```R
> chromosomes_2 <- c(3, 11, 'X', 6)
> typeof(chromosomes_2)
[1] "character"
> chromosomes_2[1]
[1] "3"
```

We can use the `as.` functions to explicitly coerce values from one form into another.

> <hands-on-title>Coerce values</hands-on-title>
> 1. Create a `positions` object with "8762685", "66560624", "67545785", "154039662"
>
>    ```R
>    positions <- c("8762685", "66560624", "67545785", "154039662")
>    ```
>
> 2. Check the type and first element
>
>    ```R
>    > typeof(positions)
>    [1] "character"
>    > positions[1]
>    [1] "8762685"
>    ```
>
> 3. Coerce into a numeric type using `as.numeric()`
>
>    ```R
>    > snp_positions_2 <- as.numeric(positions)
>    > typeof(snp_positions_2)
>    [1] "double"
>    > snp_positions_2[1]
>    [1] 8762685
>    ```
{: .hands_on}

Here the coercion is straight forward, but what would happen if we tried using `as.numeric()` on `snp_chromosomes_2`

> <hands-on-title>Issue with coercing values</hands-on-title>
> 1. Create a `chromosomes_2` object with 3, 11, 'X', 6
>
>    ```R
>    chromosomes_2 <- c(3, 11, 'X', 6)
>    ```
>
> 2. Coerce into a numeric type using `as.numeric()`
>
>    ```R
>    > snp_chromosomes_2 <- as.numeric(chromosomes_2)
>    Warning message:
>    NAs introduced by coercion
>    ```
>
> 3. Check the content of `snp_chromosomes_2`
>
>    ```R
>    > snp_chromosomes_2
>    [1]  3 11 NA  6
>    ```
{: .hands_on}

We see that an `NA` value (R's default value for missing data) has been introduced.

Let's summarize this section on coercion with a few take home messages:

- Be careful to check the result when you explicitly coerce one data type into another (this is known as **explicit coercion**)

    Ideally, you should try to see if its possible to avoid steps in your analysis that force you to coerce.

- Check the structure (`str()`) of your data before working with them!

    R will sometimes coerce without you asking for it. This is called (appropriately) **implicit coercion**. For example when we tried to create a vector with multiple data types, R chose one type through implicit coercion.

# Lists

Lists are quite useful in R. You may come across lists in the way that some bioinformatics programs may store and/or return data to you. One of the key attributes of a list is that, unlike a vector, a list may contain data of more than one mode.

> <comment-title>Learning more about lists</comment-title>
> Learn more about creating and using lists using this [nice tutorial](https://r4ds.had.co.nz/vectors.html#lists). In this one example, we will create a named list and show you how to retrieve items from the list.
{: .comment}

> <hands-on-title>Create and manipulate list objects</hands-on-title>
> 1. Create a named list containing the genes, reference SNPs, chromomosome and position objects we create before
>
>    ```R
>    # Create a named list using the 'list' function and our SNP examples
>    snp_data <- list(genes = snp_genes,
>                     reference_snp = snps,
>                     chromosome = snp_chromosomes,
>                     position = snp_positions)
>    ```
>
>    > <comment-title>Easy reading</comment-title>
>    > For easy reading we have placed each item in the list on a separate line. Nothing special about this, you can do this for any multiline commands. Just make sure the entire command (all 4 lines) are highlighted before running.
>    >
>    > As we are doing all this inside the list() function use of the `=` sign is good style
>    {: .comment}
>
> 2. Examine the structure of the list
>
>    ```R
>    > str(snp_data)
>    List of 4
>     $ genes         : chr [1:8] "OXTR" "ACTN3" "AR" "OPRM1" ...
>     $ refference_snp: chr [1:5] "rs53576" "rs1815739" "rs6152" "rs1799971" ...
>     $ chromosome    : chr [1:5] "3" "11" "X" "6" ...
>     $ position      : num [1:5] 8.76e+06 6.66e+07 6.75e+07 1.54e+08 1.17e+08
>    ```
>
> 3. Get all values for the `position` object in the list
>
>    ```R
>    > snp_data$position
>    [1]   8762685  66560624  67545785 154039662 116792991
>    ```
>
> 4. Get the first value in the `position` object
>
>    ```R
>    > snp_data$position[1]
>    [1] 8762685
>    ```
{: .hands_on}


topic: data-science
tutorial: gnmx-lecture6
title: Data manipulation with Pandas
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 9e3aace33f1c4dbbef9b5a18eff14423
last_modified: 2026-05-27T09:01:47.956270
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture6/tutorial.html

content:
#  SARS-CoV-2 as a research problem

To learn about pandas we will use SARS-CoV-2 data. Before actually jumping to Pandas let's learn about the coronavirus molecular biology.

The following summary is based on these publications:

- [Masters:2006](http://dx.doi.org/10.1016/S0065-3527(06)66005-3)
- [Fehr and Perlman:2015](http://dx.doi.org/10.1007/978-1-4939-2438-7_1)
- [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218)
- [Kirchdoerfer:2016](http://dx.doi.org/10.1038/nature17200)
- [Walls:2020](http://dx.doi.org/10.1016/j.cell.2020.02.058)
- [Jackson:2022](https://www.nature.com/articles/s41580-021-00418-x)

## Genome organization

All coronaviruses contain non-segmented positive-strand RNA genome approx. 30 kb in length. It is invariably 5'-leader-UTR-replicase-S-E-M-N-3'UTR-poly(A). In addition,
it contains a variety of accessory proteins interspersed throughout the genome (see Fig. below; (From [Fehr and Perlman:2015](http://dx.doi.org/10.1007/978-1-4939-2438-7_1))).  

![Genome organozation](https://media.springernature.com/original/springer-static/image/chp%3A10.1007%2F978-1-4939-2438-7_1/MediaObjects/317916_1_En_1_Fig1_HTML.gif "Genomic organization of representative α, β, and γ CoVs. An illustration of the MHV genome is depicted at the top. The expanded regions below show the structural and accessory proteins in the 3′ regions of the HCoV-229E, MHV, SARS-CoV, MERS-CoV and IBV. Size of the genome and individual genes are approximated using the legend at the top of the diagram but are not drawn to scale. HCoV-229E human coronavirus 229E, MHV mouse hepatitis virus, SARS-CoV severe acute respiratory syndrome coronavirus, MERS-CoV Middle East respiratory syndrome coronavirus, IBV infectious bronchitis virus.")

The long replicase gene encodes a number of non-structural proteins (nsps) and occupies 2/3 of the genome. Because of -1 ribosomal frameshifting within ORFab approx. in 25% of the cases
polyprotein pp1ab is produced in place of pp1a. pp1a encodes 11 nsps while pp1ab encodes 15 (Fig. below from [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218)). 

![Discontinuous transcription](https://www.annualreviews.org/docserver/fulltext/virology/2/1/vi20265.f1.gif "Coronavirus genome structure and gene expression. (a) Coronavirus genome structure. The upper scheme represents the TGEV genome. Labels indicate gene names; L corresponds to the leader sequence. Also represented are the nsps derived from the processing of the pp1a and pp1ab polyproteins. PLP1, PLP2, and 3CL protease sites are depicted as inverted triangles with the corresponding color code of each protease. Dark gray rectangles represent transmembrane domains, and light gray rectangles indicate other functional domains. (b) Coronavirus genome strategy of sgmRNA expression. The upper scheme represents the TGEV genome. Short lines represent the nested set of sgmRNAs, each containing a common leader sequence (black) and a specific gene to be translated (dark gray). (c) Key elements in coronavirus transcription. A TRS precedes each gene (TRS-B) and includes the core sequence (CS-B) and variable 5′ and 3′ flanking sequences. The TRS of the leader (TRS-L), containing the core sequence (CS-L), is present at the 5′ end of the genome, in an exposed location (orange box in the TRS-L loop). Discontinuous transcription occurs during the synthesis of the negative-strand RNA (light blue), when the copy of the TRS-B hybridizes with the TRS-L. Dotted lines indicate the complementarity between positive-strand and negative-strand RNA sequences. Abbreviations: EndoU, endonuclease; ExoN, exonuclease; HEL, helicase; MTase, methyltransferase (green, N7-methyltransferase; dark purple, 2′-O-methyltransferase); nsp, nonstructural protein; PLP, papain-like protease; RdRp, RNA-dependent RNA polymerase; sgmRNA, subgenomic RNA; TGEV, transmissible gastroenteritis virus; TRS, transcription-regulating sequence; UTR, untranslated region.")

## Virion structure

Coronavirus is a spherical particle approx. 125 nm in diameter. It is covered with S-protein projections giving it an appearance of solar corona - hence the term coronavirus. Inside the envelope is a nucleocapsid that has helical symmetry far more typical of (-)-strand RNA viruses. There are four main structure proteins: spike (S), membrane (M), envelope (E), and nucleocapsid (N; Fig. below [Masters:2006](http://dx.doi.org/10.1016/S0065-3527(06)66005-3)): 

![Virion structure](https://ars.els-cdn.com/content/image/1-s2.0-S0065352706660053-gr1.jpg "Schematic of the coronavirus virion, with the minimal set of structural proteins")

Mature S protein is a trimer of two subunits: S1 and S2. The two subunits are produced from a single S-precursor by host proteases (see [Kirchdoerfer:2016](http://dx.doi.org/10.1038/nature17200); this however is not the case for all coronaviruses such as SARS-CoV). S1 forms the receptor-binding domain, while S2 forms the stalk. 

M protein is the most abundant structural component of the virion and determines its shape. It possibly exists as a dimer.

E protein is the least abundant protein of the capsid and possesses ion channel activity. It facilitates the assembly and release of the virus. In SARS-CoV it is not required for replication but is essential for pathogenesis. 

N proteins form the nucleocapsid. Its N- and C-terminal domains are capable of RNA binding. Specifically, it binds to transcription regulatory sequences and the genomic packaging signal. It also binds to `nsp3`
and M protein possible tethering viral genome and replicase-transcriptase complex. 

## Entering the cell

Virion attaches to the cells as a result of interaction between S-protein and a cellular receptor. In the case of SARS-CoV-2 (as is in SARS-CoV) angiotensin converting enzyme 2 (ACE2) serves as the receptor binding to C-terminal portion of S1 domain. After receptor binding S protein is cleaved at two sites within the S2 subdomain. The first cleavage separates the receptor-binding domain of S from the membrane fusion domain. The second cleavage (at S2') exposes the fusion peptide. The SARS-CoV-2 is different from SARS-CoV in that it contains a [furin](https://en.wikipedia.org/wiki/Furin) cleavage site that is processed during viral maturation within endoplasmatic reticulum (ER; see [Walls:2020](http://dx.doi.org/10.1016/j.cell.2020.02.058)). The fusion peptide inserts into the cellular membrane and triggers joining on the two heptad regions with S2 forming an antiparallel six-helix bundle resulting in fusion and release of the genome into the cytoplasm (Fig. below from [Jackson:2022](https://www.nature.com/articles/s41580-021-00418-x). 

![Entering the cell](https://i.imgur.com/DD7EO3G.png "Two spike (S) protein cleavage events are typically necessary for the coronavirus entry process: one at the junction of the S1 and S2 subunits and the other at the S2′ site, internal to the S2 subunit. In the case of SARS-CoV-2, the polybasic sequence at the S1–S2 boundary is cleaved during virus maturation in an infected cell, but the S2′ site is cleaved at the target cell following angiotensin-converting enzyme 2 (ACE2) binding. Virus binding to ACE2 (step 1) induces conformational changes in the S1 subunit and exposes the S2′ cleavage site in the S2 subunit. Depending on the entry route taken by SARS-CoV-2, the S2′ site is cleaved by different proteases. Left: If the target cell expresses insufficient transmembrane protease, serine 2 (TMPRSS2) or if a virus–ACE2 complex does not encounter TMPRSS2, the virus–ACE2 complex is internalized via clathrin-mediated endocytosis (step 2) into the endolysosomes, where S2′ cleavage is performed by cathepsins, which require an acidic environment for their activity (steps 3 and 4). Right: In the presence of TMPRSS2, S2′ cleavage occurs at the cell surface (step 2). In both entry pathways, cleavage of the S2′ site exposes the fusion peptide (FP) and dissociation of S1 from S2 induces dramatic conformational changes in the S2 subunit, especially in heptad repeat 1, propelling the fusion peptide forward into the target membrane, initiating membrane fusion (step 5 on the left and step 3 on the right). Fusion between viral and cellular membranes forms a fusion pore through which viral RNA is released into the host cell cytoplasm for uncoating and replication (step 6 on the left and step 4 on the right). Several agents disrupt interaction between the S protein and ACE2: ACE2 mimetics, therapeutic monoclonal antibodies targeting the neutralizing epitopes on the S protein and antibodies elicited by vaccination block virus binding to ACE2 and thus inhibit both entry pathways. By contrast, strategies targeting post-receptor-binding steps differ between the two pathways. Being a serine protease inhibitor, camostat mesylate restricts the TMPRSS2-mediated entry pathway. Hydroxychloroquine and chloroquine block endosomal acidification, which is necessary for cathepsin activity, and thus restrict the cathepsin-mediated entry pathway")

## Replication

The above figure shows that in addition to the full length (+)-strand genome there is a number of (+)-strand subgenomic RNAs corresponding to 3'-end of the complete viral sequence. All of these *subgenomic* RNAs (sgRNAs) share the same leader sequence that is present only once at the extreme 5'-end of the viral genome. These RNAs are produced via discontinuous RNA synthesis when the RNA-dependent RNA-polymerase (RdRp) switches template:

<iframe title="vimeo-player" src="https://player.vimeo.com/video/133088346" width="640" height="480" frameborder="0" allowfullscreen></iframe>

  &#9757; Model for the formation of genome high-order structures regulating N gene transcription. The upper linear scheme represents the coronavirus genome. The red line indicates the leader sequence in the 5′ end of the genome. The hairpin indicates the TRS-L. The gray line with arrowheads represents the nascent negative-sense RNA. The curved blue arrow indicates the template switch to the leader sequence during discontinuous transcription. The orange line represents the copy of the leader added to the nascent RNA after the template switch. The RNA-RNA interactions between the pE (nucleotides 26894 to 26903) and dE (nucleotides 26454 to 26463) and between the B-M in the active domain (nucleotides 26412 to 26421) and the cB-M in the 5′ end of the genome (nucleotides 477 to 486) are represented by solid lines. Dotted lines indicate the complementarity between positive-strand and negative-strand RNA sequences. Abbreviations: AD, active domain secondary structure prediction; B-M, B motif; cB-M, complementary copy of the B-M; cCS-N, complementary copy of the CS-N; CS-L, conserved core sequence of the leader; CS-N, conserved core sequence of the N gene; dE, distal element; pE, proximal element; TRS-L, transcription-regulating sequence of the leader (From [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218)).

Furthermore, [Sola:2015](https://www.annualreviews.org/doi/full/10.1146/annurev-virology-100114-055218) suggests the coronavirus transcription model in which transcription initiation complex forms at the 3'-end of (+)-strand genomic RNA:

![Coronavirus transcription](https://www.annualreviews.org/docserver/fulltext/virology/2/1/vi20265.f3.gif "Three-step model of coronavirus transcription. (1) Complex formation. Proteins binding transcription-regulating sequences are represented by ellipsoids, the leader sequence is indicated with a red bar, and core sequences are indicated with orange boxes. (2) Base-pairing scanning. Negative-strand RNA is shown in light blue; the transcription complex is represented by a hexagon. Vertical lines indicate complementarity between the genomic RNA and the nascent negative strand. (3) Template switch. Due to the complementarity between the newly synthesized negative-strand RNA and the transcription-regulating sequence of the leader, template switch to the leader is made by the transcription complex to complete the copy of the leader sequence")

## How virus is screened

Two approaches: (1) RNAseq and (2) amplicons

RNAseq approach requires a large sample quantity, but is suitable for detecting low-frequency variants. Ampliconic approaches are useful for samples containing a small fraction of viral RNA and are
by far the most widely used (From [NEB](https://primer-monitor.neb.com/lineages)):

![Primer schemes](https://i.imgur.com/9wLeHnZ.png "Nucleotide variants related to lineages of concern are indicated in yellow above a schematic SARS-CoV-2 genome. Primers that do not overlap with variants associated with these lineages are shown in blue. Overlapping primers are indicated in orange.")

# Pandas!

> This is an aggregated tutorial relying on material from the following fantastic sources:
>-  [Justin Bois](http://justinbois.github.io/bootcamp/2020/index.html). It contains modified training datasets and adopts content to Colab environment.
>- [BIOS821 course at Duke](https://people.duke.edu/~ccc14/bios-821-2017/index.html)
>- [Pandas documentation](https://pandas.pydata.org/docs/user_guide/index.html/)
{: .quote}

Pandas (from "Panel Data") is an essential piece of scientific (and not only) data analysis infrastructure. It is, in essence, a highly optimized library for manipulating very large tables (or "Data Frames"). 

Today we will be using a single notebook that explains the basics of this powerful tool:

## Pandas learning resources

- [Getting started](https://pandas.pydata.org/docs/getting_started/index.html#getting-started) - official introduction from Pandas.
- [Data Science Tools](http://people.duke.edu/~ccc14/bios-821-2017/index.html) - Data Science for Biologists from Duke University.
- [Data Carpentry](https://datacarpentry.org/) - a collection of lessons *à la* Software Carpentry.


```python
# Pandas, conventionally imported as pd
import pandas as pd
```

Throughout your research career, you will undoubtedly need to handle data, possibly lots of data. The data comes in lots of formats, and you will spend much of your time **wrangling** the data to get it into a usable form.

Pandas is the primary tool in the Python ecosystem for handling data. Its primary object, the `DataFrame` is extremely useful in wrangling data. We will explore some of that functionality here and will put it to use in the next lesson.

# Basics

## The data set

The dataset we will be using is a subset of metadata describing SARS-CoV-2 datasets from the [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra).  

It is obtained by going to https://www.ncbi.nlm.nih.gov/sra and performing a query with the following search terms: `txid2697049[Organism:noexp]`.

Results are downloaded using `Send to:` menu selecting `File` and then `RunInfo`. Let's get these results into this notebook:


```python
!wget https://zenodo.org/records/10680001/files/sra_ncov.csv.gz
```


```python
!gunzip -c  sra_ncov.csv.gz | head
```

The first line contains the **headers** for each column. The data follow. Given the file I/O skills you recently learned, you could write some functions to parse this file and extract the data you want.
You can imagine that this might be kind of painful. However, if the file format is nice and clean like we more or less have here, we can use pre-built tools. Pandas has a very powerful function, `pd.read_csv()`
that can read in a CSV file and store the contents in a convenient data structure called a **data frame**. In Pandas, the data type for a data frame is `DataFrame`, and we will use "data frame" and "`DataFrame`" interchangeably.

## Reading in data

Let's first look at the doc string of `pd.read_csv()`.


```python
pd.read_csv?
```

Holy cow! There are so many options we can specify for reading in a CSV file. You will likely find reasons to use many of these throughout your research. For now, however, we do not need most of them.
So, let's load in the data set. Note that even though the dataset is compressed with `gzip` we do not need to do anything additional - pandas magically understands and uncompresses the data while loading it into the dataframe.


```python
df = pd.read_csv('sra_ncov.csv.gz')
```

The same result can be achieved directly without downloading the file first:

```python
df = pd.read_csv('https://zenodo.org/records/10680001/files/sra_ncov.csv.gz')
```

We now have the data stored in a data frame. We can look at it in the Jupyter Notebook since Jupyter will display it in a well-organized, pretty way. Note that because our dataframe is big,
we only display the first five rows using `head()` function:


```python
df.head()
```

## Indexing data frames

The data frame is a convenient data structure for many reasons that will become clear as we start exploring. Let's start by looking at how data frames are indexed. Let's try to look at the first row.


```python
df[0]
```

Yikes! Lots of errors. The problem is that we tried to index numerically by row. **We index DataFrames, by columns.** And no column has the name `0` in this data frame, though there could be.
Instead, you might want to look at the column with the percentage of correct face-matching tasks.


```python
df['Run'].head()
```

This gave us the numbers we were after. Notice that when it was printed, the index of the rows came along with it. If we wanted to pull out a single percentage correct, say corresponding to index `4`, we can do that.


```python
df['Run'][4]
```

However, this is **not** the preferred way to do this. It is better to use `.loc`. This gives the location in the data frame we want.

> <details-title><tt>loc</tt> versus <tt>iloc</tt></details-title>
>
>`loc` and `iloc` are both methods in the Pandas library for DataFrame indexing.
>
>`loc`: It is label-based indexing, meaning that you use the actual row and column labels to make selections. This means that you specify rows and columns based on their index labels.
> For example, df.loc[3, 'column_name'] will select the value in the third row and the column labeled 'column_name'.
>
>`iloc`: It is integer-based indexing, where you use the integer position of the rows and columns to make selections. This means that you specify rows and columns based on their integer position.
> For example, df.iloc[3, 2] will select the value in the fourth row and the third column (remembering that indexing starts at 0).
>
>In summary, `loc` uses labels for indexing, while `iloc` uses integer positions.
{: .details}

```python
df.loc[4, 'Run']
```

```python
df.iloc[4:6]
```

```python
df.iloc[4:6, [0,2,4]]
```

```python
df.loc[4:6, ['Run','size_MB','LibraryStrategy']]
```

```python
df.loc[4:6, 'Run':]
```

```python
df.loc[4:6, 'Run':'LibrarySource']
```

It is also important to note that **row indices need not be integers**. And you should not count on them being integers. In practice, you will almost never use row indices, but rather use **Boolean indexing**.

## Filtering: Boolean indexing of data frames

Let's say I wanted to pull out accession numbers of runs produced by Pacific Biosciences machines (in this table such datasets are labeled as `PACBIO_SMRT`. I can use Boolean indexing to specify the row. Specifically, I want the row for which `df['Platform'] == 'PACBIO_SMRT'`. You can essentially plop this syntax directly when using `.loc`.


```python
df.loc[df['Platform'] == 'PACBIO_SMRT', 'Run']
```

If I want to pull the whole record for that participant, I can use `:` for the column index.

```python
df.loc[df['Platform'] == 'PACBIO_SMRT', :].head(10)
```

Now, let's pull out all PacBio records that were obtained from Amplicon sequencing. We can again use Boolean indexing, but we need to use an `&` operator.
We have not covered this bitwise operator before, but the syntax is self-explanatory in the example below. Note that it is important that each Boolean operation you are doing is in parentheses because of the precedence of the operators involved.

```python
df.loc[(df['Platform'] == 'PACBIO_SMRT') & (df['LibraryStrategy'] == 'AMPLICON'), :].head(3)
```

What is going on will be clearer if we set up our Boolean indexing first, as follows.

```python
inds = (df['Platform'] == 'PACBIO_SMRT') & (df['LibraryStrategy'] == 'AMPLICON')
inds[2:6]
```

Notice that `inds` is an array (actually a Pandas `Series`, essentially a `DataFrame` with one column) of `True`s and `False`s. We can apply the `unique` function from [NumPy](https://numpy.org/) to see how
many `True` and `False` rows we have:

```python
import numpy as np
np.unique(inds, return_counts=True)
```

When we index with it using .loc, we get back rows where inds is True:

```python
df.loc[inds, :]
```

```python
df.loc[(df['Platform'] == 'PACBIO_SMRT') & (df['LibraryStrategy'] == 'AMPLICON'), :].head(3)
```

## Calculating with data frames

The SRA data contains (sometimes) size of the run in MB (`size_MB`). Let's suppose we want to consider only those run where `size_MB` is above 100. We might like to add a column to the data frame that specifies whether or not the corresponding run is above 100Mb. We can conveniently compute with columns. This is done element-wise.

```python
df['size_MB'] >= 100
```

This tells us which run is above 100 Mb. We can simply add this back to the data frame.


```python
# Add the column to the DataFrame
df['Over100Mb'] = df['size_MB'] >= 100

# Take a look
df.head()
```


```python
df.assign(over100 = df['size_MB']>=100)
```

## A note of `assign`

The `assign` method in the pandas library is used to add new columns to a pandas DataFrame. The method allows you to perform calculations or operations on existing columns to generate new columns without modifying the original DataFrame.

For example, you can use the `assign` method to add a new column that calculates the sum of two existing columns or to add a column based on a complex calculation that involves multiple columns. The method also supports adding columns based on external data sources, such as NumPy arrays or other pandas DataFrames.

Using `assign` method is useful because it allows you to create new columns and add them to a DataFrame in a way that is both readable and easy to maintain. The method is also chain-able, meaning you can add multiple columns in one call, making your code more concise and efficient.

## A note about vectorization

Notice how applying the `<=` operator to a `Series` resulted in **elementwise** application. This is called `vectorization`. It means that we do not have to write a `for` loop to do operations on the elements of a `Series` or other array-like object. Imagine if we had to do that with a `for` loop.


```python
big_runs = []
for size in df['size_MB']:
    big_runs.append(size >= 100)
```

This is cumbersome. The vectorization allows for much more convenient calculation. Beyond that, the vectorized code is almost always faster when using Pandas and Numpy because the looping is done with compiled code under the hood. This can be done with many operators, including those you've already seen, like `+`, `-`, `*`, `/`, `**`, etc.

## Outputting a new CSV file

Now that we added the `Over100Mb` column, we might like to save our data frame as a new CSV that we can reload later. We use `df.to_csv()` for this with the `index` kwarg to ask Pandas not to explicitly write the indices to the file.


```python
df.to_csv('over100Mb_data.csv', index=False)
```

Let's take a look at what this file looks like.

```python
!head over100Mb_data.csv
```

# Tidy data

[Hadley Wickham](https://en.wikipedia.org/wiki/Hadley_Wickham) wrote a [great article](http://dx.doi.org/10.18637/jss.v059.i10) in favor of "tidy data." Tidy data frames follow the rules:

1. Each variable is a column.
2. Each observation is a row.
3. Each type of observation has its own separate data frame.

This is less pretty to visualize as a table, but we rarely look at data in tables. Indeed, the representation of data that is convenient for visualization is different from that which is convenient for analysis.
A tidy data frame is almost always **much** easier to work with than non-tidy formats.

Also, let's take a look at this [article](https://dtkaplan.github.io/DataComputingEbook/chap-tidy-data.html#chap:tidy-data).

## The data set

The dataset we will be using is a list of all SARS-CoV-2 datasets in [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) as of January 20, 2021.  

It is obtained by going to https://www.ncbi.nlm.nih.gov/sra and performing a query with the following search terms: `txid2697049[Organism:noexp]`.

Results are downloaded using `Send to:` menu selecting `File` and then `RunInfo`. Let's get these results into this notebook:


```python
df = pd.read_csv('https://zenodo.org/records/10680001/files/sra_ncov.csv.gz')
df = df[df['size_MB']> 0].reset_index(drop=True)

# Take a look
df
```

This data set is in tidy format. Each row represents a single SRA dataset. The properties of each run are given in each column. We already saw the power of having the data in this format when we did Boolean indexing in the last lesson.

## Finding unique values and counts

How many unique sequencing platforms do we have?

```python
df['Platform'].unique()
```


```python
df['Platform'].value_counts()
```

## Sorting

(and axes!)

Let's start by sorting on index:


```python
df_subset = df.sample(n=10)
```


```python
df_subset
```


```python
df_subset.sort_index()
```


```python
df_subset.sort_index(axis = 1)
```

Now let's try sorting by values:


```python
df_subset.sort_values(by=['LibraryLayout'])
```


```python
df_subset.sort_values(by=['LibraryLayout','size_MB'])
```


```python
df_subset.sort_values(by=['LibraryLayout','size_MB'],ascending=[True,False])
```

# Split-apply-combine

Let's say we want to compute the total size of SRA runs for each `BioProject`. Ignoring for the second the mechanics of how we would do this with Pandas, let's think about it in English. What do we need to do?

1. **Split** the data set up according to the `'BioProject'` field, i.e., split it up so we have a separate data set for each BioProject ID.
2. **Apply** a median function to the activity in these split data sets.
3. **Combine** the results of these averages on the split data set into a new, summary data set that contains classes for each platform and medians for each.

We see that the strategy we want is a **split-apply-combine** strategy. This idea was put forward by Hadley Wickham in [this paper](http://dx.doi.org/10.18637/jss.v040.i01). It turns out that this is a strategy we want to use *very* often. Split the data in terms of some criterion. Apply some function to the split-up data. Combine the results into a new data frame.

Note that if the data are tidy, this procedure makes a lot of sense. Choose the column you want to use to split by. All rows with like entries in the splitting column are then grouped into a new data set. You can then apply any function you want into these new data sets. You can then combine the results into a new data frame.

Pandas's split-apply-combine operations are achieved using the `groupby()` method. You can think of `groupby()` as the splitting part. You can then apply functions to the resulting `DataFrameGroupBy` object. The [Pandas documentation on split-apply-combine](http://pandas.pydata.org/pandas-docs/stable/groupby.html) is excellent and worth reading through. It is extensive though, so don't let yourself get intimidated by it.

## Aggregation

Let's go ahead and do our first split-apply-combine on this tidy data set. First, we will split the data set up by `BioProject`.


```python
grouped = df.groupby(['BioProject'])

# Take a look
grouped
```

There is not much to see in the `DataFrameGroupBy` object that resulted. But there is a lot we can do with this object. Typing `grouped.` and hitting tab will show you the many possibilities. For most of these possibilities, the apply and combine steps happen together and a new data frame is returned. The `grouped.sum()` method is exactly what we want.


```python
df_sum = grouped.sum()

# Take a look
df_sum
```


```python
df_sum = pd.DataFrame(grouped['size_MB'].sum())
df_sum
```

The outputted data frame has the sums of numerical columns only, which we have only one: `size_MS`. Note that this data frame has `Platform` as the name of the row index.
If we want to instead keep `Platform` (which, remember, is what we used to split up the data set before we computed the summary statistics) as a column, we can use the `reset_index()` method.


```python
df_sum.reset_index()
```

Note, though, that this was not done in-place. If you want to update your data frame, you have to explicitly do so.


```python
df_sum = df_sum.reset_index()
```

We can also use multiple columns in our `groupby()` operation. To do this, we simply pass in a list of columns into `df.groupby()`. We will **chain the methods**, performing a groupby,
applying a median, and then resetting the index of the result, all in one line.


```python
df.groupby(['BioProject', 'Platform']).sum().reset_index()
```

This type of operation is called an **aggregation**. That is, we split the data set up into groups, and then computed a summary statistic for each group, in this case the median.

You now have tremendous power in your hands. When your data are tidy, you can rapidly accelerate the ubiquitous split-apply-combine methods. Importantly, you now have a logical
framework to think about how you slice and dice your data. As a final, simple example, I will show you how to go start to finish after loading the data set into a data frame,
splitting by `BioProject` and `Platform`, and then getting the quartiles and extrema, in addition to the mean and standard deviation.


```python
df.groupby(['BioProject', 'Platform'])['size_MB'].describe()
```


```python
df.groupby(['BioProject', 'Platform'])['size_MB'].describe().reset_index()
```


```python
import numpy as np
df.groupby(['BioProject', 'Platform']).agg({'size_MB':np.mean, 'Run':'nunique'})
```

Yes, that's right. One single, clean, easy-to-read line of code. In the coming tutorials, we will see how to use tidy data to quickly generate plots.

> <tip-title>Quotes versus no quotes?</tip-title>
>
>[Why `np.mean` is without quotes and `nunique` is with quotes?](https://stackoverflow.com/questions/66443260/why-are-some-pandas-aggregation-functions-in-quotes-and-others-not)
{: .tip}

## Tidying a data set

You should always organize your data sets in a tidy format. However, this is sometimes just not possible, since your data sets can come from instruments that do not output the
data in tidy format (though most do, at least in my experience), and you often have collaborators that send data in untidy formats.

The most useful function for tidying data is `pd.melt()`. To demonstrate this we will use a dataset describing read coverage across SARS-CoV-2 genomes for several samples.


```python
df = pd.read_csv('https://zenodo.org/records/10680470/files/coverage.tsv.gz',sep='\t')

df.head()
```

Clearly, these data are not tidy. When we melt the data frame, the data within it, called **values**, become a single column. The headers, called **variables**, also become new columns.
So, to melt it, we need to specify what we want to call the values and what we want to call the variable. [`pd.melt()`](https://pandas.pydata.org/docs/reference/api/pandas.melt.html#pandas.melt) does the rest!

![Dataframe melt](https://pandas.pydata.org/docs/_images/07_melt.svg)

> Image from [Pandas Docs](https://pandas.pydata.org/docs/getting_started/intro_tutorials/07_reshape_table_layout.html#wide-to-long-format).


```python
melted = pd.melt(df, value_name='coverage', var_name=['sample'],value_vars=df.columns[3:],id_vars=['start','end'])

melted.head()
```

...and we are good to do with a tidy DataFrame! Let's take a look at the summary. This would allow us to easily plot coverage:


```python
import seaborn as sns
sns.relplot(data=melted, x='start',y='coverage',kind='line')
```


```python
sns.relplot(data=melted, x='start',y='coverage',kind='line',hue='sample')
```


```python
melted.groupby(['sample']).describe()
```


```python
melted.groupby(['sample'])['coverage'].describe()
```

To get back from melted (narrow) format to wide format we can use [`pivot()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html#pandas.DataFrame.pivot) function.

![Dataframe pivot](https://pandas.pydata.org/docs/_images/07_pivot.svg)

> Image from [Pandas Docs](https://pandas.pydata.org/docs/getting_started/intro_tutorials/07_reshape_table_layout.html#long-to-wide-table-format).


```python
melted.pivot(index=['start','end'],columns='sample',values='coverage')
```

# Working with multiple tables

Working with multiple tables often involves joining them on a common key:

![Left join](https://pandas.pydata.org/docs/_images/08_merge_left.svg)

In fact, this can be done in several different ways described below. But first, let's define two simple dataframes:


```python
!pip install --upgrade pandasql
```

```python
from pandasql import sqldf
pysqldf = lambda q: sqldf(q, globals())
```

```python
df1 = pd.DataFrame({"key": ["A", "B", "C", "D"], "value": np.random.randn(4)})
df2 = pd.DataFrame({"key": ["B", "D", "D", "E"], "value": np.random.randn(4)})
```

```python
df1
```

```python
df2
```

## Inner join

![Inner join](https://upload.wikimedia.org/wikipedia/commons/thumb/1/18/SQL_Join_-_07_A_Inner_Join_B.svg/234px-SQL_Join_-_07_A_Inner_Join_B.svg.png?20170204165143)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key")
```

Using `pysqldf`:


```python
pysqldf('select * from df1 join df2 on df1.key=df2.key')
```

## Left join

![Left join again](https://upload.wikimedia.org/wikipedia/commons/thumb/d/dc/SQL_Join_-_01b_A_Left_Join_B.svg/234px-SQL_Join_-_01b_A_Left_Join_B.svg.png?20170204144906)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key", how="left").fillna('.')
```

Using `pysqldf`:

```python
pysqldf('select * from df1 left join df2 on df1.key=df2.key').fillna('.')
```

```python
pysqldf('select df1.key, df1.value as value_x, df2.value as value_y from df1 left join df2 on df1.key=df2.key').fillna('.')
```

## Right join

![Right join](https://upload.wikimedia.org/wikipedia/commons/thumb/5/5f/SQL_Join_-_03_A_Right_Join_B.svg/234px-SQL_Join_-_03_A_Right_Join_B.svg.png?20170130230641)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key", how="right").fillna('.')
```

## Full join

![Full join](https://upload.wikimedia.org/wikipedia/commons/thumb/6/61/SQL_Join_-_05_A_Full_Join_B.svg/234px-SQL_Join_-_05_A_Full_Join_B.svg.png?20170130230643)

> Figure from Wikimedia Commons

Using pandas `merge`:


```python
pd.merge(df1, df2, on="key", how="outer").fillna('.')
```

# Putting it all together: Pandas + Altair

## Understanding [Altair](https://altair-viz.github.io/)

Vega-Altair is a declarative statistical visualization library for Python, based on Vega and Vega-Lite. It offers a powerful and concise grammar that enables you to quickly build a wide range of statistical visualizations. [This site](https://uwdata.github.io/visualization-curriculum/intro.html) contains a complete set of "how-to" tutorials explaining all aspects of this remarkable package.  

## Importing data

First, import all the packages we need:

```py
import pandas as pd
import altair as alt
from datetime import date
today = date.today()
```

Next, read a gigantic dataset from DropBox:

```py
sra = pd.read_csv(
    "https://zenodo.org/records/10680776/files/ena.tsv.gz",
    compression='gzip',
    sep="\t",
    low_memory=False
)
```

This dataset contains *a lot* of rows:

```py
len(sra)
```
Which would output:

```
1000000
```

Let's look at the five random lines from this table (scroll sideways):

```py
sra.sample(5)
```

|         | study_accession   | run_accession   | collection_date     | instrument_platform   | library_strategy   | library_construction_protocol   |
|--------:|:------------------|:----------------|:--------------------|:----------------------|:-------------------|:--------------------------------|
| 2470282 | PRJEB37886        | ERR8618165      | 2022-02-03 00:00:00 | ILLUMINA              | AMPLICON           | nan                             |
| 5747352 | PRJNA686984       | SRR21294506     | 2022-07-10 00:00:00 | OXFORD_NANOPORE       | AMPLICON           | nan                             |
| 2588443 | PRJEB37886        | ERR8997118      | 2022-02-15 00:00:00 | ILLUMINA              | AMPLICON           | nan                             |
| 3662323 | PRJNA716984       | SRR16033134     | 2021-09-08 00:00:00 | PACBIO_SMRT           | AMPLICON           | Freed primers                   |
| 5298351 | PRJNA716984       | SRR20455932     | 2022-04-10 00:00:00 | PACBIO_SMRT           | AMPLICON           | Freed primers                   |

This dataset also has *a lot* of columns:

```py
for _ in sra.columns: print(_)
```

We do not need all the columns, so let's restrict the dataset only to columns we would need. This would also make it much smaller:

```py
sra = sra[
    [ 
        'study_accession',
        'run_accession',
        'collection_date',
        'instrument_platform',
        'library_strategy',
        'library_construction_protocol' 
    ]
]
```

## Cleaning the data

The `collection_date` field will be useful for us to be able to filter out nonsense as you will see below. But to use it properly, we need tell Pandas that it is not just a text, but actually dates:

```py
sra = sra.assign(collection_date = pd.to_datetime(sra["collection_date"]))
```

Let's see what are the earliest and latest entries:

```py
print('Earliest entry:', sra['collection_date'].min())
print('Latest entry:', sra['collection_date'].max())
```

> <warning-title>Metadata is &#128169;</warning-title>
>&#128561; | Don't get surprised here - the metadata is only as good as the person who entered it. So, **when you enter metadata for you sequencing data -- pay attention!!!**
{: .warning}

> <question-title></question-title>
>
>Can you write a statement that would show how many rows contain these erroneous dates?
>
> > <solution-title></solution-title>
> >
> >```py
> > sra[sra['collection_date'] == sra['collection_date'].min()]['run_accession'].nunique()
> > sra[sra['collection_date'] == sra['collection_date'].max()]['run_accession'].nunique()
> > ```
> {: .solution}
{: .question}

The data will likely need a bit of cleaning:

```py
sra = sra[ 
    ( sra['collection_date'] >= '2020-01-01' ) 
    & 
    ( sra['collection_date'] <= '2023-02-16' ) 
]
```

Finally, in order to build the heatmap, we need to aggregate the data:

```py
heatmap_2d = sra.groupby(
    ['instrument_platform','library_strategy']
).agg(
    {'run_accession':'nunique'}
).reset_index()
```

This will look something like this:

|    | instrument_platform   | library_strategy    |   run_accession |
|---:|:----------------------|:--------------------|----------------:|
|  0 | BGISEQ                | AMPLICON            |              21 |
|  1 | BGISEQ                | OTHER               |            1067 |
|  2 | BGISEQ                | RNA-Seq             |              64 |
|  3 | BGISEQ                | Targeted-Capture    |              38 |
|  4 | BGISEQ                | WGA                 |               1 |
|  5 | CAPILLARY             | AMPLICON            |               3 |
|  6 | DNBSEQ                | AMPLICON            |             325 |
|  7 | DNBSEQ                | OTHER               |               5 |
|  8 | DNBSEQ                | RNA-Seq             |              64 |
|  9 | ILLUMINA              | AMPLICON            |         4833110 |
| 10 | ILLUMINA              | OTHER               |             204 |
| 11 | ILLUMINA              | RNA-Seq             |           34826 |
| 12 | ILLUMINA              | Targeted-Capture    |           14843 |
| 13 | ILLUMINA              | WCS                 |              74 |
| 14 | ILLUMINA              | WGA                 |           89499 |
| 15 | ILLUMINA              | WGS                 |           59618 |
| 16 | ILLUMINA              | WXS                 |               2 |
| 17 | ILLUMINA              | miRNA-Seq           |              28 |
| 18 | ION_TORRENT           | AMPLICON            |          108617 |
| 19 | ION_TORRENT           | RNA-Seq             |              75 |
| 20 | ION_TORRENT           | WGA                 |             724 |
| 21 | ION_TORRENT           | WGS                 |            1385 |
| 22 | OXFORD_NANOPORE       | AMPLICON            |          445491 |
| 23 | OXFORD_NANOPORE       | OTHER               |              31 |
| 24 | OXFORD_NANOPORE       | RNA-Seq             |           18617 |
| 25 | OXFORD_NANOPORE       | WGA                 |            7842 |
| 26 | OXFORD_NANOPORE       | WGS                 |           13670 |
| 27 | PACBIO_SMRT           | AMPLICON            |          533112 |
| 28 | PACBIO_SMRT           | FL-cDNA             |              11 |
| 29 | PACBIO_SMRT           | RNA-Seq             |            1013 |
| 30 | PACBIO_SMRT           | Synthetic-Long-Read |              29 |
| 31 | PACBIO_SMRT           | Targeted-Capture    |             465 |
| 32 | PACBIO_SMRT           | WGS                 |              48 |


## Plotting the data

Now let's create a graph. This graph will be layered: the "back" will be the heatmap squares and the "front" will be the numbers (see heatmap at the beginning of this page):

```py
back = alt.Chart(heatmap_2d).mark_rect(opacity=1).encode(
    x=alt.X(
        "instrument_platform:N",
        title="Instrument"
    ),
    y=alt.Y(
        "library_strategy:N",
        title="Strategy",
        axis=alt.Axis(orient='right')
    ),
    color=alt.Color(
        "run_accession:Q",
        title="# Samples",
        scale=alt.Scale(
            scheme="goldred",
            type="log"
        ),
    ),
    tooltip=[
        alt.Tooltip(
            "instrument_platform:N",
            title="Machine"
        ),
        alt.Tooltip(
            "run_accession:Q",
            title="Number of runs"
        ),
        alt.Tooltip(
            "library_strategy:N",
            title="Protocol"
        )
    ]
).properties(
    width=500,
    height=150,
    title={
      "text": 
        ["Breakdown of datasets (unique accessions) from ENA",
         "by Platform and Library Strategy"],
      "subtitle":"(Updated {})".format(today.strftime("%B %d, %Y"))
    }
)

back
```

This would give us a grid:

![Grid](https://i.imgur.com/74itE5M.png)

Now, it would be nice to fill the rectangles with actual numbers:

```py
front = back.mark_text(
    align="center",
    baseline="middle",
    fontSize=12,
    fontWeight="bold",
).encode(
    text=alt.Text("run_accession:Q",format=",.0f"),
    color=alt.condition(
        alt.datum.run_accession > 200,
        alt.value("white"),
        alt.value("black")
    )
)

front
```

This would give us the text:

![Text for grid](https://i.imgur.com/pKalDaC.png)

To superimpose these on top of each other we should simply do this:

```py
back + front
```

![Full graph](https://i.imgur.com/SzhM1QW.png)


topic: data-science
tutorial: gnmx-lecture1
title: A (very) brief history of genomics
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: a8c5379a685be8aaf6d0a68fcefa0c94
last_modified: 2026-05-15T22:28:11.056415
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/gnmx-lecture1/tutorial.html

content:
![Fluctuation test](./images/luria_small.png)

# What do you know?

This is the first lecture of the class. We will begin with a short quiz that is designed tio let me know what you do and do not know. This quiz will have no effect on your grade in this course. 

# Why history?

Knowing history is essential for understanding how we arrived to the current state of affairs in our field. It is also full of accidental discoveries and dramatic relationships making it quite interesting to read about. I strongly advise you to take a look at the manuscripts below:

## Classical publications

* 1965 : [A history of genetics](http://www.amazon.com/A-History-Genetics-A-H-Sturtevant/dp/0879696079)
* 1943 : [Delbruck & Luria](http://www.bx.psu.edu/~anton/bioinf1-2014/delbruck-luria-1943.pdf)
* 1944 : [Avery, MacLeod, & McCarty](http://www.bx.psu.edu/~anton/bioinf1-2014/avery-1944.pdf)
* 1952 : [Herhey & Chase](http://www.bx.psu.edu/~anton/bioinf1-2014/hershey-chase-1952.pdf)
* 1953 : [Watson & Crick](http://www.bx.psu.edu/~anton/bioinf1-2014/watsoncrick.pdf)
* 1958 : [Meselson & Stahl](http://www.bx.psu.edu/~anton/bioinf1-2014/Proc%20Natl%20Acad%20Sci%20USA%201958%20Meselson.pdf)
* 1960 : [Jacob and Monod](http://www.bx.psu.edu/~anton/bioinf1-2014/jacob-monod-1961.pdf)

## Popular (yet very informative) literature

![Books](./images/books.png)

------

# Let's get to it

Today's lecture will be organized as a, well, a lecture - we will be going over a set of slides linked at the beginning of this page. 

-----


topic: data-science
tutorial: online-resources-protein
title: One protein along the UniProt page
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["online-resources-gene"]
tags_json: ["deutsch", "espa\u00f1ol", "italiano"]
content_hash: c2c62509dfe2eeeef13e1824c7cfe950
last_modified: 2026-05-27T09:01:47.964663
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/online-resources-protein/tutorial.html

content:
When doing a biological data analysis, we might end up with some interesting proteins, that we need explore these genes. But how can we do that? What are the resources available for that? And how to navigate through them?

The aim of this tutorial is to familiarize ourselves with that, using Human opsins as an example.

> <comment-title></comment-title>
> This tutorial is a bit atypical: we will not work in Galaxy but mostly outside of it, in the [UniProt](https://uniprot.org) database pages.
{: .comment}

> <comment-title></comment-title>
> This tutorial  designed to be the continuation of the tutorial ["One gene across file formats"]({% link topics/data-science/tutorials/online-resources-gene/tutorial.md %}), but it can also be consulted as a stand-alone module. 
{: .comment}

Opsins are found in the cells of your retina. They catch light and begin the sequence of signals that result in vision, and that is the reason why, when compromised, they are associated to color-blindness and other visual impairments. 

> <comment-title>Sources of information from this tutorial</comment-title>
>
> The tutorial you are consulting was mainly developed consulting UniProtKB resources, in particular the [Explore UniProtKB entry](https://www.uniprot.org/help/explore_uniprotkb_entry) tutorial. 
> Some sentences are reported from there with no modifications.
> 
> In addition, the topic was chosen based Gale Rhodes' [Bioinformatics Tutorial](https://spdbv.unil.ch/TheMolecularLevel/Matics/index.html). Although the tutorial cannot be anymore be followed step by step due to how the mentioned resources changed in time, it could provide additional insights on opsins and in particular how one could build structural models of proteins based on evolutionary information. 
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
{: .agenda}

# The UniProtKB Entry page

The portal to visit to obtain all information about a protein is [UniProtKB](https://www.uniprot.org/). We can search it using a text search, or the gene or protein name. Let's try first a set of generic keywords, like `Human opsin`. 

> <hands-on-title>Search for Human opsin on UniProtKB</hands-on-title>
>
> 1. Open the [UniProtKB](https://www.uniprot.org/)
> 2. Type `Human opsin` in the search bar
> 3. Launch the search
>
{: .hands-on}

> <question-title></question-title>
>
> How many results did we get?
>
> > <solution-title></solution-title>
> >
> > 410 results (at the time of preparation of this tutorial)
> >
> {: .solution}
{: .question}

These 410 results give us the feeling that we need to be more specific (although - spoiler - our actual target is among the first hits).

To be specific enough, we suggest to use an unique identifier. From the [previous tutorial]({% link topics/data-science/tutorials/online-resources-gene/tutorial.md %}) we know the gene name of the protein that we are looking for, `OPN1LW`.

> <hands-on-title>Search for OPN1LW on UniProtKB</hands-on-title>
>
> 1. Type `OPN1LW` in the top search bar
> 2. Launch the search
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many results did we get?
> 2. What should we do to reduce this number?
>
> > <solution-title></solution-title>
> >
> > 1. 200+ results (at the time of preparation of this tutorial)
> > 2. We need to clarify what we are looking for: Human OPN1LW
> {: .solution}
{: .question}

We need to add `Human` to clarify what we are looking for.

> <hands-on-title>Search for Human OPN1LW on UniProtKB</hands-on-title>
>
> 1. Type `Human OPN1LW` in the top search bar
> 2. Launch the search
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many results did we get?
> 2. Do we have one result including OPN1LW as gene name?
>
> > <solution-title></solution-title>
> >
> > 1. 7 results (at the time of preparation of this tutorial)
> > 2. The first result is labeled with `Gene: OPN1LW (RCP)`
> {: .solution}
{: .question}

The first result, labeled with `Gene: OPN1LW (RCP)`, is our target, `P04000 · OPSR_HUMAN`. Before opening the page, two things to notice: 

1. The name of the protein `OPSR_HUMAN` is different than the gene name, as well as their IDs are. 
2. This entry has a golden star, which means that was manually annotated and curated. 

# Inspecting a UniProt entry

> <hands-on-title>Open a result on UniProt</hands-on-title>
>
> 1. Click on `P04000 · OPSR_HUMAN`
{: .hands-on}

![Screenshot of the UniProt entry page header](./images/UniProt.png "UniProt page")

To navigate this long page, the menu (navigation bar) on the left will be extremely useful. Just from it, we understand that this database contains information about the entry on: 
- the known functions, 
- the taxonomy, 
- the location,
- variants and associated diseases, 
- Post Translational Modification (PTMs), 
- the expression, 
- the interactions, 
- the structure, 
- the domains and their classification, 
- the sequences
- similar proteins. 

The navigation bar stays in the same place on the screen as you move up and down in an entry, so that you can quickly navigate to sections of interest. We will consult all the mentioned sections separately, but let's first focus on the headers on the left. 

At the top of the page, you can see the UniProt entry accession and name, the name of the protein and gene, the organism, whether the protein entry has been manually reviewed by a UniProt curator, its annotation score, and the evidence level for its existence.

Below the main header, you find a series of tabs (*Entry*, *Variant viewer*, *Feature viewer*, *Publications*, *External links*, *History*). The tabs allow you to switch between the entry, a graphical view of sequence features (Feature viewer), publications, and external links, but ignore them for the moment and do not move from the *Entry* tab. 

## Entry

The next menu is already part of the *Entry* tab. It allows us to run a BLAST sequence similarity search on the entry, align it with all its isoforms, download the entry in various formats, or add it to the basket to save for later.

> <question-title></question-title>
>
> 1. What are the available formats in the *Download* drop-down menu? 
> 2. What type of information would we download through these file formats?
>
> > <solution-title></solution-title>
> > 
> > 1. The formats are: `Text`, `FASTA (canonical)`, `FASTA (canonical & isoform`, `JSON`, `XML`, `RDF/XML`, `GFF`
> > 2. The`FASTA` formats should sound familiar (after the preliminary tutorial), and include the protein sequence, eventually with its isoforms (in which case it will be a multi-FASTA). Besides these, all the other formats are not protein- or even biology-specific. These are general file formats extensively used by websites to include the information included in the page. Hence, by downloading `text` (or even better the `json`) file, we would download the same annotation that we access in this page, but in a format that is easier to parse programmatically.  
> >
> {: .solution}
{: .question}

Let's now scroll the entry page, section by section. 

### Function

This section summarises the functions of this protein as follows: 

Visual pigments are the light-absorbing molecules that mediate vision. They consist of an apoprotein, opsin, covalently linked to cis-retinal.

Regardless of the level of details that you understand (depending on your background), this is impressively short and specific considering the enourmous amount of literature and studies that exist beyond the determination of a protein function. Anyway, someone did the work for us, and this protein is already fully classified in the Gene Ontology (GO), which describes any classified protein's molecular function, biological process and cellular component. 

GO is a perfect example of database / resource that builds on a very complex universe of knowledge and translates it into a simpler graph, at the risk of losing  details. This has the great advantage of organising the information, making it countable and analysable and programmatically accessible, ultimately allowing us to have this long summary pages and KnowledgeBases. 

> <question-title></question-title>
>
> 1. To which molecular functions is this protein annotated?
> 2. To which cellular components is this protein annotated?
> 3. To which biological processes this protein annotated?
>
> > <solution-title></solution-title>
> > 
> > 1. Photoreceptor protein, G-protein coupled receptor
> > 2. Photoreceptor disc membrane
> > 2. Sensory transduction, Vision 
> >
> {: .solution}
{: .question}

### Names and Taxonomy

Other examples of structured information are available in the next section, e.g. in the taxonomy. This section also reports other unique identifiers that refer to the same biological entity or to linked entities (e.g. associated diseases in the `MIM` menu).

> <question-title></question-title>
>
> 1. What is the taxonomic identifier associated with this protein?
> 2. What is the proteome identifier associated with this protein?
>
> > <solution-title></solution-title>
> > 
> > 1. 9606, i.e. Homo sapiens
> > 2. UP000005640, componenent of Chromosome Xs
> {: .solution}
{: .question}

## Subcellular location

We already know where our protein is in the human body (in the retina, as specified by the function summary), but where is it in the cell? 

> <question-title></question-title>
>
> 1. Where is our protein in the cell?
> 2. Is it consistent with the GO annotation observed before?
>
> > <solution-title></solution-title>
> > 
> > 1. The section explains that it is a "Multi-pass membrane protein", which means that it is a protein inserted in the cell membrane and goin through it multiple times.
> > 2. The GO Annotation on the top mentions that we are referring to the photoreceptor (cell) membrane in particular. 
> {: .solution}
{: .question}

The Subcellular location section includes a *Features* area detailing which sections, along the protein sequence, are inserted in the membrane (Transmembrane) and which not (Topological domain). 

> <question-title></question-title>
>
> How many Transmembrane domains and Topological domains are there?
>
> > <solution-title></solution-title>
> > 
> > 8 Transmembrane and 7 Topological domains
> {: .solution}
{: .question}

### Disease & Variants

As we know from the previous tutorial, this gene/protein is associated to multiple diseases. This section details this association also listing the specific variants that have been detected as disease-related. 

> <question-title></question-title>
>
> What types of scientific studies allow to assess the association of a genetic variant to diseases?
>
> > <solution-title></solution-title>
> > 
> > Three commonly used methods for assessing the association of a genetic variant with a disease are:
> >
> > - Genome-Wide Association Studies (GWAS)
> >
> >   GWAS are widely used to identify common genetic variants associated with diseases. They involve scanning the entire genome of a large number of individuals to identify variations linked to a particular disease or trait.
> >
> > - Case-Control Studies
> >
> >   Case-control studies are frequently employed to compare individuals with a disease to those without it, focusing on the presence or frequency of specific genetic variants in both groups.
> >
> > - Family Studies
> >
> >   Family-based studies involve analyzing genetic variants within families where multiple members are affected by a disease. By studying the inheritance patterns of genetic variants and their association with the disease within families, researchers can identify potential disease-associated genes.
> >
> >
> > These type of studies would imply extensive usage of the file types to manage genomic data, such as: SAM (Sequence Alignment Map), BAM (Binary Alignment Map), VCF (Variant Calling Format) etc.  
> >
> {: .solution}
{: .question}

This section also includes a *Features* area, where the natural variants are mapped along the sequence. Below, it also highlights that a more detailed view of features along the sequence is provided in the *Disease & Variants* tab, but let's not open it for now. 

### PTM/Processing

A post-translational modification (PTM) is a covalent processing event resulting from a proteolytic cleavage or from the addition of a modifying group to one amino acid. 

> <question-title></question-title>
>
> What are the post-translational modifications for our protein?
>
> > <solution-title></solution-title>
> > 
> > Chain, glycosylation, disulfide bond, modified residue
> {: .solution}
{: .question}

### Expression

We already know where the protein is in the cell, but for human proteins we often have information on where it is in the human body, i.e. in which tissues. This information can come from the Human [ExpressionAtlas](https://www.ebi.ac.uk/gxa/home) or other similar resources. 

> <question-title></question-title>
>
> In which tissue is the protein found?
>
> > <solution-title></solution-title>
> > 
> > The three color pigments are found in the cone photoreceptor cells.
> {: .solution}
{: .question}

### Interaction

Proteins perform their function through their interaction with the surroundings, in particular with other proteins. This section reports the interactors of our protein of interest, in a table that we can also filter by subcellular location, diseases, and type of interaction. 

The source of this information are databases like STRING, and the entry page for our protein is directly linked from this section. 

> <hands-on-title>Search for Human OPN1LW on UniProtKB</hands-on-title>
>
> 1. Click on the [STRING link](https://string-db.org/network/9606.ENSP00000358967) in a different tab 
>
{: .hands-on}

> <question-title></question-title>
>
> 1. How many different file formats can you download from there?
> 2. What kind of information will be conveyed in each file?
>
> > <solution-title></solution-title>
> > 
> >  STRING provides data in downloadable file formats to support further analyses. 
> >  The primary file format used by STRING is the 'TSV' (Tab-Separated Values) format, which presents protein interaction data in a structured, tabular layout. This format is well-suited for easy integration into various data analysis tools and software. 
> >  Additionally, STRING offers data in PSI-MI (Proteomics Standards Initiative Molecular Interactions) XML format, a standard for representing protein interaction data that allows for compatibility with other interaction databases and analysis platforms. These file formats enable researchers to harness the wealth of protein interaction information in STRING for their own studies and analyses. Researchers can also download visual representations of protein networks in image formats such as PNG and SVG, which are suitable for presentations and publications. For advanced analysis, STRING offers 'flat files' containing detailed interaction information, and 'MFA' (Multiple Alignment Format) files, which are useful for comparing multiple protein sequences. These diverse downloadable file formats enable researchers to harness the wealth of protein interaction information in STRING for their own studies and analyses.
> >
> {: .solution}
{: .question}

### Structure

Are you curious about the intricate three-dimensional structures of proteins? The *Structure* section on the UniProtKB entry page is your gateway to exploring the fascinating world of protein architecture. 

In this section, you'll find information about experimentally determined protein structures. These structures provide crucial insights into how proteins function and interact with other molecules. You will discover interactive views of the protein's structure that you can explore directly within the UniProtKB entry. This feature provides an engaging way to navigate through the protein's domains, binding sites, and other functional regions. By delving into the *Structure* section, you'll gain a deeper understanding of the physical basis of protein function and discover the wealth of information that structural data can unlock.

> <question-title></question-title>
>
> 1. What is the variant associated to color blindness?
> 2. Can you find that specific amino acid in the structure? 
> 3. Can you formulate a guess of why this mutation is disruptive?
>
> > <solution-title></solution-title>
> >
> > 1. In the *Disease & Variants* section, we discover that the change from Glycine (G) to Glutamic acid (E) at position 338 along the protein sequence is associated to Colorblindness. 
> > 2. In the structure viewer, we can move the molecule around and hover the mouse on the structure to find the AA in position 338. It might take some time to follow through the multiple helical arrangements of these structures. The Glycine at 338 is not in a helix, but in what looks like a loop just before a low confidence area in the structure. 
> > 3. Based on the information we collected so far, we could make a hypothesis of why this is distruptive. It is not in a helix (usually, in transmembrane proteins, helices are inserted in the membrane), hence, it is in one of the bigger domains that protrude from the membrane, in or out of the cell. This mutation probably doesn't distrupt the structure in its intra-membrane segments, but rather one of the functional domain. If you want to dig deeper, you can check if this is the extra- or intra-cellular segment in the **Feature viewer**.
> >
> {: .solution}
{: .question}

Where does the information in the structure viewer come from?

> <hands-on-title>Search for Human OPN1LW on UniProtKB</hands-on-title>
>
> 1. Click on the download icon below the structure
> 2. Check the file that has been downloaded
>
{: .hands-on}

This is a PDB (Protein Data Bank) file, allowing you to visualize and analyze the protein's arrangement of atoms and amino acids. 

However, there is no reference to the PDB database in the links among the *3D structure databases*. Instead, the first link refers to the AlphaFoldDB. The AlphaFold Database is a comprehensive resource that provides predicted 3D structures for a wide range of proteins. Using deep learning techniques and evolutionary information, AlphaFold accurately predicts the spatial arrangement of atoms within a protein, contributing to our understanding of protein function and interactions. 

Hence, this is a *prediction* of the structure, not a structure experimentally validated. This is the reason why it is colored by confidence: the sections in blue are those with a high confidence value, so the ones for which the prediction is very reliable, while the ones in orange are less reilable or have a disordered (more flexible and mobile) structure. Nevertheless, this information is represented through a PDB file, because it is still structural. 

### Family and Domains

The *Family and Domains* section on the UniProtKB entry page provides a comprehensive view of the evolutionary relationships and functional domains within a protein. This section offers insights into the protein's membership in protein families, superfamilies, and domains, shedding light on its structural and functional characteristics. 

The *Features* area confirms indeed that at least one of the two domains protruding from the membrane (the N-terminal one) is disordered. This area usually includes information about conserved regions, motifs, and important sequence features that contribute to the protein's role in various biological processes. The section confirms once again that we are looking at a transmembrane protein, and links to several resources of phylogenetic, protein families or domains data - guiding us on understading how proteins share common ancestry, evolve, and acquire specialized functions.

### Sequence

All this information about the protein's evolution, function, structure, is ultimately encoded in its sequence. Once again, in this section we have the opportunity to download the FASTA file transcribing it, as well as to access the source of this data: the genomic sequencing experiments that assessed it. This section also reports when isoforms have been detected. 

> <question-title></question-title>
>
> How many potential isoforms are mapped to this entry?
>
> > <solution-title></solution-title>
> >
> > 1: H0Y622
> >
> {: .solution}
{: .question}

### Similar proteins

The last section of the UniProt Entry page reports similar proteins (this is basically the result of a clustering, with 100%, 90% and 50% identity thresholds).

> <question-title></question-title>
>
> 1. How many similar proteins at 100% identity?
> 2. How many similar proteins at 90% identity?
> 3. How many similar proteins at 50% identity?
>
> > <solution-title></solution-title>
> >
> > 1. 0
> > 2. 83
> > 3. 397
> >
> {: .solution}
{: .question}

As you might have guessed while looking at this page, a lot of the processing of biological data about a protein actually consists of mapping different types of information along the sequence, and understand how they influence each other. A visual mapping (and a table with the same information) is provided by the two alternative tabs to view this entry, that is the *Variant viewer* and the *Feature viewer*. 

## Variant viewer

> <hands-on-title>Variant viewer</hands-on-title>
>
> 1. Click on *Variant viewer* tab
>
{: .hands-on}

The *Variant viewer* maps all the known alternative versions of this sequence. For some of them the effect (pathogenic or benign) is known, for others not. 

> <question-title></question-title>
>
> How many variant are likely pathogenic?
>
> > <solution-title></solution-title>
> >
> > By zooming out in the variant view, we see that we have 5 red points, so 5 variants that are likely pathogenic.
> >
> {: .solution}
{: .question}

The high number of variants that you find in this section suggests that "protein sequences" (as well as gene sequences, protein structures etc.) are actually less fixed entities than what we would think.

## Feature viewer

> <hands-on-title>Feature viewer</hands-on-title>
>
> 1. Click on *Feature viewer* tab
>
{: .hands-on}

The *Feature viewer* is basically a merged version of all the *Features* areas that we found in the *Entry* page, including *Domains & sites*, *Molecule processing*, *PTMs*, *Topology*, *Proteomics*, *Variants*. If in the viewer you click on any feature, the correspondent region in the structure will be focused, as the variant of interest

> <hands-on-title>Variant viewer</hands-on-title>
>
> 1. Expand the *Variants* part
> 2. Zoom out
> 3. Click on our variant of interest (the red point at position 338)
>
{: .hands-on}

> <question-title></question-title>
>
> What is the topology at this location?
>
> > <solution-title></solution-title>
> >
> > A topological cytoplasmic domain
> >
> {: .solution}
{: .question}

Finally, let's have a quick look at the other tabs. 

## Publications

> <hands-on-title>Publication</hands-on-title>
>
> 1. Click on *Publication* tab
>
{: .hands-on}

The *Publications* one lists scientific publications related to the protein. These are collected by merging a fully curated list in UniProtKB/Swiss-Prot and automatically imported ones. In this tab, you can filter the publications list by source and categories that are based on the type of data a publication contains about the protein (such as function, interaction, sequence, etc.), or by the number of proteins in the corresponding study it describes ("small scale" vs "large scale").

> <question-title></question-title>
>
> 1. How many publications are associated with this protein?
> 2. How many publications contains information about its function?
>
> > <solution-title></solution-title>
> >
> > 1. 57
> > 2. 23
> >
> {: .solution}
{: .question}

## External links

> <hands-on-title>External links</hands-on-title>
>
> 1. Click on *External links* tab
>
{: .hands-on}

The *External links* tab puts together all the references to external databases and resources of information that we found in each Entry page section. The links text often reports the unique identifiers that represent the same biological entity in other databases. To get a feeling of this compexity, check the following image (that is already partially outdated).

![A graph representing how all the different databases are connected by unique IDs, where major nodes are DBs, and arrows connect them to the IDs (minor nodes) that they report. The map is very crowded, especially around the UniProt Entry Name, Gene ID and Ensembl Gene ID](./images/complexDB.jpeg "Best illustration of the complex ID crossref: bioDBnet Network Diagram - [source](https://biodbnet-abcc.ncifcrf.gov/dbInfo/netGraph.php)")

## History

Finally, the *History* tab is also an interesting one. It reports and makes available for download all the previous versions of this entry annotations, that is: all the "evolution" of its annotation, in this case dating back to 1988. 

> <question-title></question-title>
>
> Was this entry ever not manually annotated?
>
> > <solution-title></solution-title>
> >
> > To answer this question you can scroll back in time through the table and check the `Database` column. Was this ever in TrEMBL instead that in SwissProt? No, so this entry was manually annotated since its beginning.
> >
> {: .solution}
{: .question}


topic: data-science
tutorial: python-conda
title: Conda Environments For Software Development
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["[]"]
tags_json: ["conda"]
content_hash: eff554d57b37ccfa6b5688d8a90e7421
last_modified: 2026-05-15T22:28:11.101717
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-conda/tutorial.html

content:
Conda environments, like Python Virtual Environments allow you to easily manage your installed packages and prevent conflicts between different project's dependencies. This tutorial follows an identical structure to the virtualenv tutorial, but with conda.

> <comment-title></comment-title>
>
> This tutorial is significantly based on [the Carpentries](https://carpentries.org) lesson ["Intermediate Research Software Development"](https://carpentries-incubator.github.io/python-intermediate-development/).
>
{: .comment}


If you have a python project you are using, you will often see something like
following two lines somewhere at the top.

```python
from matplotlib import pyplot as plt
import numpy as np
```

This means that our code requires two *external libraries* (also called third-party packages or dependencies) -
`numpy` and `matplotlib`.

Python applications often use external libraries that don’t come as part of the standard Python distribution. This means
that you will have to use a *package manager* tool to install them on your system.

Applications will also sometimes need a
specific version of an external library (e.g. because they require that a particular
bug has been fixed in a newer version of the library), or a specific version of Python interpreter.
This means that each Python application you work with may require a different setup and a set of dependencies so it
is important to be able to keep these configurations separate to avoid confusion between projects.
The solution for this problem is to create a self-contained 
*environment* per project, which contains a particular version of Python installation plus a number of
additional external libraries.

If you see something like 

```python
import pysam
```

You know you'll need additional packages installed on your system, as it relies on [htslib, a C library for working with HTS data](https://github.com/samtools/htslib). This usually means installing additional packages and things that are not always available from within Python's packaging ecosystem.

Conda environments go beyond virtual environments, and make it easier to develop, run, test and share code with others. In this tutorial, we learn how
to set up an environment to develop our code and manage our external dependencies.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Conda Environments

So what exactly are conda environments, and why use them?

A conda environment is an **isolated working copy** of specific versions of
one of more packages and all of their dependencies.

This is in fact simply a *directory with a particular structure* which includes
links to and enables multiple side-by-side installations of different packages
or different versions of the same external library to coexist on your machine
and only one to be selected for each of our projects. This allows you to work on
a particular project without worrying about affecting other projects on your
machine.

As more external libraries are added to your project over time, you can add them to
its specific environment and avoid a great deal of confusion by having separate (smaller) environments
for each project rather than one huge global environment with potential package version clashes. Another big motivator
for using environments is that they make sharing your code with others much easier (as we will see shortly).
Here are some typical scenarios where the usage of environments is highly recommended (almost unavoidable):

- You have two dependencies with conflicting dependencies! You cannot install
  the specific version of software X alongside software Y, as they both depend
  on different versions of a dependency, that cannot co-exist. This is solved by
  having different environments for both.
- You have an older project that only works under Python 2. You do not have the time to migrate the project to Python 3
  or it may not even be possible as some of the third party dependencies are not available under Python 3. You have to
  start another project under Python 3. The best way to do this on a single machine is to set up two separate Python
  environments.
- One of your Python 3 projects is locked to use a particular older version of a third party dependency. You cannot use the
  latest version of the
  dependency as it breaks things in your project. In a separate branch of your project, you want to try and fix problems
  introduced by the new version of the dependency without affecting the working version of your project. You need to set up
  a separate conda environments for your branch to 'isolate' your code while testing the new feature.

You do not have to worry too much about specific versions of external libraries that your project depends on most of the time.
Conda environments enable you to always use the latest available version without specifying it explicitly.
They also enable you to use a specific older version of a package for your project, should you need to.

> <tip-title>A Specific Package Version is Only Ever Installed Once</tip-title>
> Note that you will not have a separate package installations for each of your projects - they will only
> ever be installed once on your system (in `$CONDA/pkgs`) but will be referenced from different environments.
{: .tip}

### Managing Conda Environments

There are several commonly used command line tools for managing environments:
- `homebrew`, historically used on OSX to manage packages.
- `nix`, which has a steep learning curve but allows you to declare the state of your entire system
- `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community)
- `docker` and `singularity` are somewhat similar to other environment managers, as they can have isolated images with software and dependencies.
- Other, language specific managers

While there are pros and cons for using each of the above, all will do the job of managing
environments for you and it may be a matter of personal preference which one you go for. The Galaxy project is heavily invested in the Conda ecosystem and recommends it as an entry point as it is the most generally useful, and convenient. [The BioConda ecosystem](http://bioconda.github.io/) provides an unbelievably large number of packages for bioinformatics specific purposes, which makes it a good choice in general.

### Managing Packages

Part of managing your (virtual) working environment involves installing, updating and removing external packages
on your system. The Conda command (`conda`) is most commonly used for this - it interacts
 and obtains the packages from one or more Conda repositories (e.g. Conda Forge, BioConda, etc.)

> <tip-title>A Note on Anaconda and `conda`</tip-title>
> Anaconda is an open source Python
> distribution commonly used for scientific programming - it conveniently installs Python, package and environment management `conda`, and a 
> number of commonly used scientific computing packages so you do not have to obtain them separately. 
> `conda` is an independent command line tool (available separately from the Anaconda distribution too) with dual functionality: (1) it is a package manager that helps you find Python packages from
> remote package repositories and install them on your system, and (2) it is also a virtual environment manager. So, you can use `conda` for both tasks instead of using `venv` and `pip`.
{: .tip}

`venv` and `pip` are considered the *de facto* standards for environment and package management for Python 3.
However, the advantages of using Anaconda and `conda` are that you get (most of the) packages needed for
scientific code development included with the distribution. If you are only collaborating with others who are also using
Anaconda, you may find that `conda` satisfies all your needs. 

It is good, however, to be aware of all these tools (`pip`, `venv`, `pyenv`, etc.),
and use them accordingly. As you become more familiar with them you will realise that equivalent tools work in a similar
way even though the command syntax may be different (and that there are equivalent tools for other programming languages
too to which your knowledge can be ported).

![Python environment hell XKCD comic  showing boxes like pip, easy_install, homebrew 2.7, anaconda, homebrew 3.6, /usr/local/Cellar, ~/python/, and a chaotic mess of arrows moving between them all. At the bottom is the text: My python environment has become so degraded that my laptop has been declared a superfund site. (A superfund site is generally an environmental disaster area.)](../../images/xkcd/python_environment.png "Python Environment Hell from XKCD 1987 (CC-BY-NC 2.5)")

Let us have a look at how we can create and manage environments and their packages from the command line using `conda`.

### Instaling Miniconda

We will use Miniconda, a minimal conda installer that is commonly used, in place of the larger and slower to download full anaconda distribution.

> <hands-on-title>Installing Conda via Miniconda</hands-on-title>
> 1. Go to the [Miniconda installation page](https://docs.conda.io/en/latest/miniconda.html) and find the appropriate installer for your system.
> 2. Download and run the script.
> 3. You will probably need to close, and restart your terminal.
> 4. Check that you can run the `conda` command, otherwise something may have gone wrong.
{: .hands_on}

*If* you're running on Linux *and* following this tutorial via a Jupyter/CoCalc notebook, and you agree to the [Anaconda terms of service](https://legal.anaconda.com/policies/en/?name=terms-of-service), you can simply run the following cell:

```bash
wget -c https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b 
```

Now you can add it to your `~/.bashrc` which will cause Conda to be automatically loaded whenever you open a terminal:

```
~/miniconda3/bin/conda init bash
```

> <tip-title>Reopen Shell / Restart Jupyter Kernel</tip-title>
> Here you will need to restart your kernel, or if you're in a desktop environment, restart your terminal.
{: .tip}

### Installing our First Package

Let's install our first package, the new `libmamba` solver for Conda, as an example of how to install a package. A side benefit is that it will significant speed up your package installations!

```bash
conda install -y -q conda-libmamba-solver=22.8
```

> <tip-title>Does it get stuck?</tip-title>
> This step we have sometimes seen get "stuck", it will finish executing the transaction and hang, despite successfully installing the software. You can restart the kernel if this happens.
{: .tip}

Here we see a few things:

- `-y` - installs without asking questions like "do you want to do this". Generally people don't use this, but in a Notebook environment it's a bit nicer.
- `-q` - quiet installation, by default it prints a *lot* of progress update messages.
- `conda-libmamba-solver=22.8`, the package and version of that package that we wish to install.


We'll now configure conda to use `mamba` by default:

```bash
conda config --set experimental_solver libmamba
```

While we're at it, let's configure Conda to use the same default repositories as Galaxy:

```bash
conda config --add channels bioconda
conda config --add channels conda-forge
```

This will give us access to the vast repositories of BioConda (bioinformatics software) and Conda Forge (languages and libraries).

### Creating a new Environment
Creating a new environment is done by executing the following command:

```bash
conda create -y -n my-env
```

where `my-env` is any arbitrary name for this Conda environment. Environment names are global, so pick something meaningful when you create one!

For our project, let's create an environment called `hts`

```bash
conda create -y -n hts
```

You can list all of the created environments with

```bash
conda env list 
```

You'll notice that there is a `base` environment created by default, where you can install packages and play around with Conda. We do not recommend installing things into the `base` environment, if at all possible. Create a new environment for each tool you need to install

> <tip-title>Why separate environments?</tip-title>
> Conda's package resolution takes into account every other package installed in an environment. Especially if you use R packages, this can result in environments taking an inreasing amount to time to install new packages and resolve all of the dependencies.
>
> Thus by using isolated environments, you can be sure package resolution is quite fast.
{: .tip}

Once you’ve created an environment, you will need to activate it:

```bash
conda activate hts
```

Activating the environment will change your command line’s prompt to show what environment
you are currently using (indicated by its name in round brackets at the start of the prompt),
and modify the environment so that any packages you install will be available on the CLI.

When you’re done working on your project, you can exit the environment with:

```bash
conda deactivate
```

If you've just done the `deactivate`, ensure you reactivate the environment ready for the next part:

```bash
conda activate hts
```

### Installing External Libraries in an Environment

We noticed earlier that our code depends on two *external libraries* - `numpy` and `matplotlib` as well as `pysam` which depends on `htslib`. In order for the code to run on your machine, you need to
install these dependencies into your environment.

To install the latest version of a package with `conda` you use conda's `install` command and specify the package’s name, e.g.:

```bash
conda install -y -q python=3 numpy matplotlib pysam
```

Note that we needed to pick a version of python that we'd use, here we specify  `python=3` meaning "any Python version that starts with 3", so it won't use Python 2.7 or a future Python 4.

If you run the `conda install` command on a package that is already installed, `conda` will notice this and do nothing.

To install a specific version of a package give the package name followed by `=` and the version number, e.g.
`conda install numpy=1.21.1`.

To specify a minimum version of a Python package, you can
do `pip3 install 'numpy>=1.20'`.

To upgrade a package to the latest version, e.g. `conda update numpy`. (If it's at the latest version it will attempt to downgrade the package)

To display information about the current environment:

```bash
conda info
```

To display information about a particular package installed in your current environment:

```bash
conda list python
```

To list all packages installed with `pip` (in your current environment):

```bash
conda list
```

To uninstall a package installed in the environment do: `conda remove package-name`.
You can also supply a list of packages to uninstall at the same time.

### Exporting/Importing an Environment with `conda`

You are collaborating on a project with a team so, naturally, you will want to share your environment with your
collaborators so they can easily 'clone' your software project with all of its dependencies and everyone
can replicate equivalent environments on their machines. `conda` has a handy way of exporting,
saving and sharing environments.

To export your active environment - use `conda env export` command to
produce a list of packages installed in the environment.
A common convention is to put this list in a `environment.yml` file:

```bash
conda env export > environment.yml
cat environment.yml
```

The first of the above commands will create a `environment.yml` file in your current directory.
The `environment.yml` file can then be committed to a version control system and
get shipped as part of your software and shared with collaborators and/or users. They can then replicate your environment and
install all the necessary packages from the project root as follows:

```bash
conda env create -y -f environment.yml
```

The name is bundled directly into the environment, someone else re-creating
this environment from the yaml file will also be able to `conda activate hts`
afterwards. If you want it under a different name, you can use the `-n` flag to
supply your own name.

As your project grows - you may need to update your environment for a variety of reasons. For example, one of your project's dependencies has
just released a new version (dependency version number update), you need an additional package for data analysis
(adding a new dependency) or you have found a better package and no longer need the older package (adding a new and
removing an old dependency). What you need to do in this case (apart from installing the new and removing the
packages that are no longer needed from your environment) is update the contents of the `environment.yml` file
accordingly by re-issuing `conda env export` command and propagate the updated `environment.yml` file to your collaborators
via your code sharing platform (e.g. GitHub).

> <tip-title>Official Documentation</tip-title>
> For a full list of options and commands, consult the [official `conda` documentation](https://docs.conda.io/projects/conda/en/latest/commands.html)
{: .tip}


topic: data-science
tutorial: python-advanced-np-pd
title: Advanced Python
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 8e6832bd46c20fd5b265af65d333c366
last_modified: 2026-05-15T22:28:11.100121
zenodo_link: https://zenodo.org/record/3477564
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-advanced-np-pd/tutorial.html

content:
In this lesson, we will be using Python 3 with some of its most popular scientific libraries. This tutorial assumes that the reader is familiar with the fundamentals of the Python programming language, as well as, how to run Python programs using Galaxy. Otherwise, it is advised to follow the "Introduction to Python" tutorial available in the same platform. We will be using JupyterNotebook, a Python interpreter that comes with everything we need for the lesson. Please note:  JupyterNotebook is only currently available on the [usegalaxy.eu](https://usegalaxy.eu/) and [usegalaxy.org](https://usegalaxy.org/) sites.

> <comment-title></comment-title>
> This tutorial is **significantly** based on [the Carpentries](https://carpentries.org) [Programming with Python](https://swcarpentry.github.io/python-novice-inflammation/) and [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/), which is licensed CC-BY 4.0.
>
> Adaptations have been made to make this work better in a GTN/Galaxy environment.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Analyze data using numpy
NumPy is a python library and it stands for Numerical Python. In general, you should use this library when you want to perform operations and manipulate numerical data, especially if you have matrices or arrays. To tell Python that we’d like to start using NumPy, we need to import it:

```python
import numpy as np
```
A Numpy array contains one or more elements of the same type. To examine the basic functions of the library, we will create an array of random data. These data will correspond to arthritis patients’ inflammation. The rows are the individual patients, and the columns are their daily inflammation measurements. We will use the `random.randint()` function. It has 4 arguments as inputs `randint(low, high=None, size=None, dtype=int)`. `low` nad `high` specify the limits of the random number generator. `size` determines the shape of the array and it can be an integer or a tuple.

```python
np.random.seed(2021)  #create reproducible work
random_data = np.random.randint(1, 25, size=(50,70))
```
If we want to check the data have been loaded, we can print the variable’s value:

```python
print(random_data)
```

Now that the data are in memory, we can manipulate them. First, let’s ask what type of thing data refers to:

```python
print(type(random_data))
```


The output tells us that data currently refers to an N-dimensional array, the functionality for which is provided by the NumPy library. These data correspond to arthritis patients’ inflammation. The rows are the individual patients, and the columns are their daily inflammation measurements.

The `type` function will only tell you that a variable is a NumPy array but won’t tell you the type of thing inside the array. We can find out the type of the data contained in the NumPy array.

```python
print(random_data.dtype)
```

This tells us that the NumPy array’s elements are integer numbers.

With the following command, we can see the array’s shape:

```python
print(random_data.shape)
```

The output tells us that the data array variable contains 50 rows and 70 columns. When we created the variable `random_data` to store our arthritis data, we did not only create the array; we also created information about the array, called members or attributes. This extra information describes `random_data` in the same way an adjective describes a noun. `random_data.shape` is an attribute of `random_data` which describes the dimensions of `random_data`.

If we want to get a single number from the array, we must provide an index in square brackets after the variable name, just as we do in math when referring to an element of a matrix. Our data has two dimensions, so we will need to use two indices to refer to one specific value:

```python
print('first value in data:', random_data[0, 0])
```

```python
print('middle value in data:', random_data[25, 35])
```


The expression random_data[25, 35] accesses the element at row 25, column 35. While this expression may not surprise you, random_data[0, 0] might. Programming languages like Fortran, MATLAB and R start counting at 1 because that’s what human beings have done for thousands of years. Languages in the C family (including C++, Java, Perl, and Python) count from 0 because it represents an offset from the first value in the array (the second value is offset by one index from the first value). As a result, if we have an M×N array in Python, its indices go from 0 to M-1 on the first axis and 0 to N-1 on the second.

Slicing data
An index like [25, 35] selects a single element of an array, but we can select whole sections as well, using slicing the same way as previously with the strings. For example, we can select the first ten days (columns) of values for the first four patients (rows) like this:

```python
print(random_data[0:4, 0:10])
```


We don’t have to include the upper and lower bound on the slice. If we don’t include the lower bound, Python uses 0 by default; if we don’t include the upper, the slice runs to the end of the axis, and if we don’t include either (i.e., if we use ‘:’ on its own), the slice includes everything:

```python
small = random_data[:3, 36:]
print('small is:')
print(small)
```

The above example selects rows 0 through 2 and columns 36 through to the end of the array.

## Process the data
NumPy has several useful functions that take an array as input to perform operations on its values. If we want to find the average inflammation for all patients on all days, for example, we can ask NumPy to compute random_data’s mean value:

```python
print(np.mean(random_data))
```


Let’s use three other NumPy functions to get some descriptive values about the dataset. We’ll also use multiple assignment, a convenient Python feature that will enable us to do this all in one line.

```python
maxval, minval, stdval = np.max(random_data), np.min(random_data), np.std(random_data)

print('maximum inflammation:', maxval)
print('minimum inflammation:', minval)
print('standard deviation:', stdval)
```


How did we know what functions NumPy has and how to use them? If you are working in IPython or in a Jupyter Notebook, there is an easy way to find out. If you type the name of something followed by a dot, then you can use tab completion (e.g. type `np.` and then press Tab) to see a list of all functions and attributes that you can use. After selecting one, you can also add a question mark (e.g. `np.cumprod?`), and IPython will return an explanation of the method! This is the same as doing `help(np.cumprod)`.

When analyzing data, though, we often want to look at variations in statistical values, such as the maximum inflammation per patient or the average inflammation per day. One way to do this is to create a new temporary array of the data we want, then ask it to do the calculation:

```python
patient_0 = random_data[0, :] # 0 on the first axis (rows), everything on the second (columns)
print('maximum inflammation for patient 0:', np.max(patient_0))
```


What if we need the maximum inflammation for each patient over all days (as in the next diagram on the left) or the average for each day (as in the diagram on the right)? As the diagram below shows, we want to perform the operation across an axis:

![A cartoon shows a data structure on the left with Patient 0 to N as rows in a table, and then Day 0 to N as columns of the same table. Below is written "numpy.max(data, axis=1)" which does an aggregation and finds the max per patient (so maximum daily value for each patient.) On the right a similar table is shown but written below is numpy.max(data, axis=0) and now the maximum is calculated per day, which patient had the maximum value.](../../images/python-advanced-np-pd/Figure8_Operations_across_axis.png)

To support this functionality, most array functions allow us to specify the axis we want to work on. If we ask for the average across axis 0 (rows in our 2D example), we get:

```python
print(np.mean(random_data, axis=0))
```


As a quick check, we can ask this array what its shape is:

```python
print(np.mean(random_data, axis=0).shape)
```


The expression (70,) tells us we have an N×1 vector, so this is the average inflammation per day for all patients. If we average across axis 1 (columns in our 2D example), we get the average inflammation per patient across all days.:

```python
print(np.mean(random_data, axis=1))
```


### Stacking arrays
Arrays can be concatenated and stacked on top of one another, using NumPy’s `vstack` and `hstack` functions for vertical and horizontal stacking, respectively.

```python
import numpy as np

A = np.array([[1,2,3], [4,5,6], [7, 8, 9]])
print('A = ')
print(A)

B = np.hstack([A, A])
print('B = ')
print(B)

C = np.vstack([A, A])
print('C = ')
print(C)
```


### Remove NaN values

Sometimes there are missing values in an array, that could make it difficult to perform operations on it. To remove the `NaN` you must first find their indexes and then replace them. The following example replaces them with `0`.

```python
a = array([[1, 2, 3], [0, 3, NaN]])
print(a)
a[np.isnan(a)] = 0
print(a)
```


> <question-title>Selecting and stacking arrays</question-title>
>
> Write some additional code that slices the first and last columns of A, and stacks them into a 3x2 array. Make sure to print the results to verify your solution.
>
> > <solution-title></solution-title>
> >
> > A ‘gotcha’ with array indexing is that singleton dimensions are dropped by default. That means `A[:, 0]` is a one dimensional array, which won’t stack as desired. To preserve singleton dimensions, the index itself can be a slice or array. For example, `A[:, :1]` returns a two dimensional array with one singleton dimension (i.e. a column vector).
> > ```
> > D = np.hstack((A[:, :1], A[:, -1:]))
> > print('D = ')
> > print(D)
> > ```
> >
> {: .solution}
{: .question}

> <question-title>Selecting with conditionals</question-title>
>
> Given the followind array `A`, keep only the elements that are lower that `0.05`.
>```
>A = np.array([0.81, 0.025, 0.15, 0.67, 0.01])
>```
> > <solution-title></solution-title>
> >
> > ```
> > A = A[A<0.05]
> > ```
> >
> {: .solution}
{: .question}



# Use pandas to work with dataframes

Pandas ({% cite reback2020pandas %}, {% cite mckinney-proc-scipy-2010 %}) is a widely-used Python library for statistics, particularly on tabular data. If you are familiar with R dataframes, then this is the library that integrates this functionality. A dataframe is a 2-dimensional table with indexes and column names. The indexes indicate the difference in rows, while the column names indicate the difference in columns. You will see later that these two features are useful when you’re manipulating your data. Each column can contain different data types.

Load it with import pandas as `pd`. The alias `pd` is commonly used for pandas.

```python
import pandas as pd
```

There are many ways to create a pandas dataframe. For example you can use a numpy array as input.

```python
data = np.array([['','Col1','Col2'],
                ['Row1',1,2],
                ['Row2',3,4]])

print(pd.DataFrame(data=data[1:,1:],
                  index=data[1:,0],
                  columns=data[0,1:]))
```



For the purposes of this tutorial, we will use a file with the annotated differentially expressed genes that was produced in the [Reference-based RNA-Seq data analysis]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial

We can read a tabular file with `pd.read_csv`. The first argument is the filepath of the file to be read. The `sep` argument refers to the symbol used to separate the data into different columns. You can check the rest of the arguments using the `help()` function.

```python
data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t")
print(data)
```

The columns in a dataframe are the observed variables, and the rows are the observations. Pandas uses backslash `\` to show wrapped lines when output is too wide to fit the screen.

## Explore the data

You can use `index_col` to specify that a column’s values should be used as row headings.

By default row indexes are numbers, but we could use a column of the data. To pass the name of the column to `read_csv`, you can use its `index_col` parameter. Be careful though, because the row indexes must be unique for each row.

```python
data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
print(data)
```

You can use the `DataFrame.info()` method to find out more about a dataframe.

```python
data.info()
```


We learn that this is a DataFrame. It consists of 130 rows and 12 columns. None of the columns contains any missing values. 6 columns contain 64-bit floating point `float64` values, 2 contain 64-bit integer `int64` values and 4 contain character `object` values. It uses 13.2KB of memory.

The `DataFrame.columns` variable stores information about the dataframe’s columns.

Note that this is an attribute, not a method. (It doesn’t have parentheses.) Called a member variable, or just member.

```python
print(data.columns)
```


You could use `DataFrame.T` to transpose a dataframe. The `Transpose` (written `.T`) doesn’t copy the data, just changes the program’s view of it. Like columns, it is a member variable.

```python
print(data.T)
```

You can use `DataFrame.describe()` to get summary statistics about the data. `DataFrame.describe()` returns the summary statistics of only the columns that have numerical data.  All other columns are ignored, unless you use the argument `include='all'`. Depending on the data type of each column, the statistics that can't be calculated are replaced with  the value `NaN`.

```python
print(data.describe(include='all'))
```


> <question-title>Using pd.head and pd.tail</question-title>
>
> After reading the data, use `help(data.head)` and `help(data.tail)` to find out what `DataFrame.head` and `DataFrame.tail` do.
> 	a. What method call will display the first three rows of the data?
> 	b. What method call will display the last three columns of this data? (Hint: you may need to change your view of the data.)
>
> > <solution-title></solution-title>
> >
> > a. We can check out the first five rows of the data by executing `data.head()` (allowing us to view the head of the DataFrame). We can specify the number of rows we wish to see by specifying the parameter `n` in our call to `data.head()`. To view the first three rows, execute:
> > ```
> > data.head(n=3)
> > ```
> >
> > |  | Base mean | log2(FC) | StdErr | Wald-Stats | P-value | P-adj | Chromosome | Start | End | Strand | Feature | Gene name | GeneID |
> > | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
> > | FBgn0039155 | 1086.974295 | -4.148450 | 0.134949 | -30.740913 | 1.617357e-207 | 1.387207e-203 | chr3R | 24141394 | 24147490 | + | protein_coding | Kal1 |
> > | FBgn0003360 | 6409.577128 | -2.999777 | 0.104345 | -28.748637 | 9.419922e-182 | 4.039734e-178 | chrX | 10780892 | 10786958 | - | protein_coding | sesB |
> > | FBgn0026562 | 65114.840564 | -2.380164 | 0.084327 | -28.225437 | 2.850430e-175 | 8.149380e-172 | chr3R | 26869237 | 26871995 | - | protein_coding | BM-40-SPARC |
> >
> >
> >    b. To check out the last three rows, we would use the command, `data.tail(n=3)`, analogous to `head()` used above. However, here we want to look 	at the last three columns so we need to change our view and then use `tail()`. To do so, we create a new DataFrame in which rows and columns are 	switched:
> >```
> >data_flipped = data.T
> >```
> > We can then view the last three columns of the data by viewing the last three rows of data_flipped:
> > ```
> > data_flipped.tail(n=3)
> > ```
> > | GeneID | FBgn0039155 | FBgn0003360 | FBgn0026562 | FBgn0025111 | FBgn0029167 | FBgn0039827 | FBgn0035085 | FBgn0034736 | FBgn0264475 | FBgn0000071 | ... | FBgn0264343 | FBgn0038237 | FBgn0020376 | FBgn0028939 | FBgn0036560 | FBgn0035710 | FBgn0035523 | FBgn0038261 | FBgn0039178 | FBgn0034636 |
> > | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
> > | Strand | + | - | - | - | + | + | + | + | + | + | ... | + | - | + | + | + | - | + | + | + | - |
> > | Feature | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | lincRNA | protein_coding | ... | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding | protein_coding |
> > | Gene name | Kal1 | sesB | BM-40-SPARC | Ant2 | Hml | CG1544 | CG3770 | CG6018 | CR43883 | Ama | ... | CG43799 | Pde6 | Sr-CIII | NimC2 | CG5895 | SP1173 | CG1311 | CG14856 | CG6356 | CG10440 |
> >
> {: .solution}
{: .question}

 > <question-title>Saving in a csv file</question-title>
>
> As well as the `read_csv` function for reading data from a file, Pandas provides a `to_csv` function to write dataframes to files. Applying what you’ve learned about reading from files, write one of your dataframes to a file called `processed.csv`. You can use `help` to get information on how to use `to_csv`.
>
> > <solution-title></solution-title>
> >
> > ```
> > data_flipped.to_csv('processed.csv')
> > ```
> >
> {: .solution}
{: .question}


* Note about Pandas DataFrames/Series

A [DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) is a collection of [Series](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.html); The DataFrame is the way Pandas represents a table, and Series is the data-structure Pandas use to represent a column.

Pandas is built on top of the Numpy library, which in practice means that most of the methods defined for Numpy Arrays apply to Pandas Series/DataFrames.

What makes Pandas so attractive is the powerful interface to access individual records of the table, proper handling of missing values, and relational-databases operations between DataFrames.

## Select data
To access a value at the position `[i,j]` of a DataFrame, we have two options, depending on what is the meaning of i in use. Remember that a DataFrame provides an index as a way to identify the rows of the table; a row, then, has a position inside the table as well as a label, which uniquely identifies its entry in the DataFrame.

You can use `DataFrame.iloc[..., ...]` to select values by their (entry) position and basically specify location by numerical index analogously to 2D version of character selection in strings.

```python
print(data.iloc[0, 0])
```
You can also use `DataFrame.loc[..., ...]` to select values by their (entry) label and basically specify location by row name analogously to 2D version of dictionary keys.

```python
print(data.loc["FBgn0039155", "Base mean"])
```

You can use Python's usual slicing notation, to select all or a subset of rows and/or columns. For example, the following code selects all the columns of the row `"FBgn0039155"`.

```python
print(data.loc["FBgn0039155", :])
```

Which would get the same result as printing `data.loc["FBgn0039155"]` (without a second index).

You can select multiple columns or rows using `DataFrame.loc` and a named slice or `Dataframe.iloc` and the numbers corresponding to the rows and columns.

```python
print(data.loc['FBgn0003360':'FBgn0029167', 'Base mean':'Wald-Stats'])
print(data.iloc[1:4 , 0:3])
```

* Note the difference between the 2 outputs.

When choosing or transitioning between `loc` and `iloc`, you should keep in mind that the two methods use slightly different indexing schemes.

`iloc` uses the Python stdlib indexing scheme, where the first element of the range is included and the last one excluded. So `0:10` will select entries `0,...,9`. `loc`, meanwhile, indexes inclusively. So `0:10` will select entries `0,...,10`.

This is particularly confusing when the DataFrame index is a simple numerical list, e.g. `0,...,1000`. In this case `df.iloc[0:1000]` will return 1000 entries, while `df.loc[0:1000]` return 1001 of them! To get 1000 elements using `loc`, you will need to go one lower and ask for `df.loc[0:999]`.

The result of slicing is a new dataframe and can be used in further operations. All the statistical operators that work on entire dataframes work the same way on slices. E.g., calculate max of a slice.

```python
print(data.loc['FBgn0003360':'FBgn0029167', 'Base mean'].max())
```


## Use conditionals to select data

You can use conditionals to select data. A comparison is applied element by element and returns a similarly-shaped dataframe of `True` and `False`. The last one can be used as a mask to subset the original dataframe. The following example creates a new dataframe consisting only of the columns 'P-adj' and 'Gene name', then keeps the rows that comply with the expression `'P-adj' < 0.000005`

```python
subset = data.loc[:, ['P-adj', 'Gene name']]
print(subset)
```

```python
mask = subset.loc[:, 'P-adj'] < 0.000005
new_data = subset[mask]
print(new_data)
```

If we have not had specified the column, that the expression should be applied to, then it would have been applied to the entire dataframe. But the dataframe contains different type of data. In that case, an error would occur.

Consider the following example of a dataframe consisting only of numerical data. The expression and the mask would be normally applied to the data and the mask would return `NaN` for the data that don't comply with the expression.

```python
subset = data.loc[:, ['StdErr',	'Wald-Stats', 'P-value', 'P-adj']]
mask = subset < 0.05
new_data = subset[mask]
print(new_data)
```

This is very useful because NaNs are ignored by operations like max, min, average, etc.

```python
print(new_data.describe())
```

> <question-title>Manipulating dataframes</question-title>
>
> Explain what each line in the following short program does: what is in first, second, etc.?
>
>```
>first = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
>second = first[first['log2(FC)'] > 0 ]
>third = second.drop('FBgn0025111')
>fourth = third.drop('StdErr', axis = 1)
>fourth.to_csv('result.csv')
>```
> > <solution-title></solution-title>
> >
> > Let’s go through this piece of code line by line.
> > ```
> > first = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
> > ```
> > This line loads the data into a dataframe called first. The `index_col='GeneID'` parameter selects which column to use as the row labels in the dataframe.
> >
> > ```
> > second = first[first['log2(FC)'] > 0 ]
> > ```
> >
> > This line makes a selection: only those rows of first for which the ‘log2(FC)’ column contains a positive value are extracted. Notice how the Boolean expression inside the brackets is used to select only those rows where the expression is true.
> >
> > ```
> > third = second.drop('FBgn0025111')
> > ```
> >
> > As the syntax suggests, this line drops the row from second where the label is ‘FBgn0025111’. The resulting dataframe third has one row less than the original dataframe second.
> >
> > ```
> > fourth = third.drop('StdErr', axis = 1)
> > ```
> >
> > Again we apply the drop function, but in this case we are dropping not a row but a whole column. To accomplish this, we need to specify also the axis parameter.
> >
> > ```
> > fourth.to_csv('result.csv')
> > ```
> >
> >The final step is to write the data that we have been working on to a csv file. Pandas makes this easy with the `to_csv()` function. The only required argument to the function is the filename. Note that the file will be written in the directory from which you started the Jupyter or Python session.
> >
> {: .solution}
{: .question}


> <question-title>Finding min-max indexes</question-title>
>
> Explain in simple terms what `idxmin` and `idxmax` do in the short program below. When would you use these methods?
> ```
> data = pd.read_csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep = "\t", index_col = 'GeneID')
>
> print(data['Base mean'].idxmin())
> print(data['Base mean'].idxmax())
> ```
>
> > <solution-title></solution-title>
> >
> > `idxmin` will return the index value corresponding to the minimum; idxmax will do the same for the maximum value.
> >
> > You can use these functions whenever you want to get the row index of the minimum/maximum value and not the actual minimum/maximum value.
> >
> > Output:
> > FBgn0063667
> > FBgn0026562
> >
> {: .solution}
{: .question}

> <question-title>Selecting with conditionals</question-title>
>
> Assume Pandas has been imported and the previous dataset has been loaded. Write an expression to select each of the following:
> a. P-value of each gene
> b. all the information of gene `FBgn0039178`
> c. the information of all genes that belong to chromosome `chr3R`
>
> > <solution-title></solution-title>
> >
> > a. ```data['P-value']```
> > b.```data.loc['FBgn0039178', :]```
> > c. ```data[data['Chromosome'] == 'chr3R']```
> >
> {: .solution}
{: .question}



## Group-by and analyze the data

Many data analysis tasks can be approached using the “split-apply-combine” paradigm: split the data into groups, apply some analysis to each group, and then combine the results.

Pandas makes this very easy through the use of the `groupby()` method, which splits the data into groups. When the data is grouped in this way, the aggregate method `agg()` can be used to apply an aggregating or summary function to each group.

```python
summarised_data = data.groupby('Chromosome').agg({'Base mean':'first',
                             'log2(FC)': 'max'})
print(summarised_data)
```

There are a couple of things that should be noted. The `agg()` method accepts a dictionary as input that specifies the function to be applied to each column. The output is a new dataframe, that each row corresponds to one group. The output dataframe uses the grouping column as index. We could change the last one by simply using the `reset_index()` method.

```python
summarised_data = data.groupby('Chromosome').agg({'Base mean':'first',
                                                  'log2(FC)': 'max'}).reset_index()
print(summarised_data)
```

> <question-title>Finding the max of each group</question-title>
>
> Using the same dataset, try to find the longest genes in each chromosome.
>
>> <solution-title></solution-title>
> >
> > ```
> > data['Gene Length'] = data['End'] - data['Start']
> > data.groupby('Chromosome').agg(max_length = ('Gene Length', 'max'))
> > ```
> >
> {: .solution}
{: .question}


> <question-title>Grouping with multiple variables</question-title>
>
> Using the same dataset, try to find how many genes are found on each strand of each chromosome.
>
> > <solution-title></solution-title>
> >
> > You can group the data according to more than one column.
> > ```
> > data.groupby(['Chromosome', 'Strand']).size()
> > ```
> {: .solution}
{: .question}

# Conclusion

This tutorial aims to serve as an introduction to data analysis using the Python programming language. We hope you feel more confident in Python!


topic: data-science
tutorial: sql-game
title: SQL Educational Game - Murder Mystery
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["game", "SQL"]
content_hash: 7ea4d26c9e37ece7cc088606fc89c416
last_modified: 2026-05-15T22:28:11.120845
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/sql-game/tutorial.html

content:
This is not a tutorial like most GTN content but a fun exercise for you to play around and learn a bit about SQL in a more 'practical', and hopefully re-inforce the skills you covered in Basic and Advanced SQL skills. It makes use of the [NUKnightLab/sql-mysteries](https://github.com/NUKnightLab/sql-mysteries) SQL murder mystery project and released under open licenses:

> <comment-title>Source</comment-title>
> Original code for NUKnightLab/sql-mysteries is released under the MIT License.
> Original text and other content is released under Creative Commons CC BY-SA 4.0.
{: .comment}

Download the database and connector:

```python
# This preamble sets up the sql "magic" for jupyter. Use %%sql in your cells to write sql!
!python3 -m pip install ipython-sql sqlalchemy
!wget -c https://github.com/NUKnightLab/sql-mysteries/raw/master/sql-murder-mystery.db
```

Setup the database connection:

```python
import sqlalchemy
engine = sqlalchemy.create_engine("sqlite:///sql-murder-mystery.db")
%load_ext sql
%sql sqlite:///sql-murder-mystery.db
%config SqlMagic.displaycon=False
```

## Tables

Which tables are available to you? What columns do they contain? Here's a handy reference for you:

```python
import pandas as pd
from sqlalchemy import MetaData
m = MetaData()
m.reflect(engine)
results = []
for table in m.tables.values():
    results.append([table.name, ', '.join([c.name for c in table.c])])
pd.set_option('display.max_colwidth', None)
pd.DataFrame(results, columns=["Table", "Columns"])
```


## The search!

Begin your search for the truth

> A crime has taken place and the police are both useless *and* corrupt and it
> is up to you and your community to solve the mystery. They failed to secure
> their database, and now their crime scene reports are public, it is time to
> figure out who the murderer was.
>
> You know that the crime was a **murder** that occurred sometime on **Jan.15,
> 2018** and that it took place in **SQL City**.
>
> All the clues to this mystery are buried in a huge database, and you need to
> use SQL to navigate through this vast network of information. Your first step
> to solving the mystery is to retrieve the corresponding crime scene report
> from the police department’s database. From there, you can use your SQL
> skills to find the murderer.
{: .quote}

```python
%%sql
select * from crime_scene_report limit 8;
```

Try using the 'Insert → Cell Below' functionality to keep track of important query results as you go!

## Solution

Write the following queries in your SQL environment to check whether you've found the right murderer:

```python
%%sql
INSERT INTO solution VALUES (1, "Insert the name of the person you found here");
SELECT value FROM solution;
```


topic: data-science
tutorial: python-iterables
title: Python - Lists & Strings & Dictionaries
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["[]"]
tags_json: []
content_hash: 30df0f2dedfa5a5eeeeea1fbece9d31d
last_modified: 2026-05-15T22:28:11.103465
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/data-science/tutorials/python-iterables/tutorial.html

content:
Doing calculations with hundreds of variables called `pressure_001`, `pressure_002`, etc. would be at least as slow as doing them by hand. Using a *list* to store many values together solves that problems. Lists are surrounded by square brackets: `[`, `]`, with values separated by commas:

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


```python
pressures = [0.273, 0.275, 0.277, 0.275, 0.276]
print(f'pressures: {pressures}')
print(f'length: {len(pressures)}')
```

## Indexing

You can use an item's index to fetch it from a list.

```python
print(f'zeroth item of pressures: {pressures[0]}')
print(f'fourth item of pressures: {pressures[4]}')
```

## Replacement

Lists' values can be changed or replaced by assigning a new value to the position in the list.

```python
pressures[0] = 0.265
print(f'pressures is now: {pressures}')
```

Note how the first item has changed from `0.273`

## Appending

Appending items to a list lengthens it. You can do `list_name.append()` to add items to the end of a list.

```python
primes = [2, 3, 5]
print(f'primes is initially: {primes}')
primes.append(7)
print(f'primes has become: {primes}')
```


`.append()` is a *method* of lists. It's like a function, but tied to a particular object. You use `object_name.method_name` to call methods, which deliberately resembles the way we refer to things in a library.

We will meet other methods of lists as we go along. Use `help(list)` for a preview. `extend` is similar to `append`, but it allows you to combine two lists.  For example:

```python
teen_primes = [11, 13, 17, 19]
middle_aged_primes = [37, 41, 43, 47]
print(f'primes is currently: {primes}')
primes.extend(teen_primes)
print(f'primes has now become: {primes}')
primes.append(middle_aged_primes)
print(f'primes has finally become: {primes}')
```

Note that while `extend` maintains the "flat" structure of the list, appending a list to a list makes the result two-dimensional - the last element in `primes` is a list, not an integer.

This starts to become a more complicated data structure, and we'll use more of these later. A list containing both integers and a list can be called a "hetereogenous" list, since it has multiple different data types. This is relatively uncommon, most of the lists you'll encounter will have a single data type inside of them. Sometimes you'll see a list of lists, which can be used to store positions, like a chessboard.

## List Indices

In computer science and programming we number the positions within a list starting from `0`, rather than from `1`.

```python
# Position  0         1          2            3           4
weekdays = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday']
print(weekdays[0])
print(weekdays[4])
print(weekdays[3])
```

But if you try an access a position that is outside of the list, you'll get an error

```python
print(weekdays[9])
```

returns a `IndexError: list index out of range`.

> <tip-title>Reading Error Messages</tip-title>
> So how do you read this?
> ```
> 1 | ---------------------------------------------------------------------------
> 2 | IndexError                                Traceback (most recent call last)
> 3 | /tmp/ipykernel_648319/137030145.py in <module>
> 4 | ----> 1 print(weekdays[9])
> 5 |
> 6 | IndexError: list index out of range
> ```
>
> 1. This is just a line of `-`s as a separator
> 2. `IndexError`, here Jupyter/CoCalc/etc are trying to be helpful and highlight the error for us. This is the important bit of information!
> 3. This is the path to where the code is, Jupyter/CoCalc/etc create temporary files to execute your code.
> 4. Here an arrow points to the line number where something has broken. 1 shows that it's the first line within the cell, and it points to the print statement. Really it's pointing at the `weekdays[9]` within the print statement.
> 5. Blank
> 6. This is where we normally look for the **most important part of the Traceback**. The error message. An `IndexError`, namely that the list index (9) is out of the range of possible values (the length of the list.)
{: .tip}

However, sometimes you want to access the very end of a list! You can either start at the beginning and count along to find the last item or second to last item, or you can use Negative Indices

```python
# Position  0         1          2            3           4
weekdays = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday']
# Position  -5        -4         -3           -2          -1

print(weekdays[-1])
print(weekdays[-2])
print(weekdays[-4])
```

If you wanted to find the last value in a list, you could also use `len(elements)` and then subtract back to find the index you want

```python
elements[len(elements)-1]
```

This is essentially how negative indexes work, except you don't have to use `len(elements)`, that's done for you automatically.


## Removing Items.

You can use `del` to remove items from a list entirely. We use `del list_name[index]` to remove an element from a list (in the example, 9 is not a prime number) and thus shorten it. `del` is not a function or a method, but a statement in the language.

```python
primes = [2, 3, 5, 7, 9]
print(f'primes before removing last item: {primes}')
del primes[4]
print(f'primes after removing last item: {primes}')
```

## Empty Lists

The empty list contains no values. When you want to make a new list, use `[]` on its own to represent a list that doesn't contain any values. This is helpful as a starting point for collecting values, which we'll see soon.

## Heterogeneous Lists

Lists may contain values of different types. A single list may contain numbers, strings, and anything else.

```python
goals = [1, 'Create lists.', 2, 'Extract items from lists.', 3, 'Modify lists.']
```

## Strings are like Lists

Text is often called a "string" in the programming world. Strings of text like `name = "Helena"` or `patient_id = "19237zud830"` are very similar conceptually to lists. Except instead of being a list of numbers, they're a lists of characters.

In a number of older programming languages, strings are indeed arrays of numbers internally. However python hides a lot of that complexity from us, so we can just work with text.

Still, many of the operations you use on lists, can also be used on strings as well! Strings can be indexed like lists so you can get single elements from lists.

```python
element = 'carbon'
print(f'zeroth character: {element[0]}')
print(f'third character: {element[3]}')
```

Strings, however, cannot be modified, you can't change a single letter in a string. Things that cannot be modified after creation are called *immutable* or sometimes *frozen*, compared to things which can be modified which are called *mutable*.
Python considers the string to be a single value with parts, not a collection of values.

```python
element[0] = 'C'
```

## Bounds

You cannot access values beyond the end of the list, this will result in an error. Python reports an `IndexError` if we attempt to access a value that doesn't exist. This is a kind of **runtime error**, as it cannot be detected as the code is parsed. Imagine if you had a script which let you read in a file, depending on how many lines were in the file, whether index 90 was valid or invalid, would depend on how big your file was.

```python
print(f'99th element of element is: {element[99]}')
```

## Exercises

> <question-title>Checking suffixes</question-title>
>
> 1. How could you check that the extension of a filename is `.csv`
> 2. Can you find another way? Maybe check the help page for `str`
>
> > <solution-title></solution-title>
> >
> > 1. `a[-4:] == "csv"` (Here we use `==` for comparing two values)
> > 2. `a.endswith('.csv')`
> >
> {: .solution}
{: .question}

```python
# Test code here!
a = "1234.csv"
b = "1273.tsv"
c = "9382.csv"
d = "1239.csv"
```

> <question-title>Say it loud!</question-title>
>
> 1. Can you find a method in the `str`'s help that converts the string to upper case
> 2. or lower case?
> 3. Can you use it to fix mixed case DNA sequence?
>
> > <solution-title></solution-title>
> > 1. `"shout it out".upper()`
> > 2. `"WHISPER THIS".lower()`
> > 3. `terrible_sequence.upper()`
> {: .solution}
{: .question}

```python
# Test answers here!
print("shout it out")
print("WHISPER THIS")
# Fix this mess to be all capital
terrible_sequence = "AcTGAGccGGTt"
print(terrible_sequence)
```


> <question-title>Splitting</question-title>
> 1. We use `.split()` to split a string by some character. Here we have a comma separated list of values, try splitting that up by a comma, but we actually wanted it separated by `|` characters. Can you split it up, and then re-join it with that new character?
> 2. Does `help(str)` give you another option for replacing a character like that.
> 3. What happens if you split by another value like `3`?
>
> > <solution-title></solution-title>
> > 1. `data.split(",")`
> > 2. `data.replace(",", "|")`
> > 3. Those characters will disappear! If you want to reconstruct the same string
> {: .solution}
{: .question}

```python
# Split me
data = "0,0,1,3,1,2,4,7,8,3,3,3,10,5,7,4,7,7,12,18,6,13,11,11,7,7,4,6,8,8,4,4,5,7,3,4,2,3,0,0"
```

# Slicing & Dicing

All of the data types we've talked about today can be sliced, and this will be a key part of using lists.

```python
elements = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F']
# Instead of accessing a single element
print(elements[0])
# We'll access a range
print(elements[0:4])
```

Accessing only a portion of a list is commonly used, say if you have a list of FastQ files from paired end sequencing, perhaps you want two of them at a time. You could access those with `[0:2]`.

```python
# You don't need to start at 0
print(elements[6:8])
```

```python
# But your end should be bigger than your start.
# What do you think this will return?
# Make a guess before you run it
print(elements[6:5])
```

If you don't supply an end value, Python will default to going to the end of the list. Likewise, if you don't provide a start value, Python will use `0` as the start by default, until whatever end value you provide.


> <question-title>Valid and Invalid Slices</question-title>
> Which of these do you think will be valid? Which are invalid? Predict what they will return:
> ```
> # 1
> elements = ['H', 'He', 'Li', 'Be', 'B', 'C', 'N', 'O', 'F']
> # 2
> elements[0:3]
> # 3
> elements[:3]
> # 4
> elements[-3:3]
> # 5
> elements[-8:-3]
> # 6
> elements[:]
> # 7
> elements[0:20]
> # 8
> elements['H':'Li']
> # 9
> elements[1.5:]
> ```
>
> > <solution-title></solution-title>
> > All of these are valid except the last two.
> > 1. If you dont' fill in a position, Python will use the default. 0 for the left hand side of the `:`, and `len(elements)` for the right hand side.
> > 2. You can request a slice longer than your list (e.g. up to 20), but Python may not give you that many items back.
> > 3. List slicing can only be done with integers, not floats.
> {: .solution}
{: .question}

```python
# Check your answers here!
```

## Stride

However, list slicing can be more complicated. You can additionally use a 'stride' parameter, which is how Python should strep through the list. To take every other element from a list:

```python
values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
print(values[0:12:2]) # every other value
print(values[1:12:2]) # every other value from the second value
print(values[::2]) # the start and end are optional
print(values[::3]) # every third value in the list.
```

So list slicing together is either `list[low:high]` or `list[low:high:stride]`, where low and high are optional if you just want to go to the end of the list.

## Sorting

Lists occasionally need to be sorted. For example, you have a list of students you might want to alphabetise, and here you can use the function `sorted` to help you.

```python
students = [
    'Koos Christabella',
    'Zackary Habiba',
    'Jumana Rostam',
    'Sorina Gaia',
    'Kalyani Bessarion',
    'Enéas Nirmala',
    '王奕辰',
    '刘依诺',
]
students = sorted(students)
print(students)
```

> <tip-title>Sorting names is hard!</tip-title>
> Some people have 1 name, some have 4 or more! Some cultures have surnames first, some not. Sorting names is a complex situation, so be sure you consider your data before sorting and assuming it's correct. Test with multiple values to make sure your code works!
{: .tip}

> <tip-title>Results can be dependent on analysis order!</tip-title>
> Some analyses (especially simultaions) can be dependent on data input order or data sorting. This was recently seen in {% cite Bhandari_Neupane_2019 %} where the data files used were sorted one way on Windows, and another on Linux, resulting in different results for the same code and the same datasets! Yikes!
>
> If you know your analyses are dependent on file ordering, then you can use `sorted()` to make sure the data is provided in a uniform way every time.
>
> If you're not sure if your results will be dependent, you can try sorting anyway. Or better yet, randomising the list of inputs to make sure your code behaves properly in any scenario.
{: .tip}

## Type Conversion

Just list with converting `"1.5"` to an float with the `float()` function, or `3.1` to a string with `str()`, we can do the same with lists using the `list()` function, and sets with `set()`:

```python
# Convert text to a list
print(list("sometext"))
```

Converting a list back into text is likewise possible, but you need to use the special function `join`. Join is a function of a `str`, which accepts a list

```python
word = ['c', 'a', 'f', 'e']
print("-".join(word))
```

It takes the string you called it on, and uses that as a separator. Then for the list that you provide, it joins that together with the separator.

# Exercise Time

> <question-title>Fill in the Blanks</question-title>
>
> Fill in the blanks so that the program below produces the output shown.
>
> ```
> values = ____
> values.____(1)
> values.____(3)
> values.____(5)
> print(f'first time: {values}')
> values = values[____]
> print(f'second time: {values}')
> ```
>
> ```
> first time: [1, 3, 5]
> second time: [3, 5]
> ```
>
> > <solution-title></solution-title>
> > ```
> > values = []
> > values.append(1)
> > values.append(3)
> > values.append(5)
> > print(f'first time: {values}')
> > values = values[1:]
> > print(f'second time: {values}')
> > ```
> {: .solution}
{: .question}

```python
# Fill in the blanks here!

values = ____
values.____(1)
values.____(3)
values.____(5)
print(f'first time: {values}') # Should print [1, 3, 5]
values = values[____]
print(f'second time: {values}') # should print [3, 5]
```

> ## How Large is a Slice?
>
> If `start` and `stop` are both non-negative integers,
> how long is the list `values[start:stop]`?
>
> > ## Solution
> > The list `values[start:stop]` has up to `stop - start` elements.  For example,
> > `values[1:4]` has the 3 elements `values[1]`, `values[2]`, and `values[3]`.
> > Why 'up to'?
> > If `stop` is greater than the total length of the list `values`,
> > we will still get a list back but it will be shorter than expected.
> {: .solution}
{: .question}

> ## From Strings to Lists and Back
>
> Given this:
>
> ```
> print(f'string to list: {list('tin')}')
> print(f'list to string: {''.join(['g', 'o', 'l', 'd'])}')
> ```
>
> 1.  What does `list('some string')` do?
> 2.  What does `'-'.join(['x', 'y', 'z'])` generate?
>
> > ## Solution
> > 1. [`list('some string')`](https://docs.python.org/3/library/stdtypes.html#list) converts a string into a list containing all of its characters.
> > 2. [`join`](https://docs.python.org/3/library/stdtypes.html#str.join) returns a string that is the _concatenation_
> >    of each string element in the list and adds the separator between each element in the list. This results in
> >    `x-y-z`. The separator between the elements is the string that provides this method.
> {: .solution}
{: .question}

```python
# Test code here
```

> ## Working With the End
>
> What does the following program print?
>
> ```
> element = 'helium'
> print(element[-1])
> ```
>
> 1.  How does Python interpret a negative index?
> 2.  If a list or string has N elements,
>     what is the most negative index that can safely be used with it,
>     and what location does that index represent?
> 3.  If `values` is a list, what does `del values[-1]` do?
> 4.  How can you display all elements but the last one without changing `values`?
>     (Hint: you will need to combine slicing and negative indexing.)
>
> > ## Solution
> > The program prints `m`.
> > 1. Python interprets a negative index as starting from the end (as opposed to
> >    starting from the beginning).  The last element is `-1`.
> > 2. The last index that can safely be used with a list of N elements is element
> >    `-N`, which represents the first element.
> > 3. `del values[-1]` removes the last element from the list.
> > 4. `values[:-1]`
> {: .solution}
{: .question}

```python
# Test code here
```

> ## Stepping Through a List
>
> What does the following program print?
>
> ```
> element = 'fluorine'
> print(element[::2])
> print(element[::-1])
> ```
>
> 1.  If we write a slice as `low:high:stride`, what does `stride` do?
> 2.  What expression would select all of the even-numbered items from a collection?
>
> > ## Solution
> > The program prints
> > ```
> > furn
> > eniroulf
> > ```
> > 1. `stride` is the step size of the slice.
> > 2. The slice `1::2` selects all even-numbered items from a collection: it starts
> >    with element `1` (which is the second element, since indexing starts at `0`),
> >    goes on until the end (since no `end` is given), and uses a step size of `2`
> >    (i.e., selects every second element).
> {: .solution}
{: .question}

```python
# Test code here
```

> ## Slice Bounds
>
> What does the following program print?
>
> ```
> element = 'lithium'
> print(element[0:20])
> print(element[-1:3])
> ```
>
> > ## Solution
> > ```
> > lithium
> >
> > ```
> >
> > The first statement prints the whole string, since the slice goes beyond the total length of the string.
> > The second statement returns an empty string, because the slice goes "out of bounds" of the string.
> {: .solution}
{: .question}

```python
# Test code here
```

# Dictionaries

When you think of a Dictionary, you should think of a real life Dictionary, they map some key to a value. Like a term to it's definition

Key | Value
--- | ---
`Eichhörnchen` | Squirrel
`火锅` | Hot Pot

Or a Country to it's population

Key               | Value
---               | ---
South Sudan       | 492,970
Australia         | 411,667
Guinea            | 1,660,973
Morocco           | 573,895
Maldives          | 221,678
Wallis and Futuna | 1,126
Eswatini          | 94,874
Namibia           | 325,858
Turkmenistan      | 1,031,992

In Python we create a dictionary with `{}` and use `:` to separate keys and values. Turning the above list into a Python dictionary, it would look like:

```python
populations = {
  "South Sudan": 492970,
  "Australia": 411667,
  "Guinea": 1660973,
  "Morocco": 573895,
  "Maldives": 221678,
  "Wallis and Futuna": 1126,
  "Eswatini": 94874,
  "Namibia": 325858,
  "Turkmenistan": 1031992,
}
```

You can see a string (the country name) being used for the key, and then the number (an integer) as the value. (Would a float make sense? Why or why not?)

> <tip-title>Other Names</tip-title>
> They're also sometimes called associative arrays (because they're an array or list of values that associate a key to a value) or maps (because they map a key to a value), depending on what you're reading.
{: .tip}

## Methods

You can access both the keys, and the values

```python
print(populations.keys())
print(populations.values())
```

These will print out two list-like objects. They will become more useful in the future when we talk about looping over dictionaries and processing all of the values within.

## Accessing Values

Just like lists where you access by the position in the list

```python
print(populations["Namibia"])
```

And just like lists, if you try an access a key that isn't there or an index outside of the range of the list:

```python
print(populations["Mars"])
```


> <tip-title>Dictionaries are faster than lists for looking up values</tip-title>
> Just like in real life, searching a dictionary for a specific term is quite fast. Often a lot faster than searching a list for a specific value.
>
> For those of you old enough to remember the paper version of a dictionary, you knew that As would be at the start and Zs at the end, and probably Ms around the middle. And if you were looking for a word like "Squirrel", you'd open the dictionary in the middle, maybe decide it was in the second half of the book, randomly choose a page in the second half, and you could keep deciding if it was "before" or "after" the current page, never even bothering to search the first half.
>
> Conceptually, compared with a list, you can't make this guess of if the item is in the first or second half. You need to search item by item, it would be like reading page by page until you get to Squirrel in the dictionary.
{: .tip}

## Modifying Dictionaries

Adding new values to a dictionary is easy, it's very similar to replacing a value in a list.

```python
# For lists we did
x = ['x', 'y', 'z']
x[0] = 'a'
print(x)
```

For dictionaries, it's essentially the same, we access the 'place' in the dictionary just like we did with a list, and set it to a value

```python
populations["Mars"] = 6 # robots
print(populations)
```

And similarly, removing items is the same as it was for lists:

```python
print(x)
del x[0] # Removes the first item
print(x)
```

And with dictionaries you delete by specifying which position/key you want to remove

```python
del populations['Australia']
print(populations)
```

## Exercises

> <question-title>DNA Complement</question-title>
>
> DNA is usually in the form of dsDNA, a paired strand, where A maps to T and C maps to G and vice versa.
> But when we're working with DNA sequences in bioinformatics, we often only store one strand, because we can calculate the *complement* on the fly, when we need.
>
> Write a dictionary that lets you look up the letters A, C, T, and G and find their complements.
>
> > <solution-title></solution-title>
> > You need to have the complements of every base. If you just defined 'A' and 'C', how would you look up the complement when you want to translate a 'T' or a 'G'? It's not easily possible to look up a key by a value, only to search a key and find a value.
> > ```
> > translation = {
> > 'A': 'T',
> > 'T': 'A',
> > 'C': 'G',
> > 'G': 'C',
> > }
> > ```
> {: .solution}
{: .question}

```python
# Test code here!
translation = {

}
print(translation)
```

> <question-title>Modifying an array</question-title>
>
> Fill in the blanks to make the execution correct:
>
> ```
> variants = {
>   'B.1.1.7': 26267,
>   'B.1.351': 439,
> }
> variants[_____] =  _____
> print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384}
> __________
> print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384, 'B.1.617.2': 43486}
> # Maybe we've exterminated B.1.1.7 and B.1.351, remove their numbers.
> del _______
> del _______
> print(variants[______]) # Should print 384
> print(variants[______]) # Should print 43486
> ```
>
> > <solution-title></solution-title>
> >
> > variants = {
> >   'B.1.1.7': 26267,
> >   'B.1.351': 439,
> > }
> > variants['P.1'] = 384
> > print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384}
> > variants['B.1.617.2'] = 43486
> > print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384, 'B.1.617.2': 43486}
> > # Maybe we've exterminated B.1.1.7 and B.1.351, remove their numbers.
> > del variants['B.1.1.7']
> > del variants['B.1.351']
> > print(variants['P.1']) # Should print 384
> > print(variants['B.1.617.2']) # Should print 43486
> > ```
> >
> {: .solution}
{: .question}

```python
# Test code here!
variants = {
  'B.1.1.7': 26267,
  'B.1.351': 439,
}
variants[_____] =  _____
print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384}
__________
print(variants) # Should print {'B.1.1.7': 26267, 'B.1.351': 439, 'P.1': 384, 'B.1.617.2': 43486}
# Maybe we've exterminated B.1.1.7 and B.1.351, remove their numbers.
del _______
del _______
print(variants[______]) # Should print 384
print(variants[______]) # Should print 43486
```


# Choosing the Right Data Type

Choosing the correct data type can sometimes require some thought, and even discussion with colleagues. And don't be afraid to search the internet for how other people have done it!


Data type       | Examples                  | When to use it | When **not** to use it
---             | ---                       | ---- | ----
Boolean (`bool`) | `True`, `False` | If there are only two possible states, true or false | If your data is not binary
Integer (`int`) | 1, 0, -1023, 42           | Countable, singular items. How many patients are there, how many events did you record, how many variants are there in the sequence | If doubling or halving the value would not make sense: do not use for e.g. patient IDs, or phone numbers. If these are integers you might accidentally do math on the value.
Float (`float`) | 123.49, 3.14159, -3.33334 | If you need more precision or partial values. Recording distance between places, height, mass, etc.
Strings (`str`) | 'patient_12312', 'Jane Doe', '火锅' | To store free text, identifiers, sequence IDs, etc. | If it's truly a numeric value you can do calculations with, like adding or subtracting or doing statistics.
List / Array (`list`) | `['A', 1, 3.4, ['Nested']]` | If you need to store a list of items, like sequences from a file. Especially if you're reading in a table of data from a file. | If you want to retrieve individual values, and there are clear identifiers it might be better as a dict.
Dictionary / Associative Array / map (`dict`) | `{"weight": 3.4, "age": 12, "name": "Fluffy"}` | When you have identifiers for your data, and want to look them up by that value. E.g. looking up sequences by an identifier, or data about students based on their name. Counting values. | If you just have a list of items without identifiers, it makes more sense to just use a list.


## Exercises

> <question-title>Which Datatype</question-title>
>
> 1. Chromosome Length
> 1. Name
> 2. Weight
> 3. Sex
> 4. Hair Colour
> 5. Money/Currency
>
> > <solution-title></solution-title>
> >
> > 1. Here you need to use an integer, a fractional or float value would not make sense. You cannot have half an A/C/T/G.
> > 1. Here a string would be a good choice. (And probably just a single `name` string, rather than a `first` and `last` name, as not all humans have two names! And some have more than two.)
> > 2. An integer is good type for storing weight, if you are using a small unit (e.g. grams). Otherwise you might consider a float, but you'd need to be careful to format it properly (e.g. `{value:0.2f}`) when printing it out. It depends on the application.
> > 3. This is a case where you should consider carefully the application, but `bool` is usally the *wrong answer*. Are you recording patient data? Is their expressed gender the correct variable or did you need sex? {% cite Miyagi_2021 %} goes into detail on this multifaceted issue in a medical research context. For example chromosomal sex is also more complicated and cannot be stored with a true/false value, as people with [Kleinfelters](https://en.wikipedia.org/wiki/Klinefelter_syndrome) exist. A string can be an ok choice here.
> > 4. There is a limited vocabulary humans use to describe hair colour, so a string can be used, or a data type we haven't discussed! An `enum` is an `enumeration`, and when you have a limited set of values that are possible, you can use a `enum` to double check that whatever value is being used (or read from a file, or entered by a user) matches one of the "approved" values.
> > 5. A float is a good guess, but with floats come weird rounding issues. Often times people choose to use an integer storing the value in cents (or fractional cents, to whatever the desired precision is).
> >
> {: .solution}
{: .question}


