# tutorials

topic: introduction
tutorial: galaxy-intro-peaks2genes
title: From peaks to genes
difficulty: introductory
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 1030a7dab4c168ed5f69943bb67c8c7c
last_modified: 2026-02-26T21:12:17.036071
zenodo_link: https://doi.org/10.5281/zenodo.1025586
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html

content:
We stumbled upon a paper ({% cite Li2012 %}) called *"The histone acetyltransferase MOF is a key regulator of the embryonic stem cell core transcriptional network"*. The paper contains the analysis of possible target genes of an interesting protein called Mof. The targets were obtained by ChIP-seq in mice and the raw data is available through [GEO](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE37268).
However, the list of genes is neither in the supplement of the paper, nor part of the GEO submission.
The closest thing we could find is a file in GEO containing a list of the regions where the signal is significantly enriched (so called *peaks*):

1 | 3660676 | 3661050 | 375 | 210 | 62.0876250438913 | -2.00329386666667
1 | 3661326 | 3661500 | 175 | 102 | 28.2950833625942 | -0.695557142857143
1 | 3661976 | 3662325 | 350 | 275 | 48.3062708406486 | -1.29391285714286
1 | 3984926 | 3985075 | 150 | 93 | 34.1879823073944 | -0.816992
1 | 4424801 | 4424900 | 100 | 70 | 26.8023246007435 | -0.66282

**Table 1** Subsample of the available file

The goal of this exercise is to **turn this list of genomic regions into a list of possible target genes**.

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

> <hands-on-title>Open Galaxy</hands-on-title>
>
> 1. Browse to a Galaxy instance: the one recommended by your instructor or one in the list **Galaxy instance** on the head of this page
> 2. Log in or register (top panel)
>
>    ![Login or Register on the top panel](../../images/login_register.png)
{: .hands_on}

The Galaxy interface consist of three main parts. The available tools are listed on the left, your analysis history is recorded on the right, and the central panel will show the tools and datasets.

![Galaxy interface screenshot showing history panel on the right, tools panel on the left, and main panel at the center](../../images/galaxy_interface.png "The Galaxy interface")

Let's start with a fresh history.


> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you have an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to make it easy to recognize
>
>    > <tip-title>Rename a history</tip-title>
>    >
>    > * Click on the title of the history (by default the title is `Unnamed history`)
>    >
>    >   ![Renaming history]({% link shared/images/rename_history.png %})
>    >
>    > * Type `Galaxy Introduction` as the name
>    > * Press <kbd>Enter</kbd>
>    >
>    {: .tip}
>
{: .hands_on}

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Download the list of peak regions (the file [`GSE37268_mof3.out.hpeak.txt.gz`](https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE37268&format=file&file=GSE37268%5Fmof3%2Eout%2Ehpeak%2Etxt%2Egz)) from [GEO](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE37268) to your computer
> 2. Click on the upload button in the upper left of the interface
>
>    ![Upload data icon](../../../galaxy-interface/images/upload_button.png)
>
> 3. Press **Choose local files** and search for your file on your computer
> 4. Select `interval` as **Type**
> 5. Press **Start**
> 6. Press **Close**
> 7. Wait for the upload to finish. Galaxy will automatically unpack the file.
>
> 8. After this you will see your first history item in Galaxy’s right pane. It will go through
> the gray (preparing/queued) and yellow (running) states to become green (success):
>
>    ![History section](../../images/intro_01.png)
>
>    Directly uploading files is not the only way to get data into Galaxy
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="interval" %}
>
>    > <tip-title>Importing data to Galaxy</tip-title>
>    > There are [more options]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}) for advanced users.
>    {: .tip}
>
{: .hands_on}


> <comment-title>Interval file format</comment-title>
> **Interval** format is a Galaxy format for representing genomic intervals. It is tab-separated, but has the added requirement that three of the columns must be:
> - chromosome ID
> - start position (0-based)
> - end position (end-exclusive)
>
> An optional strand column can also be specified, and an initial header row can be used to label the columns, which do not have to be in any special order. Unlike BED format (see below) arbitrary additional columns can also be present.
>
> You can find more information about formats that can be used in Galaxy at the [Galaxy Data Formats page](https://usegalaxy.org/static/formatHelp.html).
{: .comment}


> <hands-on-title>Inspect and edit attributes of a file</hands-on-title>
>
> 1. Click on the file in the history panel
>
>    Some meta-information (e.g. format, reference database) about the file and the header of the file are then displayed, along with the number of lines in the file (48,647):
>
>    ![Expanded file in history](../../images/input_expanded_file.png)
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon (**View data**) in your dataset in the history
>
>    The content of the file is displayed in the central panel
>
> 3. Click on the {% icon galaxy-pencil %} (pencil) icon (**Edit attributes**) in your dataset in the history
>
>    A form to edit dataset attributes is displayed in the central panel
>
> 4. Search for `mm9` in **Database/Build** attribute and select `Mouse July 2007 (NCBI37/mm9)` (the paper tells us the peaks are from `mm9`)
>
>    ![The database/build version can be selected from a dropdown menu. Users can start typing the database name to filter the list](../../images/Search-for-mm9.PNG)
>
> 5. Click on **Save** at the bottom
> 6. Add a tag called `#peaks` to the dataset to make it easier to track in the history
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    The dataset should now look like below in the history
>
>    ![Peaks file](../../images/input_tagged_file.png){: width="250px" height="300px"}
>
{: .hands_on}

In order to find the related genes to these peak regions,
we also need a list of genes in mice, which we can obtain from UCSC.

> <hands-on-title>Data upload from UCSC</hands-on-title>
>
> 1. Search for `UCSC Main` in the tool search bar (top left)
>
>     ![UCSC Main tool in tools section](../../images/101_01.png)
>
> 2. Click on `UCSC Main` {% icon tool %}
>
>     You will be taken to the **UCSC table browser**, which looks something like this:
>
>     ![UCSC table browser interface](../../images/intro_02.png)
>
> 3. Set the following options:
>     - *"Clade"*: `Mammal`
>     - *"Genome"*: `Mouse`
>     - *"Assembly"*: `July 2007 (NCBI37/mm9)`
>     - *"Group"*: `Genes and Gene Predictions`
>     - *"Track"*: `RefSeq Genes`
>     - *"Table"*: `refGene`
>     - *"Region"*: `Genome`
>     - *"Output format"*: `BED - browser extensible data`
>     - *"Send output to"*: `Galaxy` (only)
>
> 4. Click on the **Get output** button
>
>    You will see the next screen:
>
>    ![Output settings](../../images/intro_03.png)
>
> 5. Make sure that *"Create one BED record per"* is set to `Whole Gene`
> 6. Click on the **Send query to Galaxy** button
> 7. Wait for the upload to finish
> 8. Rename our dataset to something more recognizable like `Genes`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Genes" %}
>
> 9. Add a tag called `#genes` to the dataset to make it easier to track in the history
>
{: .hands_on}

> <comment-title>BED file format</comment-title>
> The **BED - Browser Extensible Data** format provides a flexible way to encode gene regions. BED lines have three required fields:
> - chromosome ID
> - start position (0-based)
> - end position (end-exclusive)
>
> There can be up to and nine additional optional fields, but the number of fields per line must be consistent throughout any single set of data.
>
> You can find more information about it at [UCSC](https://genome.ucsc.edu/FAQ/FAQformat#format1) including a description of the optional fields.
{: .comment}

Now we have collected all the data we need to start our analysis.

# Part 1: Naive approach

We will first use a "naive" approach to try to identify the genes that the peak regions are associated with. We will identify genes that overlap at least 1bp with the peak regions.

## File preparation

Let's have a look at our files to see what we actually have here.

> <hands-on-title>View file content</hands-on-title>
>
> 1. Click on the {% icon galaxy-eye %} (eye) icon (**View data**) of the peak file to view the content of it
>
>    It should look like this:
>
>    ![Contents of the peak file](../../images/intro_04.png)
>
> 2. View the content of the regions of the genes from UCSC
>
>    ![Contents of UCSC file](../../images/intro_05.png)
>
{: .hands_on}

> <question-title></question-title>
>
> While the file from UCSC has labels for the columns, the peak file does not. Can you guess what the columns stand for?
>
>
> > <solution-title></solution-title>
> >
> > This peak file is not in any standard format and just by looking at it, we cannot find out what the numbers in the different columns mean. In the paper the authors mention that they used the peak caller [HPeak](https://www.ncbi.nlm.nih.gov/pubmed/20598134).
> >
> > By looking at the HPeak manual we can find out that the columns contain the following information:
> >
> >  - chromosome name by number
> >  - start coordinate
> >  - end coordinate
> >  - length
> >  - location within the peak that has the highest hypothetical DNA fragment coverage (summit)
> >  - not relevant
> >  - not relevant
> >
> {: .solution}
{: .question}

In order to compare the two files, we have to make sure that the chromosome names follow the same format.
As we can see, the peak file lacks `chr` before any chromosome number. But what happens with chromosome 20 and 21? Will it be X and Y instead? Let's check:

> <hands-on-title>View end of file</hands-on-title>
>
> 1. Search for {% tool [Select last lines from a dataset (tail)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.5+galaxy2) %} tool and run with the following settings:
>     - *"Text file"*: our peak file `GSE37268_mof3.out.hpeak.txt.gz`
>     - *"Operation"*: `Keep last lines`
>     - *"Number of lines"*: Choose a value, e.g. `100`
> 2. Click **Run Tool**
> 3. Wait for the job to finish
> 4. Inspect the file through the {% icon galaxy-eye %} (eye) icon (**View data**)
>
>    > <question-title></question-title>
>    >
>    > 1. How are the chromosomes named?
>    > 2. How are the chromosomes X and Y named?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The chromosomes are just given by their number. In the gene file from UCSC, they started with `chr`
>    > > 2. The chromosomes X and Y are named 20 and 21
>    > {: .solution }
>    {: .question}
{: .hands_on}

In order to convert the chromosome names we have therefore two things to do:

1. add `chr`
2. change 20 and 21 to X and Y

> <hands-on-title>Adjust chromosome names</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2) %} in a specific column with the following settings:
>     - *"File to process"*: our peak file `GSE37268_mof3.out.hpeak.txt.gz`
>     - *"in column"*: `Column: 1`
>     - *"Find pattern"*: `[0-9]+`
>
>         This will look for numerical digits
>
>     - *"Replace with"*: `chr&`
>
>         `&` is a placeholder for the find result of the pattern search
>
> 2. Rename your output file `chr prefix added`.
>
> 3. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2) %}  : Let's rerun the tool with two more replacements
>    - *"File to process"*: the output from the last run, `chr prefix added`
>    - *"in column"*: `Column: 1`
>    - {% icon param-repeat %} Replacement
>      - *"Find pattern"*: `chr20`
>      - *"Replace with"*: `chrX`
>    - {% icon param-repeat %} Insert Replacement
>      - *"Find pattern"*: `chr21`
>      - *"Replace with"*: `chrY`
>
>    > <tip-title>Rerunning a tool</tip-title>
>    >
>    > * Expand the dataset information
>    > * Press the {% icon galaxy-refresh %} icon (**Run this job again**)
>    {: .tip}
>
> 4. Inspect the latest file through the {% icon galaxy-eye %} (eye) icon. Have we been successful?
>
>    We have quite a few files now and need to take care to select the correct ones at each step.
>
>    > <question-title></question-title>
>    >
>    > How many regions are in our output file? You can click the name of the output to expand it and see the number.
>    >
>    > > <solution-title></solution-title>
>    > > It should be equal to the number of regions in your first file, `GSE37268_mof3.out.hpeak.txt.gz`: 48,647
>    > > If yours says 100 regions, then you have run it on the `Tail` file and need to re-run the steps.
>    > {: .solution }
>    {: .question}
>
> 5. Rename the file to something more recognizable, e.g. `Peak regions`
{: .hands_on}

## Analysis

Our goal is to compare the 2 region files (the genes file and the peak file from the publication)
to know which peaks are related to which genes. If you only want to know which peaks are located **inside** genes (within the gene body) you
can skip the next step. Otherwise, it might be reasonable to include the **promoter** region of the genes into the comparison, e.g. because
you want to include transcriptions factors in ChIP-seq experiments. There is no strict definition for promoter region but 2kb upstream of the TSS (start of region) is commonly used. We'll use the **Get Flanks** tool to get regions 2kb bases upstream of the start of the gene to 10kb bases downstream of the start (12kb in length). To do this we tell the Get Flanks tool we want regions upstream of the start, with an offset of 10kb, that are 12kb in length, as shown in the diagram below.

![Get Flanks](../../images/intro_get_flanks.png)

> <hands-on-title>Add promoter region to gene records</hands-on-title>
>
> 1. {% tool [Get Flanks](toolshed.g2.bx.psu.edu/repos/devteam/get_flanks/get_flanks1/1.0.0) %} returns flanking region/s for every gene, with the following settings:
>     - *"Select data"*: `Genes` file from UCSC
>     - *"Region"*: `Around Start`
>     - *"Location of the flanking region/s"*: `Upstream`
>     - *"Offset"*: `10000`
>     - *"Length of the flanking region(s)"*: `12000`
>
>     This tool returns flanking regions for every gene
>
> 2. Compare the rows of the resulting BED file with the input to find out how the start and end positions changed
>
>    > <tip-title>Inspecting several files using the scratchbook</tip-title>
>    >
>    > * Click **Enable/Disable Scratchbook** on the top panel
>    >
>    >    ![Enable/Disable Scratchbook](../../images/intro_scratchbook_enable.png)
>    >
>    > * Click on the {% icon galaxy-eye %} (eye) icon of the files to inspect
>    > * Click on **Show/Hide Scratchbook**
>    >
>    >    ![Show/Hide Scratchbook](../../images/intro_scratchbook_show_hide.png)
>    {: .tip}
>
> 3. Rename your dataset to reflect your findings (`Promoter regions`)
{: .hands_on}

The output is regions that start from 2kb upstream of the TSS and include 10kb downstream. For input regions on the positive strand e.g. `chr1 134212701 134230065` this gives `chr1 134210701 134222701`. For regions on the negative strand e.g. `chr1 8349819 9289958` this gives `chr1  9279958 9291958`.

You might have noticed that the UCSC file is in `BED` format and has a database associated to it. That's what we want for our peak file as well. The **Intersect** tool we will use can automatically convert interval files to BED format but we'll convert our interval file explicitly here to show how this can be achieved with Galaxy.

> <hands-on-title>Change format and database</hands-on-title>
>
> 1. Click on the {% icon galaxy-pencil %} (pencil) icon in the history entry of our peak region file
> 2. Switch to the **Datatypes** tab
> 3. In section **Convert to Datatype** under *"Target datatype"* select: `bed (using 'Convert Genomic Interval To Bed')`
> 4. Press **Create Dataset**
> 5. Check that the "Database/Build" is `mm9` (the database build for mice used in the paper)
> 6. Again rename the file to something more recognizable, e.g. `Peak regions BED`
{: .hands_on}

It's time to find the overlapping intervals (finally!). To do that, we want to extract the genes which overlap/intersect with our peaks.

> <hands-on-title>Find Overlaps</hands-on-title>
>
> 1. {% tool [Intersect](toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0) %} the intervals of two datasets, with the following settings:
>     - *"Return"*: `Overlapping Intervals`
>     - *"of"*: the UCSC file with promoter regions (`Promoter regions`)
>     - *"that intersect"*: our peak region file from **Replace** (`Peak regions BED`)
>     - *"for at least"*: `1`
>
>    > <comment-title></comment-title>
>    > The order of the inputs is important! We want to end up with a list of **genes**, so the corresponding dataset with the gene information needs to be the first input (`Promoter regions`).
>    {: .comment}
>    ![Genes overlapping peaks](../../images/intro_overlapping_genes.png)
{: .hands_on}

We now have the list of genes (column 4) overlapping with the peak regions, similar to shown above.

To get a better overview of the genes we obtained, we want to look at their distribution across the different chromosomes.
We will group the table by chromosome and count the number of genes with peaks on each chromosome

> <hands-on-title>Count genes on different chromosomes</hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} data by a column and perform aggregate operation on other columns, with the following settings:
>     - *"Select data"* to the result of the intersection
>     - *"Group by column"*:`Column: 1`
>     - Press **Insert Operation** and choose:
>         - *"Type"*: `Count`
>         - *"On column"*: `Column: 1`
>         - *"Round result to nearest integer?"*: `No`
>
>    > <question-title></question-title>
>    >
>    > Which chromosome contained the highest number of target genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The result varies with different settings, for example, the annotation may change due to updates at UCSC. If you followed step by step, with the same annotation, it should be chromosome 11 with 2164 genes. Note that for reproducibility, you should keep all input data used within the analysis. Rerunning the analysis with the same set of parameters, stored Galaxy, can lead to a different result if the inputs changed e.g. the annotation from UCSC.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

## Visualization

We have some nice aggregated data, so why not draw a barchart of it?

Before we do that we should polish our grouped data a bit more though.

You may have noticed that the mouse chromosomes are not listed in their correct
order in that dataset (the **Group** tool tried to sort them, but did so
alphabetically).

We can fix this by running a dedicated tool for sorting on our data.

> <hands-on-title>Fix sort order of gene counts table</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} data in ascending or descending order, with the following settings:
>     - *"Sort Query"*: result of running the Group tool
>     - in {% icon param-repeat %} *"Column selections"*
>       - *"on column"*: `Column: 1`
>       - *"in"*: `Ascending order`
>       - *"Flavor"*: `Natural/Version sort (-V)`
>
>    {% snippet faqs/galaxy/tools_similar.md toolname="Sort data in ascending or descending order" toolversion="1.1.1" %}
>
{: .hands_on}

Great, we are ready to plot things!

> <hands-on-title>Draw barchart</hands-on-title>
>
> 1. Click on {% icon galaxy-barchart %} (visualize) icon on the output from the **Sort** tool
> 2. Select `Bar, Line and Scatter` from the main panel
> 3. Click on the **<<** in the upper right corner
> 4. Choose a title below **Title**, e.g. `Gene counts per chromosome`
> 5. Switch to the **Settings** tab and change the axis labels
> 6. Play around with the settings in the **Tracks** tab
> 7. When you are happy, click the {% icon cloud-upload %} icon to **Save** visualization in the top right of the *main panel*
>
>    This will store it to your saved visualisations. Later you can view,
>    download, or share it with others by clicking again on **Visualization** on the left menu bar. In the opened menu bar click on the blue **Saved Visualizations** button on top. In the main panel you can find your saved plot.
>
{: .hands_on}

## Extracting workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step.
Wouldn't it be nice to just convert this history into a workflow that we'll be able to execute again and again?

Galaxy makes this very simple with the `Extract Workflow` option. This means that any time you want to build a workflow, you can just perform it manually once, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis. It also allows you to easily share or publish your analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking on the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click on {% icon galaxy-history-options %} (**History options**) at the top of your history panel and select **Extract Workflow**.
>
>    ![`Extract Workflow` entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
> 3. Replace the **Workflow name** to something more descriptive, for example: `From peaks to genes`
>
> 4. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes.
>
>    Since we did some steps which where specific to our custom peak file, we might want to exclude:
>    - **Select last** {% icon tool %}
>    - all **Replace Text** {% icon tool %} steps
>    - **Convert Genomic Intervals to BED**
>    - **Get flanks** {% icon tool %}
>
> 5. Click on the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created. But where did it go?
>
> 6. Click on **Workflow** in the left menu of Galaxy
>
>    Here you have a list of all your workflows
>
> 7. Select the newly generated workflow and click on **Edit**
>
>    You should see something similar to this:
>
>    ![Editing workflow interface](../../images/intro_06.png)
>
>    > <comment-title>The workflow editor</comment-title>
>    > We can examine the workflow in Galaxy's workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.
>    {: .comment}
>
>     Although we have our two inputs in the workflow they are missing their connection to the first tool (**Intersect** {% icon tool %}), because we didn't carry over some of the intermediate steps.
>
> 8. Connect each input dataset to the **Intersect** {% icon tool %} tool by dragging the arrow pointing outwards on the right of its box (which denotes an output) to an arrow on the left of the **Intersect** box pointing inwards (which denotes an input)
> 9. Rename the input datasets to `Reference regions` and `Peak regions`
> 10. Press **Auto Re-layout** to clean up our view
>     ![Auto re-layouting](../../images/intro_07.png)
> 11. Click on the {% icon galaxy-save %} **Save** icon (top) to save your changes
>     ![Save workflow button]({% link topics/contributing/images/save_workflow.png %}){: width="50%"}
>
>    > <tip-title>Hiding intermediate steps</tip-title>
>    > When a workflow is executed, the user is usually primarily interested in the final product and not in all intermediate steps. By default all the outputs of a workflow will be shown, but we can explicitly tell Galaxy which output to show and which to hide for a given workflow. This behaviour is controlled by the little asterisk next to every output dataset:
>    >
>    > ![Workflow editor mark output](../../../../shared/images/workflow_editor_mark_output.png)
>    >
>    > If you click on this asterisk for any of the output datasets, then *only* files with an asterisk will be shown, and all outputs without an asterisk will be hidden (Note that clicking *all* outputs has the same effect as clicking *none* of the outputs, in both cases all the datasets will be shown).
>    {: .tip}
>
{: .hands_on}

Now it's time to reuse our workflow for a more sophisticated approach.

# Part 2: More sophisticated approach

In part 1 we used an overlap definition of 1 bp (default setting) to identify genes associated with the peak regions. However, the peaks could be broad, so instead, in order to get a more meaningful definition, we could identify the genes that overlap where most of the reads are concentrated, the **peak summit**. We will use the information on the position of the peak summit contained in the original peak file and check for overlap of the summits with genes.

## Preparation

We again need our peak file, but we'd like to work in a clean history. Instead of uploading it twice, we can copy it to a new history.

> <hands-on-title>Copy history items</hands-on-title>
>
> 1. Create a new history and give it a new name like `Galaxy Introduction Part 2`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Click on the **History options** {% icon galaxy-history-options %} at the top right of your history. Click on the **Show Histories Side-by-Side**
>
>       You should see both of your histories side-by-side now
>
> 3. Drag and drop the edited peak file (`Peak regions`, after the replace steps), which contains the summit information, to your new history.
> 4. Click on the Galaxy name in the top menu bar (top left) to go back to your analysis window
>
{: .hands_on}

## Create peak summit file

We need to generate a new BED file from the original peak file that contains the positions of the peak summits. The start of the summit is the start of the peak (column 2) plus the location within the peak that has the highest hypothetical DNA fragment coverage (column 5, rounded down to the next smallest integer because some peak summits fall in between to bases). As the end of the peak region, we will simply define `start + 1`.

> <hands-on-title>Create peak summit file</hands-on-title>
>
> 1. {% tool [Compute on rows](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters:
>   - *"Input file"*: our peak file `Peak regions` (the interval format file)
>   - *"Input has a header line with column names?": `No`
>   - In *"Expressions"*:
>       - {% icon param-repeat %} *"Expressions"*
>           - *"Add expression"*: `c2 + int(c5)`
>           - *"Mode of the operation"*: Append
>       - {% icon param-repeat %} *"Expressions"*
>           - *"Add expression"*: `c8 + 1`
>           - *"Mode of the operation"*: Append
>
>   This will create an 8th and a 9th column in our table, which we will use in our next step:
>
> 2. Rename the output `Peak summit regions`
>
{: .hands_on}

Now we cut out just the chromosome plus the start and end of the summit:

> <hands-on-title>Cut out columns</hands-on-title>
> 1. {% tool [Cut](Cut1) %} columns from a table with the following settings:
>   - *"Cut columns"*: `c1,c8,c9`
>   - *"Delimited by Tab"*: `Tab`
>   - *"From"*: `Peak summit regions`
>
>    The output from **Cut** will be in `tabular` format.
>
> 2. Change the format to `interval` (use the {% icon galaxy-pencil %}) since that's what the tool **Intersect** expects.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="interval" %}
>
>    The output should look like below:
>
>    ![Peak summits](../../images/intro_summits.png){: width="200px"}
{: .hands_on}

## Get gene names

The RefSeq genes we downloaded from UCSC did only contain the RefSeq identifiers, but not the gene names. To get a list of gene names in the end, we use another BED file from the Data Libraries.

> <comment-title></comment-title>
> There are several ways to get the gene names in, if you need to do it yourself. One way is to retrieve a mapping through Biomart and then join the two files (**Join two Datasets side by side on a specified field** {% icon tool %}). Another is to get the full RefSeq table from UCSC and manually convert it to BED format.
{: .comment}

> <hands-on-title>Data upload</hands-on-title>
>
> 1. {% tool [Import](upload1) %} `mm9.RefSeq_genes_from_UCSC.bed` from [Zenodo](https://zenodo.org/record/1025586) or from the data library:
>
>    ```
>    https://zenodo.org/record/1025586/files/mm9.RefSeq_genes_from_UCSC.bed
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md genome="mm9" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md path='Click on "GTN - Material", "Introduction to Galaxy Analyses", "From peaks to genes", and then "DOI: 10.5281/zenodo.1025586"' %}
>
>    As default, Galaxy takes the link as name, so rename them.
>
> 2. Inspect the file content to check if it contains gene names.
>    It should look similar to below:
>    ![Gene names](../../images/intro_gene_names.png)
>
> 3. Rename it `mm9.RefSeq_genes`
> 4. Apply the tag `#genes`
>
{: .hands_on}

## Repeat workflow

It's time to reuse the workflow we created earlier.

> <hands-on-title>Run a workflow</hands-on-title>
> 1. Open the workflow menu (left menu bar)
> 2. Find the workflow you made in the previous section, and select the option **Run** {% icon workflow-run %}
> 3. Choose as inputs our `mm9.RefSeq_genes` (`#genes`) BED file and the result of the **Cut** tool (`#peaks`)
> 4. Click **Run workflow**
>
>    The outputs should appear in the history but it might take some time until they are finished.
>
{: .hands_on}

We used our workflow to rerun our analysis with the peak summits. The **Group** tool again produced a list containing the number of genes found in each chromosome.
But wouldn't it be more interesting to know the number of peaks in each unique gene? Let's rerun the workflow with different settings!

> <hands-on-title>Run a workflow with changed settings</hands-on-title>
> 1. Open the workflow menu (left menu bar)
> 2. Find the workflow you made in the previous section, and select the option **Run**
> 3. Click on **Workflow run settings** {% icon galaxy-wf-options %} and then on **Expanded workflow form**
> 4. Choose as inputs our `mm9.RefSeq_genes` (`#genes`) BED file and the result of the **Cut** tool (`#peaks`)
> 5. Click on the title of the {% icon tool %} **Group** tool to expand the options
> 6. Change the following settings by clicking on the {% icon galaxy-pencil %} (pencil) icon on the left:
>     - *"Group by column"*: `7`
>     - In *"Operation"*:
>       - *"On column"*: `7`
> 7. Click **Run workflow**
{: .hands_on}

Congratulations! You should have a file with all the unique gene names and a count on how many peaks they contained.

> <question-title></question-title>
>
> The list of unique genes is not sorted or you workflow included a sorting step but for the gene names. Try to sort it on your own!
>
> > <solution-title></solution-title>
> > You can use the tool "{% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} data in ascending or descending order" or rerun to sort tool of your workflow. Sort on column 2 and "fast numeric sort".
> {: .solution }
{: .question}


# Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

To share a history, click on the {%  icon galaxy-history-options %} history options and select `Share & Manage Access`. On this tab (**Share or Publish**) you can **Make History accessible** or **Share History with Individual Users**:


1. **Make History accessible**

    This generates a link that you can give out to others. Anybody with this link will be able to view your history.

2. **Make History publicly available**

    This will not only create a link, but will also publish your history. This means your history will be listed under `Histories → Published Histories` in the left menu and other users can access the history there.

3. **Share with Individual Users**

    This will share the history only with specific users on the Galaxy instance.

![The menu for sharing histories includes buttons for making the history accessible, publishing it on this Galaxy server, and displays a sharable link to the history. At the bottom is a button for sharing the history with individual users.](../../images/publish.png)

> <hands-on-title>Share history and workflow</hands-on-title>
>
> 1. Share one of your histories with your neighbour
> 2. See if you can do the same with your workflow!
> 3. Find the history and/or workflow shared by your neighbour
>
>    Histories shared with specific users can be accessed by those users under `Histories → Shared with Me`.
>
{: .hands_on}

# Conclusion


{% icon trophy %} You have just performed your first analysis in Galaxy. You also created a workflow from your analysis so you can easily repeat the exact same analysis on other datasets. Additionally you shared your results and methods with others.


topic: introduction
tutorial: galaxy-intro-short
title: A short introduction to Galaxy
difficulty: introductory
time_estimation: 40m
tools_json: []
requirements_json: []
tags_json: ["espa\u00f1ol"]
content_hash: 468ccc996a9cf0f54a3c469aa00a0563
last_modified: 2026-02-26T21:12:17.037771
zenodo_link: https://doi.org/10.5281/zenodo.582600
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-short/tutorial.html

content:
# Overview


* This is a short introduction to the Galaxy user interface - the web page that you interact with.
* We will cover key tasks in Galaxy: uploading files, using tools, viewing histories, and running workflows.

> <agenda-title></agenda-title>
> 1. TOC
> {:toc}
>
{: .agenda}

## Create an account on a Galaxy instance/server
If you already have an account, skip to the next section!

In Galaxy, *server* and *instance* are often used interchangeably. These terms basically mean that different regions have different Galaxy servers/instances, with slightly different tool installations and appearances. If you don't have a specific server/instance in mind, we recommend registering at one of the main public servers/instances, detailed below.

{% snippet faqs/galaxy/account_create.md %}

## What does Galaxy look like?

> <hands-on-title>Log in to Galaxy</hands-on-title>
> 1. Open your favorite browser (Chrome, Safari, Edge or Firefox as your browser, not Internet Explorer!)
> 2. Browse to your Galaxy instance
> 3. Log in or register
>
> ![Screenshot of Galaxy Australia with the register or login button highlighted](../../images/galaxy-login.png)
>
>   > <comment-title>Different Galaxy servers</comment-title>
>   >  This is an image of Galaxy Australia, located at [usegalaxy.org.au](https://usegalaxy.org.au/)
>   >
>   > The particular Galaxy server that you are using may look slightly different and have a different web address.
>   >
>   > You can also find more possible Galaxy servers at the top of this tutorial in **Available on these Galaxies**
>   {: .comment}
{: .hands_on}

The Galaxy homepage is divided into four sections (panels):
* The Activity Bar on the left: _This is where you will navigate to the resources in Galaxy (Tools, Workflows, Histories etc.)_
* Currently active "Activity Panel" on the left: _By default, the {% icon tool %} **Tools** activity will be active and its panel will be expanded_
* Viewing panel in the middle: _The main area for context for your analysis_
* History of analysis and files on the right: _Shows your "current" history; i.e.: Where any new files for your analysis will be stored_

![Screenshot of the Galaxy interface with aforementioned structure](../../images/galaxy_interface.png)

The first time you use Galaxy, there will be no files in your history panel.

# Key Galaxy actions

## Name your current history

Your "History" is in the panel at the right. It is a record of the actions you have taken.

> <hands-on-title>Name history</hands-on-title>
> 1. Go to the **History** panel (on the right)
> 2. Click {% icon galaxy-pencil %} (**Edit**) next to the history name (which by default is "Unnamed history")
>
>    ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value. An input box is below it.]({% link shared/images/rename_history.png %}){:width="250px"}
>
>    > <comment-title></comment-title>
>    >
>    > In some previous versions of Galaxy, you will need to click the history name to rename it as shown here:
>    > ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value.](../../../../shared/images/rename_history_old.png){:width="320px"}
>    {: .comment}
>
> 3. Type in a new name, for example, "My Analysis"
> 4. Click **Save**
>
> > <comment-title>Renaming not an option?</comment-title>
> > If renaming does not work, it is possible you aren't logged in, so try logging in to Galaxy first. Anonymous users are only permitted to have one history, and they cannot rename it.
> {: .comment}
>
{: .hands_on}

## Upload a file

The "Activity Bar" can be seen on the left-most part of the interface.

> <hands-on-title>Upload a file from URL</hands-on-title>
> 1. At the top of the **Activity Bar**, click the {% icon galaxy-upload %} **Upload** activity
>
>    ![upload data button shown in the galaxy interface](../../images/upload-data.png)
>
>    This brings up a box:
>
>    ![the complicated galaxy upload dialog, the 'regular' tab is active with a large textarea to paste subsequent URL](../../images/upload-box.png)
>
> 3. Click **Paste/Fetch data**
> 4. Paste in the address of a file:
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    ```
>
> 5. Click **Start**
> 6. Click **Close**
>
{: .hands_on}

Your uploaded file is now in your current history.
When the file has uploaded to Galaxy, it will turn green.

> <comment-title></comment-title>
> After this you will see your first history item (called a "dataset") in Galaxy's right panel. It will go through
> the gray (preparing/queued) and yellow (running) states to become green (success).
>
{: .comment}

> <details-title>Is this step taking a while?</details-title>
>   Sometimes during courses, data upload gets a little slow. You can also import data through a history link.
>
> 1. Import history from: [example input history](https://humancellatlas.usegalaxy.eu/u/wendi.bacon.training/h/short-introduction-to-galaxy--input)
>
>    {% snippet faqs/galaxy/histories_import.md %}
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .details}

What is this file?

> <hands-on-title>View the dataset content</hands-on-title>
> 1. Click the {% icon galaxy-eye %} (eye) icon next to the dataset name, to look at the file content
>
>    ![galaxy history view showing a single dataset mutant_r1.fastq. Display link is being hovered.](../../images/eye-icon.png){:width="320px"}
{: .hands_on}

The contents of the file will be displayed in the central Galaxy panel. If the dataset is large, you will see a warning message which explains that only the first megabyte is shown.

This file contains DNA sequencing reads from a bacteria, in FASTQ format:

   ![preview of a fastq file showing the 4 line structure described in fig caption. 3 reads are shown.](../../images/fastq.png "A FastQ file has four lines per record: the record identifier (`@mutant-no_snps.gff-24960/`), the sequence (`AATG…`), the plus character (`+`), and then the quality scores for the sequence (`5??A…`)."){:width="620px"}

## Use a tool

Let's look at the quality of the reads in this file.

> <hands-on-title>Use a tool</hands-on-title>
> 1. Type **FastQC** in the tools panel search box (top)
> 2. Click the {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} tool (it appears as **FastQC Read Quality reports** without the version number)
>
>    The tool will be displayed in the central Galaxy panel.
>
> 3. Select the following parameters:
>    - {% icon param-file %} *"Raw read data from your current history"*: the FASTQ dataset that we uploaded (should be already selected)
>    - No change in the other parameters
> 4. Click **Run Tool**
>
{: .hands_on}

This tool will run and two new output datasets will appear at the top of your history panel.

{% snippet faqs/galaxy/tutorial_mode.md %}

## View results

We will now look at the output dataset called *FastQC on data 1: Webpage*.

> <comment-title></comment-title>
> * Note that Galaxy has given this dataset a name according to both the tool name ("FastQC") and the input ("data 1") that it used.
> * The name "data 1" means the dataset number 1 in Galaxy's current history (our FASTQ file).
>
{: .comment}


> <hands-on-title>View results</hands-on-title>
> * Once it's green, click the {% icon galaxy-eye %} (eye) icon next to the "Webpage" output dataset.
>
>    The information is displayed in the central panel
>
>    ![Graph from fastqc's report. fastqc's images themselves are inaccessible, but this graph shows overall mostly green (good) sequences scores across the length of the read.](../../images/fastqc-out.png){:width="620px"}
{: .hands_on}

This tool has summarised information about all of the reads in our FASTQ file.

> <question-title></question-title>
>
> 1. What was the sequence length of reads in the input FASTQ file?
> 2. Do these reads have higher quality scores in the centre or at the ends?
>
>   > <solution-title></solution-title>
>   > 1. 150 bp
>   > 2. In the center
>   {: .solution}
{: .question}


## Run another tool

Let's run a tool to filter out lower-quality reads from our FASTQ file.


> <hands-on-title>Run another tool</hands-on-title>
> 1. Type **Filter by quality** in the tools panel search box (top)
> 2. Click the tool {% tool [Filter by quality](toolshed.g2.bx.psu.edu/repos/devteam/fastq_quality_filter/cshl_fastq_quality_filter/1.0.2+galaxy2) %}
> 3. Set the following parameters:
>    - {% icon param-file %} *"Input FASTQ file"*: our initial FASTQ dataset
>    - *"Quality cut-off value"*: 35
>    - *"Percent of bases in sequence that must have quality equal to / higher than cut-off value"*: 80
> 4. Click **Run Tool**
{: .hands_on}

After the tool has run, its output dataset will appear at the top of your History panel.
* This dataset will be called "Filter by quality on data 1".
* Remember that Galaxy has named this file according to the tool it used ("Filter by quality") and the input dataset ("data 1").
* The actual numbers in front of the datasets in the history are not important.

What are the results from this filtering tool?

We could click on the eye icon to view the contents of this output file, but it will not be very informative - we will just see a list of reads.

> <hands-on-title>Get metadata about a file</hands-on-title>
> 1. Click the dataset (around where its name is) in the History panel.
>
>    This expands the information about the file. _(By default all datasets are collapsed)_
>
>    ![Diagram of how to locate the information. As above, clicking the name expands the dataset, and an info section is shown with the filter settings. 1786 (14%) discarded.](../../images/filter-fastq1.png)
>
{: .hands_on}

> <question-title></question-title>
>
> How many reads have been discarded?
>
>   > <solution-title></solution-title>
>   > 1786 low-quality reads were discarded
>   {: .solution}
{: .question}

## Re-run that tool with changed settings

We can now try to filter our input reads to an even higher standard, and see how this changes the resulting output (an exploratory analysis). We will change the filter settings and re-run the tool.

> <hands-on-title>Re-run the tool</hands-on-title>
> 1. Click the {% icon galaxy-refresh %} icon (**Run Job Again**) for the output dataset of **Filter by quality** {% icon tool %}
>
>    ![A dataset is expanded showing the Run Job Again button highlighted](../../images/rerun.png)
>
>    This brings up the tool interface in the central panel with the parameters set to the values used previously to generate this dataset.
>
> 2. Change the settings to something even stricter
>
>    For example, you might decide you want 80 percent of bases to have a quality of 36 or higher, instead of 35.
>
> 3. Click **Run Tool**
> 4. View the results: Click the output dataset name to expand the information
>
>    > <comment-title></comment-title>
>    > Take care not to click the {% icon galaxy-eye %} (eye) icon as this will display the contents of the file in the central Galaxy panel instead.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> How many reads were discarded under these new filtering conditions?
>
>   > <solution-title></solution-title>
>   > If you selected 80% of bases with 36 as quality cut-off, then 11517 reads (92%) should have been discarded, which indicates that we have gone too far with the filtering in this case.
>   {: .solution}
{: .question}

You can re-run a tool many times with different settings. Each time you re-run the tool, its new output datasets will appear at the top of your current history.

## Share your history

Let's imagine that you had a problem in your analysis and you want to ask for help. The easiest way to ask for help is to share your history. Try and create a link for your history and share it with...yourself!

{% snippet faqs/galaxy/histories_sharing.md %}


## Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning (at the bottom) to the end (on top). The history in Galaxy records details of every tool you run and preserves all parameter settings applied at each step. But when you need to analyze new data, it would be tedious to do each step one-by-one again. Wouldn't it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the `Extract workflow` option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click {% icon galaxy-history-options %} (**History options**) at the top of your history panel and select **Extract workflow**.
>
>    !['Extract Workflow' entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
>    ![Selection of steps for Extract Workflow from history. All three of fastqc, filter by quality, and the second filter by quality are selected.](../../images/intro_short_workflow_extract.png)
>
> 3. Replace the **Workflow name** to something more descriptive, for example: `QC and filtering`.
>
> 4. **Rename** the workflow input in the box at the top of second column to: `FASTQ reads`
>
> 5. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes. In this case, uncheck the second **Filter by quality** tool at the bottom, where we used a too high quality cut-off.
>
> 6. Click the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created.
>
{: .hands_on}

In a minute we will see how to find the extracted workflow and how to use it.

## Create a new history

Let's create a new history.

> <hands-on-title>New history</hands-on-title>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename your history, *e.g.* "Next Analysis"
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

This new history does not have any datasets in it yet.

## Look at multiple histories

Where is your first history, called "My Analysis"?

> <hands-on-title>View histories in <i>History Multiview</i></hands-on-title>
> One of the ways to view multiple histories at once in Galaxy is through the **History Multiview**
> 1. There are multiple ways to get to the multview:
>    - Click the {% icon galaxy-multihistory %} **History Multiview** activity in the activity bar
>    - Or, click {% icon galaxy-history-options %} (**History options**) and then click the {% icon galaxy-columns %} **Show Histories side-by-side** option
>    ![History options menu dropdown showing you have 163 histories, and a show histories side-by-side button.](../../images/galaxy_interface_history_switch.png){:width="320px"}
>
>    A new page will appear with your histories displayed side-by-side here.
>
> 2. Copy a dataset into your new history
>    1. Click the FASTQ dataset in "My Analysis" history
>    2. Drag it into the "Next Analysis" history
>
>    ![Gif of copying datasets between histories in the side-by-side history view. For now this feature is not keyboard accessible, it is a known issue.](../../images/copy-dataset.gif "Copy a dataset between histories by dragging it")
>
>    This makes a copy of the dataset in the new history (without actually using additional disk space).
>
> 3. Click the Galaxy logo (Home) at the top-left (or **Analyze Data** on older versions of Galaxy) in the top panel to go back to your analysis window
>
{: .hands_on}

Your main Galaxy window will now show "Next Analysis" as the current history, and it will have one dataset in it.

> <comment-title></comment-title>
>   This is not the only way to view your histories in Galaxy:
>
> 1. An exhaustive list (table) of all your histories is available in the **My Histories** tab in the *Histories List* accessible via clicking the {% icon galaxy-histories-activity %} **Histories** activity
> 2. You can quickly switch to another history by clicking the {% icon switch-histories %} **Switch to history** button next to {% icon galaxy-history-options %} **History options**:
>
>    {% snippet faqs/galaxy/histories_switch.md %}
>
> _Here is a tutorial on [Understanding Galaxy history system]( {% link topics/galaxy-interface/tutorials/history/tutorial.md %} )_
{: .comment}

## Run workflow in the new history

Now that we have built our workflow, let's use it to re-create our small analysis in a single step. The same workflow could also be used on some new FASTQ data to quickly repeat the same analysis on different inputs.

> <hands-on-title>Run workflow</hands-on-title>
>
> 1. Click the {% icon galaxy-workflows-activity %} **Workflows** activity in the activity bar.
>
>    Here you have a list of all your workflows _(the **My Workflows** tab is active by default)_.
>
>    Your newly created workflow should be listed at the top:
>
>    ![Workflow list page showing a single workflow named QC and filtering.](../../images/intro_short_workflow_list.png)
>
>    You can see all available actions for the workflow on the workflow card, e.g. edit, copy, rename, share etc. Any other options (e.g.: delete, export etc.) are available by clicking the {% icon galaxy-dropdown %} **Workflow actions** button on the top right of the card.
>
> 2. Click the {% icon workflow-run %} (*Run workflow*) button on the bottom right of the workflow card.
>
>    The central panel will change to allow you to configure and launch the workflow.
>
>    ![Run workflow form with a single input: FASTQ reads. mutant_r1.fastq is selected as the input dataset for that parameter.](../../images/intro_short_run_workflow.png)
>
> 3. Check that the *"FASTQ reads"* input is set to the FASTQ dataset we have copied to the new history.
>
>    In this page we could change any parameter for the tools composing the workflow as we would do when running them one by one.
>
> 4. Click the **Run Workflow** button at the top-right of the screen.
>
>    You should see a message that the workflow was successfully invoked. Then jobs will start to run and datasets appear in your "Next Analysis" history, replicating the steps of your previous history.
>
{: .hands_on}


# Conclusion

Well done! You have completed the short introduction to Galaxy, where you named the history, uploaded a file, used a tool, viewed results and run a workflow. Additional tutorials are available for a more in-depth introduction to Galaxy's features.


topic: introduction
tutorial: igv-introduction
title: IGV Introduction
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 79fa2016641c2ec96bcb49f19bb0559a
last_modified: 2026-02-26T21:12:17.042199
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/igv-introduction/tutorial.html

content:
# Using the Integrative Genomics Viewer

**Bioinformatics Galaxy Course**

**MPI-IE & University of Freiburg**

To start IGV, go to the IGV downloads page: https://www.broadinstitute.org/software/igv/download.

## User Interface

## Main Window

![MainWindowIGV](https://www.broadinstitute.org/software/igv/sites/cancerinformatics.org.igv/files/images/igv_desktop_callouts.jpg)


**Tool bar** ![callout_1.jpg](../../images/callout_1.jpg)

- The [tool bar](https://www.broadinstitute.org/software/igv/?q=ToolBar) provides access to commonly used functions. The [menu bar](https://www.broadinstitute.org/software/igv/?q=MenuBar) and [pop-up menus](http://www.broadinstitute.org/software/igv/?q=PopupMenus) (not shown) provide access to all other functions.

**Chromosome** ![callout_2.jpg](../../images/callout_2.jpg)

- The red box on the chromosome ideogram indicates which portion of the chromosome is displayed. When zoomed out to display the full chromosome, the red box disappears from the ideogram.

**Ruler** ![callout_3.jpg](../../images/callout_3.jpg)

- The ruler reflects the visible portion of the chromosome. The tick marks indicate chromosome locations. The span lists the number of bases currently displayed.

**Tracks** ![callout_4.jpg](../../images/callout_4.jpg)

- IGV displays data in horizontal rows called tracks. Typically, each track represents one sample or experiment. This example shows methylation, gene expression, copy number and mutation data.

**Features** ![callout_5.jpg](../../images/callout_5.jpg)

- IGV also displays features, such as genes, in tracks. By default, IGV displays data in one panel and features in another, as shown here. Drag-and-drop a track name to move a track from one panel to another. Combine data and feature panels by selecting that option on the General tab of the [Preferences window](https://www.broadinstitute.org/software/igv/?q=Preferences).

**Track names** ![callout_6.jpg](../../images/callout_6.jpg)

- Track names are listed in the far left panel. Legibility of the names depends on the height of the tracks; i.e., the smaller the track the less legible the name.

**Attributes** ![callout_7.jpg](../../images/callout_7.jpg)

- Attribute names are listed at the top of the attribute panel. Colored blocks represent attribute values, where each unique value is assigned a unique color. Hover over a colored block to see the attribute value. Click an attribute name to sort tracks based on that attribute value.


## Menu Bar

**File**

- Load from File - Displays genomic data from one or more files.

- Load from URL - Displays genomic data from a file identified by URL.

- Load from Server - Displays genomic data from the IGV data server.

- Load from DAS - Displays genomic data from a Distributed Annotation Server (DAS)

- New Session - Unloads all currently loaded data, as if you exited and restarted IGV

- Open Session - Opens a previously saved session file.

- Save Session - Saves your current settings to a named session file

- Save Image - Saves a snapshot of the IGV window to a graphics file, omitting the menu bar and tool bar.  Can save in .png, .jpg, or .svg format.

- Exit - Closes IGV.

**Genomes**

- Load Genome from File - Loads a genome into IGV from your file system.

- Load Genome from URL - Loads a genome into IGV from a web URL.

- Load Genome from Server - Loads a genome into IGV from the IGV data server.

- Create .genome File - Save your genome file in .genome format.

- Manage Genome List - Choose which genomes to display in the genome drop-down menu in the tool bar.

**View**

- Preferences - Opens a tabbed menu of data display preferences.

- Color Legends - Displays color legends for track data, which may be modified.

- Show Name Panel - Shows/hides the track name panel.

- Set Name Panel Width - Resets the track name panel width.

- Show Attribute Display - Shows/hides attributes and attribute values.

- Select Attributes to Show - Shows/hides selected attributes and attribute values.

- Show Header Panel - Shows/hides the chromosome location header panel.

- Reorder Panels - Allows the user to reorder the display panels.

- Go to - View and select loci visited in your navigation history.

**Tracks**

- Sort Tracks - Sorts track data.

- Group Tracks - Groups track data.

- Filter Tracks - Filters track data.

- Fit Data to Window - Sets the track height to display all of the data, or as much data as possible.

- Set Track Height - Sets the track height to a specified value.

**Regions**

- Region Navigator - Opens the region navigator.

- Gene Lists - Opens the gene lists window.

- Export Regions - Saves currently defined regions of interest to a BED file. If no regions of interest are defined, no BED file is created.

- Import Regions - Imports regions of interest from a BED file.

**Tools**

- Run Batch Script - Executes a series of sequential tasks.  Users can load at .txt file that contains a list of commands, one per line, that will be run by IGV.  The accepted commands are the same as the [IGV Port Commands](https://www.broadinstitute.org/software/igv/batch).

- Run igvtools - Launches the igvtools interface window.

- Find Motif - Search for a particular nucleotide sequence in the reference genome.

- Gitools Heatmaps - Data and results are represented as browsable heatmaps. Data can be exported from IGV in gitools format, or loaded directly into a running gitools session. See http://www.gitools.org for details.

- BEDTools - A fast, flexible suite of tools used to compare sets of genomic features. IGV uses BEDTools to compare features in loaded tracks, and view the results as a new track.

**GenomeSpace**

- Load File from GenomeSpace - Load a file into IGV from your GenomeSpace directory.

- Load Genome from GenomeSpace - Load a genome into IGV from your GenomeSpace directory.

- Save Session to GenomeSpace - Save current IGV session to your GenomeSpace directory.

- Load Session from GenomeSpace - Load a previous session from your GenomeSpace directory.

- Logout - Log out of GenomeSpace

- Register - Register a new account at [GenomeSpace](http://www.genomespace.org/)

**Help**

- User Guide... - Displays the IGV [User Guide](https://www.broadinstitute.org/software/igv/UserGuide).



- Help Forum... - In your default web browser, opens the home page for the [igv-help forum](https://groups.google.com/forum/#!forum/igv-help).



- About IGV - Displays IGV version and build number.



## Tool Bar


**Genome drop-down box** ![icon_genome_dropdown](../../images/icon_genome_dropdown.jpg)

- Loads a genome.

**Chromosome drop-down box** ![igv_chromdrop](../../images/igv_chromdrop.jpg)

- Zooms to a chromosome.

**Search box** ![searchbox](../../images/searchbox.jpg)

- Displays the chromosome location being shown.
To scroll to a different location, enter the gene name, locus, or track name and click Go.

**Whole genome view** ![icon_genome](../../images/icon_genome.jpg)

- Zooms to whole genome view.

![icon_backforward](../../images/icon_backforward.jpg)

- Moves backward and forward through views of the genome like the back and forward buttons in a web browser.

**Refresh** ![icon_refresh](../../images/icon_refresh.jpg)

- Refreshes the display.

**Define a region** ![icon_region_of_interest](../../images/icon_region_of_interest.jpg)

- Defines a region of interest on the chromosome.

![icon_squish](../../images/icon_squish.jpg)

- Reduces the row height on all tracks to fit all data for the region in view into the window; will also expand tracks (to their maximum preferred size) to fill the view, if needed.

![icon_popuponoff](../../images/icon_popuponoff.jpg)

- Toggles the pop-up information windows in IGV on or off.

**Zoom slider** ![icon_zoom](../../images/icon_zoom.jpg)

- Zooms in and out on a chromosome.

## Pop-up Menus

To select tracks and display the pop-up menu, do one of the following:

- Right-click a track to select it and display the pop-up menu.

- Right-click an attribute value to select all tracks with that attribute value and display the pop-up menu. Tip: Keep in mind that right-clicking an attribute may select tracks that are not visible in the data panel. Scroll down the data panel to view all the selected tracks.

- Control-click track names (Mac: Command-click) to select the tracks, then right-click one of the selections to display the pop-up menu.

Commands in the track pop-up menu change the display options for the selected tracks. Most changes made via the pop-up menu are lost when you exit IGV unless you [save the session](https://www.broadinstitute.org/software/igv/Sessions). In a few cases, changing the pop-up menu also changes an option in the Preferences window; these changes are persistent.

## Using IGV: Examples

### LOOKING AT ENCODE RNA-SEQ DATA


- Open IGV (.igv.sh from command line)

- Load the human genome :  In the toolbar, Click Genome > Load Genome from Server > Search and select Human (hg19)

- Load the encode RNA-Seq BAM files: Go to File > Load from ENCODE > Select C2C12 RNA-Seq BAM files (See image in the Pdf)

![load_encode_RNA-seq_BAM](../../images/load_encode_RNA-seq_BAM.png)


- Go to your favorite location in the genome by entering the location in the Search box, zoom in to the reads by using Zoom button.

- You can change the track properties by right-click on the track. Select View as pairs to see the left and right pairs of paired-end reads.

![view_pairs](../../images/view_pairs.png)

- You can make a sashimi plot to see reads aligning to each junction. Right-click > Sashimi plot.

![sashimi_plot](../../images/sashimi_plot.png)

- Save what you see by going to File > Save Image.



### LOOKING AT ENCODE ChIP-Seq DATA


- Go to File > Load from ENCODE > Select C2C12 ChIP-Seq bigwig files

![ChIP-Seq_bigwig](../../images/ChIP-Seq_bigwig.png)

- Reset the track heights to make them comparable. Right click on the track > Set data range. Then change the scale.

![data_range](../../images/data_range.png)

- You can also change the track colors, name and chart type (line,bar,heatmap).

- Save the image (File > Save Image)

## POPULAR FILE FORMATS THAT YOU CAN LOAD IN TO IGV


-  BAM

-  BED

-  BedGraph

-  bigBed

-  bigwig

-  FASTA

-  GFF/GTF

-  MAF

-  SAM/BAM

-  VCF

-  PSL

-  Narrowpeak

-  Many more

## SOME OTHER THINGS YOU CAN DO
 - Take multiple images from regions in a bed file. First make a batch file from the bed file using bedtools (bedtoIgv). Then load the script in Tools > Run Batch Script.

- Find a given motif. Tools > Find Motif.

- Set startup preferences. View > preferences.

- For more, have a look at the IGV documentation.





## FOR IGV HELP AND DOCUMENTATION

**GOTO: https://www.broadinstitute.org/igv/ or Click on Help in IGV menu.**


topic: introduction
tutorial: galaxy-cite
title: Best Practices for Citing Galaxy
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["fair", "how-to-cite-galaxy"]
content_hash: 1052319be2a4ed5bdf36d3c775952a71
last_modified: 2026-02-26T21:12:17.030534
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-cite/tutorial.html

content:
# Overview

This is a short primer on how to cite Galaxy in your publication and also how to prepare your Methodology based on your Galaxy history or histories. Following these steps will make it easy to cite Galaxy, share your methods, and adhere to best practices for FAIR and open science.


> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Citing Galaxy
Citing Galaxy comes in two main actions: 

1. Citing the global **Galaxy Project** that provides the core code that all Galaxy instances operate on. 
2. Citing the **instance of Galaxy**, you did your work on.

## Citing Galaxy Project
Please cite Galaxy Project’s primary publication: 

- ***The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update***
  - Nucleic Acids Research, Volume 52, Issue W1, 5 July 2024, Pages W83–W94, [https://doi.org/10.1093/nar/gkae410](https://doi.org/10.1093/nar/gkae410)

This is also found on the Galaxy Project Hub with more specific examples of how to cite individual aspects of Galaxy Project - [https://galaxyproject.org/citing-galaxy/](https://galaxyproject.org/citing-galaxy/)

> <comment-title></comment-title>
> The main Galaxy Project publication will be included in the list if you follow the instructions below to extract tool citations from your History.
>
{: .comment}

## Citing a Galaxy Service

### Service Descriptions
Each public Galaxy service benefits from being able to track the amazing work done by researchers using its service. Provided below are the DOIs for the Galaxy Australia, France and Main `usegalaxy.*` services. The linked papers are available on Zenodo in the Galaxy Project Zenodo Community at [https://zenodo.org/communities/galaxy/](https://zenodo.org/communities/galaxy/):

- Galaxy Australia - [https://doi.org/10.5281/zenodo.17298823](https://doi.org/10.5281/zenodo.17298823)
- Galaxy France - [https://doi.org/10.5281/zenodo.17298858](https://doi.org/10.5281/zenodo.17298858)
- Galaxy Main - [https://doi.org/10.5281/zenodo.17298864](https://doi.org/10.5281/zenodo.17298864)

We recommend including the relevant service name and DOI link in your Methods.


### Acknowledging a Service
Each public Galaxy service is provided through a complex mixture of funding, governance and infrastructure providers. As such, each service provides a cut/paste **Acknowledgement** statement that supports their particular configuration and this can be found on each service. If your publication supports Acknowledgements, we strongly recommend you include the service-recommended **Acknowledgements**.

**Acknowledgement** statements for the four main `usegalaxy.*` services can be found using the following links:

- Galaxy Australia - [https://site.usegalaxy.org.au/about#acknowledgement-statement](https://site.usegalaxy.org.au/about#acknowledgement-statement)
- Galaxy Europe - [https://usegalaxy-eu.github.io/about](https://usegalaxy-eu.github.io/about)
- Galaxy France - [https://www.france-bioinformatique.fr/en/services/acknowledgement/](https://www.france-bioinformatique.fr/en/services/acknowledgement/)
- Galaxy Main - [https://usegalaxy.org/#about](https://usegalaxy.org/#about)

# Writing a Methods section based on your Galaxy work

## Making a History public

We provide the following recommendations and process for making your History persistent and your options for public archiving of your History. Sharing your History will make it easier for others to understand or replicate your analysis. You can also add **annotations** and **tags** to ensure your history aligns to the FAIR principles.

### Preparing a History

We recommend cleaning up your History when preparing to make your History public.

> <hands-on-title>Clean up Unneeded Datasets</hands-on-title>
>
> 1. `Delete` and `Delete (permanently)` (purge) any failed jobs.
> 2. `Delete` and `Delete (permanently)` (purge) any datasets/job outputs that were used for testing and are not required for the final, published analysis.
> 3. `Delete` and `Delete (permanently)` (purge) any datasets that you don't want to share publicly (e.g. restricted data, unpublished results) - but make sure you save them to a private history first.
> 
> {% snippet faqs/galaxy/datasets_deleting.md %}
>
{: .hands_on}

We also suggest providing a description/context of the History in the `History Annotation`.

> <hands-on-title>Annotating a History</hands-on-title>
>
> 1. Add an annotation to your History.
>
> {% snippet faqs/galaxy/histories_annotation.md %}
>
{: .hands_on}

Users can add tags to their Galaxy histories to organise and connect them to specific analyses. Tags improve the findability and traceability of your work, aligning with the FAIR principles. Applying relevant tags before publishing your history is recommended to enhance its clarity and usefulness for both yourself and others.

> <hands-on-title>Adding Tags to a History</hands-on-title>
>
> 1. Add Tags to your History
>
> {% snippet faqs/galaxy/histories_tagging.md %}
>
{: .hands_on}

### Sharing a History

Galaxy provides a number of options for sharing histories which allows others to import and access the datasets, parameters, and steps of your history.

> <hands-on-title>Sharing a History</hands-on-title>
>
> 1. Choose one or several ways to share your History.
>
> {% snippet faqs/galaxy/histories_sharing.md %}
>
{: .hands_on}

### Exporting a History to Zenodo

Integration of InvenioRDM and Zenodo into Galaxy now enables you to export your history directly to Zenodo. Publishing your history on Zenodo will generate a DOI for your history, which you can include in your publication and share with other researchers to make your work easily citable.

For instructions on connecting to Zenodo and exporting your history, see the following tutorial on the Galaxy Training Network:

- [Integrating InvenioRDM-compatible Repositories with Galaxy]({% link topics/fair/tutorials/inveniordm-integration/tutorial.md %})

## Making a Workflow public

We recommend publishing a workflow alongside your data and in your article. In addition to providing a complete description of your method, the published workflow will also make your approach reproducible and citeable.

If you have not created and used a workflow for your analysis, Galaxy can automatically create a workflow based on the analysis you have performed in a history.

> <hands-on-title>Extract a Workflow from a History</hands-on-title>
>
> 1. Extract a Workflow from a Galaxy History
>
> {% snippet faqs/galaxy/workflows_extract_from_history.md %}
>
{: .hands_on}

For instructions on preparing your workflow, see the following tutorial on the Galaxy Training Network:

- [Annotate, prepare tests and publish Galaxy workflows in workflow registries]({% link topics/galaxy-interface/tutorials/workflow-fairification/tutorial.md %})

Two repository options that we recommend for publishing your Workflows are:

- WorkflowHub ([https://workflowhub.eu/](https://workflowhub.eu/))
- Dockstore ([https://dockstore.org/](https://dockstore.org/)) 

## Citing Public Workflows

If you used a workflow that was published on Galaxy, in Galaxy's workflow library, [the IWC](https://iwc.galaxyproject.org/)], or a repository such as WorkflowHub, then we recommend you cite the original workflow. You can cite a workflow as you would any other software. If the information is available, we recommend including the creator’s name, the date of creation, the name of the repository, and the workflow’s DOI or URL.

## Citing Public Data

If you used a public data source in your work, we recommend you cite both the data source publication DOI and the dataset itself. Most citation styles will include options for citing datasets. You should include the repository name and accession code in your Methods section to make it easy to find.

## Extracting Tool Citations from a Galaxy History

Galaxy can all citations of the tools used in your history. You can directly extract them from your history. It is the quickest way to add all these tools to your references, and you might find this helpful when writing up your methods.

> <hands-on-title>Extracting Tool Citations</hands-on-title>
>
> 1. Export a list of tool citations from your History
>
> {% snippet faqs/galaxy/cite-tools-used-in-a-history.md %}
> 
> > <comment-title></comment-title>
> > 
> > You will also get the Galaxy Project publication in all History citation exports!
> > 
> {: .comment}
> 
{: .hands_on}

 
# Conclusion

In this tutorial, we have covered how to cite Galaxy Project and the specific Galaxy server that you used and how to prepare your methodology based on your Galaxy History.


topic: introduction
tutorial: galaxy-intro-ngs-data-managment
title: NGS data logistics
difficulty: introductory
time_estimation: 1H30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: e55b6d79d377a5e2e4da911a21aeb208
last_modified: 2026-02-26T21:12:17.033433
zenodo_link: https://zenodo.org/records/15354240
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-ngs-data-managment/tutorial.html

content:
<!--

ALL IMAGES USED IN THIS TUTORIAL ARE AVAILABLE IN EDITABLE FORM HERE:

https://drive.google.com/drive/folders/0B7a9KfygumZFYWxIZlJqSkJwc2c?resourcekey=0-C7W_lazWNzjkKiplb9y01A&usp=sharing

-->

In this tutorial, we will look at practical aspects of manipulation of next-generation sequencing data. We will start with the FASTQ format produced by most sequencing machines and will finish with a table of variants present in four human samples we will analyze.

# The Story

To make this tutorial as realistic as possible we wanted to use an example from the real world. We will start with four sequencing datasets (fastq files) representing four individuals positive for malaria---a life-threatening disease caused by *Plasmodium* parasites---transmitted to humans through the bites of infected female *Anopheles* mosquitoes.

Our goal is to understand whether the malaria parasite ([*Plasmodium falciparum*](https://brc-analytics.dev.clevercanary.com/data/organisms/5833)) infecting these individuals is resistant to [Pyrimethamine](https://en.wikipedia.org/wiki/Pyrimethamine)---an antimalarial drug. Resistance to Pyrimethamine is conferred by a mutation in `PF3D7_0417200` (*dhfr*) gene ({% cite Cowman1988 %}). Given sequencing data from four individuals we will determine which one of them is infected with a *Plasmodium falciparum* carrying mutations in this gene.

An outline of our analysis looks like this:

![analysis_outline](../../../galaxy-interface/images/collections/collection_lifecycle.svg "This analysis begins with a set of fastq files representing four individuals. The sequencing was performed using a paired-end protocol. This means that each sample is represented by two files: forward reads (red) and reverse reads (blue). Before analysis begins these files are combined into a <i>Collection</i>. This allows processing all of them at once. The data is mapped against a reference genome of <i>P. falciparum</i>. This produces a series of BAM files. The BAM files are further passed through a variant caller. This produces VCF files. VCF files are then converted to tabular format and concatenated to create one final dataset. This dataset contains the answer to our question: which of the individual carries drug resistant mutations.").

# From reads to variants

In this tutorial we will use data from four infected individuals sequenced within the [MalariaGen](https://www.malariagen.net/data_package/open-dataset-plasmodium-falciparum-v70/) effort. We use the following four samples:

| Accession | Location |
|------------|------------|
| [ERR636434](https://www.ncbi.nlm.nih.gov/sra/?term=ERR636434) | Ivory coast |
| [ERR636028](https://www.ncbi.nlm.nih.gov/sra/?term=ERR636028) | Ivory coast |
| [ERR042232](https://www.ncbi.nlm.nih.gov/sra/?term=ERR042232) | Colombia |
| [ERR042228](https://www.ncbi.nlm.nih.gov/sra/?term=ERR042228) | Colombia |

These accessions correspond to datasets stored in the [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) at NCBI.

Let's do that 🚀

## Upload data into Galaxy

For this tutorial we down-sampled the data (made datasets smaller) to make sure that you can go through it quickly. We deposited these fastq datasets in a Zenodo library. Let's upload these datasets into Galaxy!

> <details-title>What is fastq data?</details-title>
>
> [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) is not a very well defined format. In the beginning various manufacturers of sequencing instruments were free to interpret FASTQ as they saw fit, resulting in a multitude of FASTQ flavors. This variation stemmed primarily from different ways of encoding quality values as described [on the Wikipedia article for FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) (below you will find an explanation of quality scores and their meaning). Today, the FASTQ Sanger version ({% cite Cock2009 %}) of the format is considered to be the standard form of FASTQ. Galaxy is using FASTQ Sanger as the only legitimate input for downstream processing tools and provides a number of utilities for converting FASTQ files ({% cite Blankenberg2010 %}) into this form (see **FASTQ Quality Control** section of Galaxy tools which is available on some usegalaxy.* instances).
>
> The FASTQ format looks like this:
>
> ```
> @M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
> CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCTGAACCAGCCAAGTAGCGTGCAG
> +
> ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE<,,CD,CEFC,@E9<FCFCF?9
> @M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
> CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACTGATCCAGCCATGCCGCGTGCAG
> +
> ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF<,CEF,CFGGD8FFCFCFEGCF
> @M02286:19:000000000-AA549:1:1101:11116:1322 1:N:0:23
> CCTACGGGAGGCAGCAGTAGGGAATCTTCGGCAATGGACGGAAGTCTGACCGAGCAACGCCGCGTGAGT
> +
> AAC<CCF+@@>CC,C9,F9C9@9-CFFFE@7@:+CC8-C@:7,@EFE,6CF:+8F7EFEEF@EGGGEEE
> ```
>
> Each sequencing read is represented by four lines:
>
> 1. `@` followed by read ID and optional information about sequencing run
> 2. sequenced bases
> 3. `+` (optionally followed by the read ID and some additional info)
> 4. Quality scores for each base of the sequence encoded as [ASCII symbols](https://en.wikipedia.org/wiki/ASCII)
>
> **Paired end data**
>
> It is common to prepare pair-end and mate-pair sequencing libraries. This is highly beneficial for a number of applications discussed in subsequent topics. For now, let's just briefly discuss what these are and how they manifest themselves in FASTQ form.
>
> ![Paired-end and mate-pair reads](../../images/pe_mp.png)
> In paired end sequencing (left) the actual ends of rather short DNA molecules (less than 1kb) are determined, while for mate pair sequencing (right) the ends of long molecules are joined and prepared in special sequencing libraries. In these mate pair protocols, the ends of long, size-selected molecules are connected with an internal adapter sequence (i.e. linker, yellow) in a circularization reaction. The circular molecule is then processed using restriction enzymes or fragmentation. Fragments are enriched for the linker and outer library adapters are added around the two combined molecule ends. The internal adapter can then be used as a second priming site for an additional sequencing reaction in the same orientation or sequencing can be performed from the second adapter, from the reverse strand (From Ph.D. dissertation by <a href="https://ul.qucosa.de/api/qucosa%3A11231/attachment/ATT-0/">Martin Kircher</a>
>
> Thus in both cases (paired-end and mate-pair) a single physical piece of DNA (or RNA in the case of RNA-seq) is sequenced from two ends and so generates two reads. These can be represented as separate files (two FASTQ files with first and second reads) or a single file were reads for each end are interleaved. Here are examples:
>
> *Two single files*
>
> File 1
>
> ```
> @M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
> CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCT
> +
> ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE
> @M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
> CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACT
> +
> ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF
> ```
>
> File 2
>
> ```
> @M02286:19:000000000-AA549:1:1101:12677:1273 2:N:0:23
> CACTACCCGTGTATCTAATCCTGTTTGATACCCGCACCTTCGAGCTTA
> +
> --8A,CCE+,,;,<CC,,<CE@,CFD,,C,CFF+@+@CCEF,,,B+C,
> @M02286:19:000000000-AA549:1:1101:15048:1299 2:N:0:23
> CACTACCGGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATC
> +
> -6AC,EE@::CF7CFF<<FFGGDFFF,@FGGGG?F7FEGGGDEFF>FF
> ```
>
> > <comment-title>Read order is important</comment-title>
> > Note that read IDs are **identical** in two files and they are listed in **the same** order. In some cases read IDs in the first and second file may be appended with `/1` and `/2` tags, respectively (however, this is not guaranteed).
> {: .comment}
>
>
> *Interleaved file*
>
> ```
> @1/1
> AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
> +
> EGGEGGGDFGEEEAEECGDEGGFEEGEFGBEEDDECFEFDD@CDD<ED
> @1/2
> CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
> +
> GHHHDFDFGFGEGFBGEGGEGEGGGHGFGHFHFHHHHHHHEF?EFEFF
> @2/1
> AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
> +
> HHHHHHEGFHEEFEEHEEHHGGEGGGGEFGFGGGGHHHHFBEEEEEFG
> @2/2
> CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
> +
> HHHHHHHHHHHHGHHHHHHGHHHHHHHHHHHFHHHFHHHHHHHHHHH
> ```
>
> In this tutorial paired-end data is represented as two different files.
>
{: .details}

> <hands-on-title>Upload accessions into Galaxy</hands-on-title>
>
> 1. Go to your Galaxy instance of choice such as one of the [UseGalaxy.* instances](https://galaxyproject.org/usegalaxy/) or any other.
> 1. Click *Upload Data* button:
> ![Data upload button](../../images/upload_button.png)
> 1. In the dialog box that would appear click "*Paste/Fetch*" button
> ![Choose local files button](../../images/paste.png)
> 1. Paste the following accessions into the box (red box in the image below):
> ```
> https://zenodo.org/records/15354240/files/ERR042228_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR042228_R.fq.gz
> https://zenodo.org/records/15354240/files/ERR042232_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR042232_R.fq.gz
> https://zenodo.org/records/15354240/files/ERR636028_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR636028_R.fq.gz
> https://zenodo.org/records/15354240/files/ERR636434_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR636434_R.fq.gz
> ```
> 1. Change datatype to `fastqsanger.gz` (green box in the image below)
> ![Name dataset and change datatype](../../images/upload_name_datatype.svg)
> 1. Click *Start* button
> 1. Close dialog by pressing **Close** button
> This will create eight datasets in your history on the right side of the interface:
> ![Four samples and eight files](../../images/f_and_r.svg)
> There are eight datasets because each sample has *forward* and *reverse* read files associated with it, so 4 &#215; 2 = 8
>
{: .hands_on}

## Bundle data into *Collection*

We are going to perform exactly the same analysis on all four samples. So it does not make sense to repeat the same operation eight times (imagine if you had a hundred or a thousand datasets). So before we go any further we will bundle the datasets we have in the history into a *Collection*. *Collections* in Galaxy are logical groupings of datasets that reflect the semantic relationships between them in the experiment / analysis. In this case, we will create a *paired collection* that will have two "levels"  (For more information on Collections see the [Collections tutorial]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}):

![Paired collection](../../images/paired_collection.svg "A paired collection is nested: the first level contains samples and the second contains individual forward and reverse reads associated with each sample")

> <hands-on-title>Creating a paired-collection</hands-on-title>
>
> To create a paired collection follow the steps shown in the video below (the video is 2 minutes long 😁):
>
> <div style="text-align: center;">
> <iframe width="560" height="315" src="https://www.youtube.com/embed/OW4n61BIHAc?si=namdcFYfUITktKRz" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
> </div>
>
> Explore the collections by first **clicking** on the collection name in the history panel. This takes you inside the collection and shows you the datasets in it.  You can then navigate back to the outer level and, finally, full history.
>
{: .hands_on}

<!-- The above embedding uses iframe because include _includes/youtube.html does not work from within hands-on section -->

## Assessing the quality of the data with **fastp** and **multiqc**

Raw fastq data is often contaminated with fragments of sequencing adapters used in library preparation and contain low quality reads or reads with low quality portions. {% tool fastp %} can automatically detect widely used sequencing adapters and remove them as well as prune low quality segments from reads. So, let's pass our data through **fastp**:

> <hands-on-title>Running <b>fastp</b></hands-on-title>
>
> Run {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.24.0+galaxy4) %} with the following parameters:
>    - "*Single-end or paired reads*": `Paired collection` (<font color="red">red outline</font>).
>    - "*Select paired collection(s)*": Collection we created in the previous step (<font color="red">red arrow</font>).
>
> ![fastp interface](../../images/fastp.svg)
>
> Fastp modifies files by removing standard Illumina adapters and applies a number of quality filters generating "Cleaned up data" shown above) as well as HTML and JSON reports as three collections:
>
> ![fastp history items](../../images/fastp_history.svg)
>
{: .hands_on}

You can click on individual HTML reports to get an idea about the quality of the data and degree of "cleanup". However, clicking on each dataset individually can become problematic if the number of datasets is large (you don't want to click on hundred datasets, do you?). We can visualize the QC data provided by **fastp** for **ALL** samples by feeding its JSON output to {% tool multiqc %}:

> <hands-on-title>Running <b>multiqc</b> on <b>fastp</b> JSON data</hands-on-title>
>
> Run {% tool [multiqc](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>
>    - "*Which tool was used to generate logs?*": `fastp` (<font color="red">red outline</font>).
>    - "*Output of fastp*": JSON output generated by **fastp** at the previous step (<font color="red">red arrow</font>).
>
> ![multiqc interface](../../images/multiqc.svg)
>
> `multiqc` will produce two outputs, but the one you care about has a word "Webpage" in it:
>
> ![multiqc history item](../../images/multiqc_history.svg)
>
> Click on the {% icon galaxy-eye %} (eye) icon and you will see the QC report.
>
{: .hands_on}

Figure below shows one of the plots produced by `multiqc`---distribution of quality values across positions for forward reads after processing.

![distribution of quality values across positions for forward reads after processing](../../images/multiqc_f.png "Quality score distribution after filtering. Here you can see that two samples have very high quality values and 100 bp reads. Two other samples have somewhat lower but still acceptable values and shorter, 80bp, reads.")

> <details-title>What are base qualities? What is good and what is bad?</details-title>
>
> **What are base qualities?**
>
> As we've seen above, FASTQ datasets contain two types of information:
>
> - *sequence of the read*
> - *base qualities* for each nucleotide in the read.
>
> The base qualities allow us to judge how trustworthy each base in a sequencing read is. The following excerpt from an excellent [tutorial](https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf) by Friederike D&uuml;ndar, Luce Skrabanek, Paul Zumbo explains what base qualities are:
>
> > <comment-title>From "Introduction to differential gene expression analysis using RNA-seq"</comment-title>
> > Illumina sequencing is based on identifying the individual nucleotides by the fluorescence signal emitted upon their incorporation into the growing sequencing read. Once the fluorescence intensities are extracted and translated into the four letter code. The deduction of nucleotide sequences from the images acquired during sequencing is commonly referred to as base calling.
> ><br><br>
> > Due to the imperfect nature of the sequencing process and limitations of the optical instruments, base calling will always have inherent uncertainty. This is the reason why FASTQ files store the DNA sequence of each read together with a position-specific quality score that represents the error probability, i.e., how likely it is that an individual base call may be incorrect. The score is called [Phred score](http://www.phrap.com/phred/), $$Q$$, which is proportional to the probability $$p$$ that a base call is incorrect, where $$Q = −10lg(p)$$. For example, a Phred score of 10 corresponds to one error in every ten base calls ($$Q = −10lg(0.1)$$), or 90% accuracy; a Phred score of 20 corresponds to one error in every 100 base calls, or 99% accuracy. A higher Phred score thus reflects higher confidence in the reported base.
> ><br><br>
> > To assign each base a unique score identifier (instead of numbers of varying character length), Phred scores are typically represented as ASCII characters. At http://ascii-code.com/ you can see which characters are assigned to what number.
> ><br><br>
> > For raw reads, the range of scores will depend on the sequencing technology and the base caller used (Illumina, for example, used a tool called Bustard, or, more recently, RTA). Unfortunately, Illumina has been anything but consistent in how they calculated and ASCII-encoded the Phred score (see below)! In addition, Illumina now allows Phred scores for base calls with as high as 45, while 41 used to be the maximum score until the HiSeq X. This may cause issues with downstream applications that expect an upper limit of 41.
> {: .comment}
>
> ![Illumina quality score](../../images/illumina_qs.png)
>
> Base call quality scores are represented with the Phred range. Different Illumina (formerly Solexa) versions
> used different scores and ASCII offsets. Starting with Illumina format 1.8, the score now represents the standard
> Sanger/Phred format that is also used by other sequencing platforms and the sequencing archives.
>
> ![FASTQ quality score](../../images/fastq_qs.png)
>
>The ASCII interpretation and ranges of the different Phred score notations used by Illumina and the original Sanger interpretation. Although the Sanger format allows a theoretical score of 93, raw sequencing reads typically do not exceed a Phred score of 60. In fact, most Illumina-based sequencing will result in maximum scores of 41 to 45 (image from Wikipedia)
>
> **What is good and what is bad?**
>
> One of the first steps in the analysis of NGS data is seeing how good the data actually is. [FastqQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) is the most famous tool allowing you to assess the quality of FASTQ datasets (and deciding whether to blame or not to blame whoever has done sequencing for you). [Falco](https://github.com/smithlabcode/falco) is an alternative tool for quality control. Falco is a improved implementation of FastQC for high throughput sequence quality control.
>
> ![Good quality in FastQC](../../images/qc_plot.svg)
> Here, you can see **FastQC** base quality reports (the tools gives you many other types of data) for two datasets: <b>A</b> and <b>B</b>. The A dataset has long reads (250 bp) and very good quality profile with no qualities dropping below phred score of 30. The B dataset is significantly worse with ends of the reads dipping below phred score of 20. The B reads may need to be trimmed for further processing. **Falco** will generate almost the same reports about the quality of the reads.")
{: .details}

## Mapping reads

We can proceed with mapping reads. Galaxy has a number of mappers including **bowtie**, **bwa-mem**, and **bwa-mem2**. For this analysis we will use {% tool bwa-mem2 %}---the latest version of this popular and "battle-tested" tool.

### Upload reference genome

The key question when mapping reads against a genome is whether the index for this genome---a datastructure **bwa-mem2** uses to quickly find matches---is already installed on Galaxy or not. Let's assume that it is **NOT** present in Galaxy. In this case you will need to upload the genome. In this case we will use reference genome of 3D7 strain *P. falciparum*.

> <hands-on-title>Uploading the genome for <i>P. falciparum</i></hands-on-title>
>
> To download the genome paste the following URL into the {% tool Upload %} tool:
>
> ```
> https://zenodo.org/records/15354240/files/GCF_000002765.6.fa.gz
> ```
> The only difference is that here with the example we've seen above is that you need to set datatype (green box) to `fasta.gz`:
> ![Genome upload](../../images/upload_genome.svg)
>
{: .hands_on}

### Map the reads

Now we can map the reads against the uploaded genome:

> <hands-on-title>Map sequencing reads with <b>bwa-mem</b></hands-on-title>
>
> Run {% tool [BWA-MEM2](toolshed.g2.bx.psu.edu/repos/iuc/bwa_mem2/bwa_mem2/2.2.1+galaxy4) %} with the parameters shown in the image below:
>
>    - "*Will you select a reference genome from your history or use a built-in index?*": `Use a reference genome from history and build index if necessary` (<font color="red">red outline</font>).
>    - "*Use the following dataset as the reference*": Choose the reference we uploaded at the previous step (<font color="red">red arrow</font>).
>    - "*Single or Paired-end reads*": `Paired collection` (<font color="green">green outline</font>).
>    - "*Select a paired collection*": Select the collection of fastq datasets produced by **fastp** (<font color="green">green arrow</font>).
>
> ![BWA-MEM2 interface](../../images/bwamem2.svg)
>
{: .hands_on}

**bwa-mem2** produces four BAM outputs which we will process further and will use to call variants.

> <details-title>What are BAM files?</details-title>
>
> **SAM/BAM data**
>
> The [SAM/BAM](https://samtools.github.io/hts-specs/SAMv1.pdf) format is an accepted standard for storing aligned reads (it can also store unaligned reads and some mappers such as {% tool [Map with BWA](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.19) %} are accepting unaligned BAM as input). The binary form of the format (BAM) is compact and can be rapidly searched (if indexed). In Galaxy, BAM datasets are always indexed (accompanies by a .bai file) and sorted in coordinate order. In the following discussion, we once again rely on [tutorial](https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf) by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo.
>
> The Sequence Alignment/Map (SAM) format is, in fact, a generic nucleotide alignment format that describes the alignment of sequencing reads (or query sequences) to a reference. The human readable, TAB delimited SAM files can be compressed into the Binary Alignment/Map format. These BAM files are bigger than simply gzipped SAM files, because they have been optimized for fast random access rather than size reduction. Position-sorted BAM files can be indexed so that all reads aligning to a locus can be efficiently retrieved without loading the entire file into memory.
>
> As shown below, SAM files typically contain a short header section and a very long alignment section where each row represents a single read alignment. The following sections will explain the SAM format in a bit more detail. For the most comprehensive and updated information go to [SAM/BAM and related specifications GitHub repository.](https://github.com/samtools/hts-specs)
>
> ![BAM structure](../../images/bam_structure.png)
> Each line of the optional header section starts with <tt>@</tt>, followed by the appropriate abbreviation (e.g., SQ for sequence dictionary which lists all chromosomes names (SN) and their lengths (LN)). The vast majority of lines within a SAM file typically correspond to read alignments where each read is described by the 11 mandatory entries (black font) and a variable number of optional fields (grey font). From <a href="https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf">a tutorial</a> by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo."
>
> **SAM Header**
>
> The header section includes information about how the alignment was generated and stored. All lines in the header section are tab-delimited and begin with the “@” character, followed by tag:value pairs, where tag is a two-letter string that defines the content and the format of value. For example, the “@SQ” line in the header section contains the information about the names and lengths of the **reference** sequences to which the reads were aligned. For a hypothetical organism with three chromosomes of length 1,000 bp, the SAM header should contain the following three lines:
>
> ```
> @SQ SN:chr1 LN:1000
> @SQ SN:chr2 LN:1000
> @SQ SN:chr3 LN:1000
> ```
>
> **SAM alignment section**
>
> The optional header section is followed by the alignment section where each line corresponds to one sequenced read. For each read, there are 11 mandatory fields that always appear in the same order:
>
> ```
> <QNAME> <FLAG> <RNAME> <POS> <MAPQ> <CIGAR> <MRNM> <MPOS> <ISIZE> <SEQ> <QUAL>
> ```
>
> If the corresponding information is unavailable or irrelevant, field values can be ‘0’ or ‘*’ (depending on the field, see below), but they cannot be missing! After the 11 mandatory fields, a variable number of optional fields can be present. Here is an example of one single line of a real-life SAM file (you may need to scroll sideways):
>
> ```
> ERR458493 .552967 16 chrI 140 255 12 M61232N37M2S * 0 0 CCACTCGTTCACCAGGGCCGGCGGGCTGATCACTTTATCGTGCATCTTGGC BB?HHJJIGHHJIGIIJJIJGIJIJJIIIGHBJJJJJJHHHHFFDDDA1+B NH:i:1 HI:i:1 AS:i:41 nM:i:2
> ```
>
> The following table explains the format and content of each field. The `FLAG`, `CIGAR`, and the optional fields (marked with pale blue background) are explained in more detail below. The number of optional fields can vary widely between different SAM files and even between reads within in the same file. The field types marked in blue are explained in more detail in the text below.
>
> ![SAM fields](../../images/sam_fields.png)
>
> **`FLAG` field**
>
> The FLAG field encodes various pieces of information about the individual read, which is particularly important for Paired-End (PE) reads. It contains an integer that is generated from a sequence of bits (0, 1). This way, answers to multiple binary (Yes/No) questions can be compactly stored as a series of bits, where each of the single bits can be addressed and assigned separately.
>
> The following table gives an overview of the different properties that can be encoded in the FLAG field. The developers of the SAM format and samtools tend to use the hexadecimal encoding as a means to refer to the different bits in their documentation. The value of the FLAG field in a given SAM file, however, will always be the decimal representation of the sum of the underlying binary values (as shown in Table below, row 2).
>
> ![SAM flag](../../images/sam_flag.png)
> The <tt>FLAG</tt> field of SAM files stores information about the respective read alignment in one single decimal number. The decimal number is the sum of all the answers to the Yes/No questions associated with each binary bit. The hexadecimal representation is used to refer to the individual bits (questions). A bit is set if the corresponding state is true. For example, if a read is paired, <tt>0x1</tt> will be set, returning the decimal value of 1. Therefore, all <tt>FLAG</tt> values associated with paired reads must be uneven decimal numbers. Conversely, if the <tt>0x1</tt> bit is unset (= read is not paired), no assumptions can be made about <tt>0x2</tt>, <tt>0x8</tt>, <tt>0x20</tt>, <tt>0x40</tt> and <tt>0x80</tt> because they refer to paired reads. From <a href="https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf">a tutorial</a> by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo."
>
> In a run with single reads, the flags you most commonly see are:
>
> - 0: This read has been mapped to the forward strand. (None of the bit-wise flags have been set.)
> - 4: The read is unmapped (`0x4` is set).
> - 16: The read is mapped to the reverse strand (`0x10` is set)
>
> (`0x100`, `0x200` and `0x400` are not used by most aligners/mappers, but could, in principle be set for single reads.) Some common `FLAG` values that you may see in a PE experiment include:
>
>
> |**69** (= 1 + 4 + 64) | The read is paired, is the first read in the pair, and is unmapped.|
> |**77** (= 1 + 4 + 8 + 64) | The read is paired, is the first read in the pair, both are unmapped.|
> |**83** (= 1 + 2 + 16 + 64) | The read is paired, mapped in a proper pair, is the first read in the pair, and it is mapped to the reverse strand.|
> |**99** (= 1 + 2 + 32 + 64) | The read is paired, mapped in a proper pair, is the first read in the pair, and its mate is mapped to the reverse strand.|
> |**133** (= 1 + 4 + 128) | The read is paired, is the second read in the pair, and it is unmapped.|
> |**137** (= 1 + 8 + 128) | The read is paired, is the second read in the pair, and it is mapped while its mate is not.|
> |**141** (= 1 + 4 + 8 + 128) | The read is paired, is the second read in the pair, but both are unmapped.|
> |**147** (= 1 + 2 + 16 + 128) | The read is paired, mapped in a proper pair, is the second read in the pair, and mapped to the reverse strand.|
> |**163** (= 1 + 2 + 32 + 128) | The read is paired, mapped in a proper pair, is the second read in the pair, and its mate is mapped to the reverse strand.|
>
>
> **`CIGAR` string**
>
> `CIGAR` stands for *Concise Idiosyncratic Gapped Alignment Report*. This sixth field of a SAM file
> contains a so-called CIGAR string indicating which operations were necessary to map the read to the reference sequence at that particular locus.
>
> The following operations are defined in CIGAR format (also see figure below):
>
> - **M** - Alignment (can be a sequence match or mismatch!)
> - **I** - Insertion in the read compared to the reference
> - **D** - Deletion in the read compared to the reference
> - **N** - Skipped region from the reference. For mRNA-to-genome alignments, an N operation represents an intron. For other types of alignments, the interpretation of N is not defined.
> - **S** - Soft clipping (clipped sequences are present in read); S may only have H operations between them and the ends of the string
> - **H** - Hard clipping (clipped sequences are NOT present in the alignment record); can only be present as the first and/or last operation
> - **P** - Padding (silent deletion from padded reference)
> - **=** - Sequence match (not widely used)
> - **X** - Sequence mismatch (not widely used)
>
> The sum of lengths of the **M**, **I**, **S**, **=**, **X** operations must equal the length of the read. Here are some examples:
>
> ![CIGAR](../../images/cigar.png)
> From <a href="https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf">a tutorial</a> by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo.")
>
> **Optional fields**
>
> Following the eleven mandatory SAM file fields, the optional fields are presented as key-value
> pairs in the format of `<TAG>:<TYPE>:<VALUE>`, where `TYPE` is one of:
>
> - `A` - Character
> - `i` - Integer
> - `f` - Float number
> - `Z` - String
> - `H` - Hex string
>
> The information stored in these optional fields will vary widely depending on the mapper and new tags can be added freely. In addition, reads within the same SAM file may have different numbers of optional fields, depending on the program that generated the SAM file. Commonly used optional tags include:
>
> - `AS:i` - Alignment score
> - `BC:Z` - Barcode sequence
> - `HI:i` - Match is i-th hit to the read
> - `NH:i` - Number of reported alignments for the query sequence
> - `NM:i` - Edit distance of the query to the reference
> - `MD:Z` - String that contains the exact positions of mismatches (should complement the CIGAR string)
> - `RG:Z` - Read group (should match the entry after ID if @RG is present in the header.
>
> Thus, for example, we can use the NM:i:0 tag to select only those reads which map perfectly to the reference (i.e., have no mismatches). While the optional fields listed above are fairly standardized, tags that begin with `X`, `Y`, and `Z` are reserved for particularly free usage and will never be part of the official SAM file format specifications. `XS`, for example, is used by TopHat (an RNA-seq analysis tool we will discuss later) to encode the strand information (e.g., `XS:A:+`) while Bowtie2 and BWA use `XS:i:` for reads with multiple alignments to store the alignment score for the next-best-scoring alignment (e.g., `XS:i:30`).
>
> **Read Groups**
>
> One of the key features of SAM/BAM format is the ability to label individual reads with readgroup tags. This allows pooling results of multiple experiments into a single BAM dataset. This significantly simplifies downstream logistics: instead of dealing with multiple datasets one can handle just one. Many downstream analysis tools such as variant callers are designed to recognize readgroup data and output results on per-readgroup basis.
>
> One of the best descriptions of BAM readgroups is on [GATK support site](https://gatkforums.broadinstitute.org):
>
> ![Read groups](../../images/rg.png)
>
> GATK forum also provides the following example:
>
> ![Read group example](../../images/rg_example.png)
>
> **Manipulating SAM/BAM datasets**
>
> We support four major toolsets for processing of SAM/BAM datasets:
>
>  * [DeepTools](https://deeptools.readthedocs.io) - a suite of user-friendly tools for the visualization, quality control and normalization of data from deep-sequencing DNA sequencing experiments.
>  * [SAMtools](http://www.htslib.org/) - various utilities for manipulating alignments in the SAM/BAM format, including sorting, merging, indexing and generating alignments in a per-position format.
>  * [BEDtools](https://bedtools.readthedocs.io/en/latest/) - a toolkit originally written for BED format was expanded for analysis of BAM and VCF datasets.
>  * [Picard](https://broadinstitute.github.io/picard/) - a set of Java tools for manipulating high-throughput sequencing data (HTS) data and formats.
{: .details}

### Remove duplicates

In many cases there could be artifacts that can be detected in mapped reads. One of such artifacts are duplicate reads.

> <details-title>What are read duplicates?</details-title>
>
> **PCR duplicates**
>
> Preparation of sequencing libraries (at least at the time of writing) for technologies such as Illumina (used in this example) involves PCR amplification. It is required to generate sufficient number of sequencing templates so that a reliable detection can be performed by base callers. Yet, PCR has its biases, which are especially profound in cases of multitemplate PCR used for construction of sequencing libraries ({% cite Kanagawa2003 %}).
>
> ![PCR duplicates](../../images/pcr-duplicates.png)
> Analyzing molecules aligning with the same outer coordinates, a mapping quality of at least 30 and a length of at least 30 nucleotide, resulted in an average coverage of 12.9 per PCR duplicate and an empirical coverage distribution similar to an exponential/power law distribution (left upper panel). This indicates that many molecules are only observed for deeper sequencing while other molecules are available at higher frequencies. Analyzing length (left middle panel) and GC content (left lower panel) patterns as well as the combination (right panel) shows higher PCR duplicate counts for a GC content between 30% to 70% as well as for shorter molecules compared to longer molecules. This effect may be due to an amplification bias from the polymerase or the cluster generation process necessary for Illumina sequencing. (From Ph.D. dissertation by <a href=\"https://ul.qucosa.de/api/qucosa%3A11231/attachment/ATT-0/\">Martin Kircher</a>)."
>
> Duplicates can be identified based on their outer alignment coordinates or using sequence-based clustering. One of the common ways for identification of duplicate reads is the `MarkDuplicates` utility from [Picard](https://broadinstitute.github.io/picard/command-line-overview.html) package, which we will use later in this tutorial.
>
> **Sampling coincidence duplicates**
>
> However, one has to be careful when removing duplicates in cases when the sequencing targets are small (e.g., sequencing of bacterial, viral, or organellar genomes as well as amplicons). This is because when sequencing target is small reads will have the same coordinates by chance and not because of PCR amplification issues. The figure below illustrates the fine balance between estimates allele frequency, coverage, and variation in insert size (from ({% cite Zhou2014 %})):
>
> ![Sampling bias](../../images/sampling-bias.png)
>The Variant Allele Frequency (VAF) bias determined by coverage and insert size variance. Reads are paired-end and read length is 76. The insert size distribution is modeled as a Gaussian distribution with mean at 200 and standard deviation shown on the X-axis. The true VAF is 0.05. The darkness at each position indicates the magnitude of the bias in the VAF ({% cite Zhou2014 %}).")
{: .details}

Removing duplicates is particularly important for identification of sequence variants as they can affect their frequencies. It is performed with the {% tool MarkDuplicates %} tool:

> <hands-on-title>Remove duplicates</hands-on-title>
>
> Run {% tool [MarkDuplicates](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/3.1.1.0)%} with the following parameters:
>    - {% icon param-file %} *"Select SAM/BAM dataset or dataset collection"*: `bam_output` (output of {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.19) %}) as shown with the <font color="red">red arrow</font>.
>    - *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"*: `Yes` as shown with <font color="green">green arrow</font>.
>
> ![Mark Duplicates](../../images/markdups.svg)
{: .hands_on}

## Calling variants

Now we are ready to proceed with variant calling. For this purpose we will use a tool called **lofreq**. One thing to keep in mind in our case is that the samples are from human blood. In humans *Plasmodium* parasites exist in **haploid** state (only zygote is diploid and this stage happens in mosquito). **lofreq** is specifically designed for calling variants in this scenario.

### Realign reads

One of the key issues with accurate identification of sequence variants is normalizing indels (insertions and deletions).

> <details-title>What is the problem with indels?</details-title>
>
>Left aligning of indels (a variant of re-aligning) is extremely important for obtaining accurate variant calls. For illustrating how left-aligning works, we expanded on an example provided by Tan et al. [2015](https://pubmed.ncbi.nlm.nih.gov/25701572/). Suppose you have a reference sequence and a sequencing read:
>
>
>```
>Reference GGGCACACACAGGG
>Read      GGGCACACAGGG
>```
>
>If you look carefully you will see that the read is simply missing a `CA` repeat. But it is not apparent to a mapper, so some of possible alignments and corresponding variant calls include:
>
>```
>Alignment                 Variant Call
>
>GGGCACACACAGGG            Ref: CAC
>GGGCAC--ACAGGG            Alt: C
>
>GGGCACACACAGGG            Ref: ACA
>GGGCA--CACAGGG            Alt: A
>
>GGGCACACACAGGG            Ref: GCA
>GGG--CACACAGGG            Alt: G
>```
>
>While we cannot know which of these arrangements is correct, left-aligning guarantees consistency.
>
{: .details}

{% tool Realign reads %} is a part of **lofreq** package. It corrects misalignment around insertions and deletions:

> <hands-on-title>Realign reads around indels</hands-on-title>
>
> Run {% tool [Realign reads](toolshed.g2.bx.psu.edu/repos/iuc/lofreq_viterbi/lofreq_viterbi/2.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Reads to realign"*: output of {% tool MarkDuplicates %} as shown with the <font color="red">red arrow</font>.
>    - *"Choose the source for the reference genome"*: `History`
>        - {% icon param-file %} *"Reference"*: set this to the [previously uploaded reference genome](#upload-reference-genome) (<font color="green">green outline and arrow</font>).
>
> ![Realign reads](../../images/realign_lofreq.svg)
>
{: .hands_on}

### Call variants

We are now ready to actually call variants:

> <hands-on-title>Call variants</hands-on-title>
>
> Run {% tool [Call variants](toolshed.g2.bx.psu.edu/repos/iuc/lofreq_call/lofreq_call/2.1.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Input reads in BAM format"*: Output of {% tool Insert indel qualities %} (<font color="red">red arrow</font>)
>    - *"Choose the source for the reference genome"*: `History` (<font color="green">green outline</font>).
>        - {% icon param-file %} *"Reference"*: set this to the [previously uploaded reference genome](#upload-reference-genome) (<font color="green">green outline and arrow</font>).
>    - *"Types of variants to call"*: `SNVs and indels` (<font color="blue">blue outline and arrow</font>).
>    - *"Variant calling parameters"*: `Configure settings` (<font color="orange">orange outline</font>).
>        - In *"Coverage"*:
>            - *"Minimal coverage"*: `10` (<font color="orange">orange arrow</font>).
>        - In *"Base-calling quality"*:
>            - *"Minimum baseQ"*: `20` (<font color="orange">orange arrow</font>).
>            - *"Minimum baseQ for alternate bases"*: `20`  (<font color="orange">orange arrow</font>).
>        - In *"Mapping quality"*:
>            - *"Minimum mapping quality"*: `20` (<font color="orange">orange arrow</font>).
>
> ![Call variants with lofreq](../../images/lofreq.svg)
{: .hands_on}

The output of this step is a collection of [VCF](https://en.wikipedia.org/wiki/Variant_Call_Format) files containing information on all variants found between the reads and the reference genome.

## Annotating variants

### Preparing {% tool SnpEff %} database

We will now annotate the variants we called in the previous step with the effect they may have on the *Plasmodium* phenotype. In order to do this we need to create a database that can be used by {% tool SnpEff %}. This process requires a reference genome (which we already uploaded and a list of genes present in this genome---a file we have not uploaded yet.

> <hands-on-title>Uploading gene annotations for <i>P. falciparum</i></hands-on-title>
>
> To download gene annotations paste the following URL into the {% tool Upload %} tool as was shown previously in this tutorial already twice (for fastq files and for reference genome).
>
> ```
> https://zenodo.org/records/15354240/files/GCF_000002765.6_GCA_000002765.ncbiRefSeq.gtf.gz
> ```
> Set datatype (<font color="green">green</font> box) to `gtf.gz`:
>
> ![Genome upload](../../images/gtf_upload.svg)
>
{: .hands_on}

Now we can run {% tool SnpEff build %}:

> <hands-on-title>Prepare snpEff database with <b>SnpEff Build</b></hands-on-title>
>
> Run {% tool [SnpEff Build](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff_build_gb/5.2+galaxy0) %} with the following parameters:
>
>    - *"Name for the database"*: `Pf` (<font color="red">red arrow</font>).
>    - *"Input annotations are in"*: `GTF` (<font color="green">green arrow</font>).
>    - *"GTF dataset to build database from"*: Select the GTF file uploaded at the previous step (<font color="blue">blue arrow</font>).
>    - *"Choose the source for the reference genome"*: `History` (<font color="orange">orange outline</font>).
>    - *"Genome in FASTA format"*: Choose reference genome uploaded earlier (<font color="orange">orange arrow</font>).
>
> ![SnpEff Build](../../images/snpeff_build.svg)
>
{: .hands_on}

### Annotate variant effects with **SnpEff eff**

We are now ready to annotate variants with {% tool SnpEff eff %} (here "eff" stands for effect):

> <hands-on-title>Annotate variant effects</hands-on-title>
>
> Run {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/5.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequence changes (SNPs, MNPs, InDels)"*: Output of {% tool lofreq %} we've run previously <font color="red">red arrow</font>)
>    - *"Genome source"*: `Custom snpEff database in your history` <font color="green">green outline</font>)
>      - "*SnpEff5.2 Genome Data*": set to the SnpEff database we built in the previous step <font color="green">green arrow</font>)
>
>![snpEff eff](../../images/snpeff.svg)
>
{: .hands_on}

### Create table of variants using **SnpSift Extract Fields**

Previous step generated a set of annotated VCF files. These can be visualized in a genome browser, but we will go a different way. We will convert them into tab-delimited files that can be easily processed.

> <hands-on-title>Create table of variants</hands-on-title>
>
> Run {% tool [SnpSift Extract Fields](toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_extractFields/4.3+t.galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Variant input file in VCF format"*: Output of {% tool SnpEff eff %} from previous step (<font color="red">red arrow</font>)
>    - *"Fields to extract"*: Copy the following string into a box highlighted with (<font color="green">green outline</font>):
>
> ```
>CHROM POS REF ALT QUAL DP AF SB DP4 ANN[*].EFFECT ANN[*].IMPACT ANN[*].GENE ANN[*].AA_POS ANN[*].HGVS_C ANN[*].HGVS_P
> ```
>
>    - *"One effect per line"*: `Yes` (<font color="blue">blue arrow</font>)
>
> ![snpsift extract fields](../../images/snpsift_extract_fields.svg)
>
{: .hands_on}

Let's explain which fields we are extracting:

| # | Field | Meaning |
|---|----------|-------------|
| 1 | `CHROM` | Name of the chromosome |
| 2 | `POS` | Position within that chromosome |
| 3 | `REF` | Reference allele (the one present in the *reference* |
| 4 | `ALT` | Alternative allele (the one observed in the reads) |
| 5 | `QUAL` | Variant quality computed by **lowfreq** |
| 6 | `DP` | Read depth (the number of reads covering this position) |
| 7 | `AF` | Alternative allele frequency |
| 8 | `SB` | Strand bias P-value from Fisher's exact test calculated by lofreq |
| 9 | `DP4` | Depth for Forward Ref Counts, Reverse Ref Counts, Forward Alt Counts, Reverse Alt Counts |
| 10 | `ANN[*].EFFECT` | Expected effect of the variant on the protein |
| 11 | `ANN[*].IMPACT` | Expected impact of the phenotype |
| 12 | `ANN[*].GENE` | Name of the gene containing the variant |
| 13 | `ANN[*].AA_POS `| Amino acid position within the gene affected by this variant |
| 14 | `ANN[*].HGVS_C` | Nucleotide change in nucleotide coordinates relative to the start of the coding region |
| 15 | `ANN[*].HGVS_P` | Amino acid change in amino acid coordinates relative to the start of the protein |

### Collapse data into a single dataset

We now extracted meaningful fields from VCF datasets. But they still exist as a collection. To move towards secondary analysis we need to **collapse** this collection into a single dataset. "Collapsing" simply concatenates the content of collection elements and attaches sample IDs, so that we know which line in the concatenated file corresponds to which sample:

![Collapsing a collection](../../images/collapse.svg "Collapsing combines a collection into a single file dataset and includes names of collection elements as a new column. Here each line in the collapsed dataset (right) is pre-pended with the element's name such as 1, 2, 3, and 4.")

For more information about collapsing collections, please watch the following YouTube video:

{% include _includes/youtube.html id="ypuFZ1RKMIY" title="Dataset Collections: Collapsing a collection" %}

> <hands-on-title>Collapse a collection</hands-on-title>
>
> Run {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: Output of {% tool SnpSift Extract Fields %} from the previous step (<font color="red">red arrow</font>).
>    - "*Keep one header line*": `Yes` (<font color="green">green arrow</font>)
>    - "*Prepend File name*": `Yes` (<font color="blue">blue arrow</font>)
>    - "*Where to add dataset name*": `Same line and each line in dataset` (<font color="orange">orange outline</font>)
>
> ![Collapse tool UI](../../images/collapse_ui.svg)
>
{: .hands_on}

You can see that this tool takes lines from all collection elements (in this tutorial we have two), add element name as the first column, and pastes everything together. So, if we have a collection as an input:

> <code-in-title>A collection with two items (<b>15</b> columns)</code-in-title>
> A collection element named `ERR042228.fq`
>
>```
>NC_004318.2   676399   C    T     1344.0   45   0.977778   0   0,0,23,22   missense_variant    MODERATE PF3D7_0415200   914     c.2740G>A    p.Glu914Lys
>NC_004318.2   676631   C    T     2000.0   60   0.966667   0   0,0,32,27   synonymous_variant  LOW      PF3D7_0415200   836     c.2508G>A    p.Lys836Lys
>```
>
>A collection element named `ERR042232.fq`:
>
>```
>NC_004318.2   671152   G    A     1324.0   42   0.952381   0   0,1,20,21   synonymous_variant  LOW      PF3D7_0415100   90      c.270G>A   p.Gln90Gln
>NC_004318.2   671641   A    T     1300.0   43   0.906977   0   0,0,16,27   missense_variant    MODERATE PF3D7_0415100   253     c.759A>T   p.Glu253Asp
>```
{: .code-in}

We will have a single dataset as the output:

> <code-out-title>A single dataset (<b>16</b> columns)</code-out-title>
>
>then the {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} will produce this:
>
>```
>ERR042228.fq NC_004318.2   676399   C    T     1344.0   45   0.977778   0   0,0,23,22   missense_variant    MODERATE PF3D7_0415200   914     c.2740G>A    p.Glu914Lys
>ERR042228.fq NC_004318.2   676631   C    T     2000.0   60   0.966667   0   0,0,32,27   synonymous_variant  LOW      PF3D7_0415200   836     c.2508G>A    p.Lys836Lys
>ERR04223


topic: introduction
tutorial: galaxy-intro-101
title: Galaxy Basics for genomics
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: b8d5afc9aa8a3b64f4284752c0d98331
last_modified: 2026-02-26T21:12:17.032080
zenodo_link: https://doi.org/10.5281/zenodo.4104428
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-101/tutorial.html

content:
This tutorial aims to familiarize you with the Galaxy user interface. It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows, and sharing your work.

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Setting the stage: Exons and SNPs

We start with the question: In human chromosome 22, which exon has the highest number of single nucleotide polymorphisms (SNPs)?

> <comment-title>Background</comment-title>
> Not everyone has the same background and that's ok! You may have studied different organisms with different gene models, or you may not be familiar with the biological aspect at all. The biological background is not necessary for following this tutorial, we just like to provide research context for any tutorial. Here are brief explanations of some of the concepts mentioned in this tutorial:
>
> - **Nucleotides** are the A, C, T, and Gs that make up DNA.
> - **Chromosomes** can be thought of as a very long piece of DNA (string of A, C, T, Gs) Some organisms have chromosomes, in this tutorial we will use Human chromosome number 22.
> - **Features** are regions of the chromosome that are interesting for one reason or another. Some examples of features include genes, terminators, transcription start sites, and repeat regions.
> - **Genes** are one kind of interesting feature, a region that will be [transcribed](https://en.wikipedia.org/wiki/Transcription_(biology)) into RNA before being [translated](https://en.wikipedia.org/wiki/Translation_(biology)) into proteins.
> - **Exons** are fundamental components of [eukaryotic](https://en.wikipedia.org/wiki/Eukaryote) [genes](https://en.wikipedia.org/wiki/Gene#Structure_and_function). A typical eukaryotic gene contains numerous *exons* separated by *introns*. An entire gene containing both exons and introns is *transcribed* into a pre-messenger RNA or pre-mRNA. During maturation introns are excised from the pre-mRNA in a process called *[splicing](https://www.nature.com/scitable/topicpage/rna-splicing-introns-exons-and-spliceosome-12375/)*.  A mature messenger RNA, or simply mRNA, is then *translated* into protein during the process of *translation*.
>
>
>   ![Eukaryotic gene model](../../images/DNA_exons_introns.png "An original piece of DNA containing introns and exons has the introns cut out before the exons are joined together to form the mRNA. (Image from <a href="https://commons.wikimedia.org/wiki/File:DNA_exons_introns.gif">WikiMedia</a>, under the Public Domain)")
>
> **SNP** is an abbreviation for [single-nucleotide polymorphism](https://en.wikipedia.org/wiki/Single-nucleotide_polymorphism). It is pronounced as "snip". SNPs are single nucleotide differences between a sequenced individual compared to some reference sequence; where one individual might have an A, another could have a C in that position. Databases of SNPs have been created for many organisms and they include any single nucleotide deviation from the reference sequence which appears in some percentage of the population (e.g. >1%). These are often especially interesting to geneticists as the causes of certain inheritable diseases.
{: .comment}

You may be familiar with the [UCSC Genome Browser](https://genome.ucsc.edu/) or another resource like it, and know that you can find the data there.
But even with your data in hand, you still have the question: "how do I actually compute this?" There is really a straightforward answer: **Galaxy**. So let's try it...

# Get your workspace ready

Browse to your favorite [Galaxy instance](https://galaxyproject.org/use/) and log in or register.

{% snippet faqs/galaxy/account_create.md %}

The Galaxy interface consists of three main parts:

1. The available **Tools** are listed on the left
2. The **Central Panel** will let you run analyses and view outputs
3. Your analysis **History** is recorded on the right


![Galaxy interface screenshot showing History panel on the right, tools panel on the left, and main panel in the center](../../images/galaxy_interface.png "<b>Galactic triptych</b>: the three panels of Galaxy interface: <em>Tools</em>, <em>Center Panel</em>, and <em>History</em>. Here you also see the 'Activity bar' that can be turned on and off (Don't see the Activity Bar? See a Tip below).")
{: .comment}


{% snippet faqs/galaxy/activity_bar_how_to_enable.md %}


When you start Galaxy for very first time, your history will be empty. Let's add some data to it.

> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. You can do this by clicking on the title of the history (which by default is *Unnamed history*) and typing **Galaxy 101** as the name. Do  not forget to hit the <kbd>Enter</kbd> key on your keyboard to save it.
>
>    ![Rename the history](../../../../shared/images/rename_history.svg)
>
{: .hands_on}

# Analysis

## Locate Exons

First we need to get some data into our history. You can upload files from your computer, or Galaxy can also fetch data directly from external sources. We know UCSC has exon locations for humans and we can use Galaxy to import the data for chromosome 22, directly from the UCSC table browser.

> <hands-on-title>Upload SNPs and Exons</hands-on-title>
> 1. At the top of the **Tools** panel (on the left), click {% icon galaxy-upload %} **Upload Data**
>
>    ![upload data button](../../images/upload-data.png)
>
>    This brings up the upload interface:
>
>    ![filebox](../../images/upload-box.png)
>
> 2. Click **Paste/Fetch data** and paste in the following URLs in the box that appears.
>
>    ```
>    https://zenodo.org/record/4104428/files/UCSC-hg38-chr22-Coding-Exons.bed
>    https://zenodo.org/record/4104428/files/UCSC-hg38-chr22-dbSNP153-Whole-Gene-SNPs.bed
>    ````
>
> 3. Click **Start**, and then **Close**
>
> 4. When they are ready, **rename** {% icon galaxy-pencil %} the datasets to `Exons` and `SNPs`, respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

For this tutorial, we made the input datasets available on Zenodo for you. However, these datasets can also obtained directly from UCSC, without leaving Galaxy.
Below we describe how you can do this, but it is not necessary for this tutorial. Note that since the data in UCSC is updated frequently, you might get slightly different results in the rest of this tutorial if you use these files.

> <comment-title></comment-title>
> In order to get the datasets from the UCSC server, you need to have an account in an instance.
{: .comment}


> <tip-title>Optional: Obtaining the Data directly from UCSC</tip-title>
>
> This tool works a bit differently than most Galaxy tools, but if you wish to obtain the newest data from UCSC, you can do that as follows:
>
> > <hands-on-title>Obtaining Exons from UCSC</hands-on-title>
> >
> > 1. **UCSC Main** {% icon tool %} table browser:
> >
> >     In the tool menu, navigate to `Get Data -> UCSC Main table browser`
> >
> >     ![`UCSC Main table browser` menu entry](../../images/101_01.png)
> >
> >     When you click on this tool, you will be taken to the **UCSC table browser**, which looks something like this:
> >
> >     ![`UCSC table browser` tool, first screen for exons](../../images/101_02.png)
> >
> >     Now set the following parameters:
> >     - *"clade"*: `Mammal`
> >     - *"genome"*: `Human`
> >     - *"assembly"*: `Dec. 2013 (GRCh38/hg38)`
> >     - *"group"*: `Genes and Gene Predictions`
> >     - *"track"*: `GENCODE v36` (or a more recent version)
> >     - *"table"*: `knownGeneV36`
> >     - {% icon param-text %} *"region"* should be changed to `position` with value `chr22`
> >     - *"output format"* should be changed to `BED - browser extensible data`
> >     - {% icon param-check %} *"Send output to"* should have the option `Galaxy` checked
> >
> >     > <comment-title></comment-title>
> >     > If the *"table"* drop down menu does not show the `knownGene` option. Set *"group"* to `All tables` and scroll down.
> >     {: .comment}
> >
> > 2. Click on the **Get output** button and you will see the next screen:
> >
> >    ![`UCSC table browser` tool, second screen for exons](../../images/101_03.png)
> >
> >    Change **Create one BED record per** to `Coding Exons` and then click on the **Send query to Galaxy** button.
> >
> >     > <comment-title></comment-title>
> >     > After this you will see your first history item in Galaxy's right panel. It will go through
> >     > the gray (preparing/queued) and yellow (running) states to become green (success):
> >     >
> >     > ![`UCSC Main on Human: knownGene` dataset is green](../../images/101_04.png)
> >     >
> >     > You might need to login to Galaxy again.
> >     {: .comment}
> >
> > 3. When the dataset is green, click on the {% icon galaxy-eye %} (eye) icon to **view the contents** of the file. It should look something like this:
> >
> >    ![Contents of the `UCSC Main on Human: knownGene` dataset](../../images/101_exons.png)
> >
> >    Each line represents an exon, the first three columns are the genomic location, and the fourth column contains the name of the exon.
> >
> > 4. Let's rename our dataset to something more recognizable.
> >    - Click on the {% icon galaxy-pencil %} **pencil icon** for the dataset to edit its attributes
> >    - In the central panel, change the **Name** field to `Exons`
> >    - Click the **Save** button
> >
> >    Your history should now look something like this:
> >
> >    ![Rename dataset to `Exons`](../../images/101_rename.png)
> {: .hands_on}
>
> We now have information about the exon locations, but our original question was which exon contains the largest number of SNPs, so let's get some information about SNP locations from UCSC as well:
>
> > <hands-on-title>Obtaining SNPs from UCSC</hands-on-title>
> > Again, search for **UCSC Main table browser** Browser on the tools search bar {% icon tool %} and set the following parameters
> >
> > 1. **UCSC Main** {% icon tool %} table browser:
> >    - *"group"* should be changed to `Variation`
> >    - {% icon param-text %} *"region"* should be changed again to `position` with value `chr22`
> >    - *"output format"* should be changed again to `BED - browser extensible data`
> >
> >    ![`UCSC table browser` tool, first screen for SNPs](../../images/101_06.png)
> >
> >    The *"track"* setting shows the version of the SNP database to get. In this example it is version 151, but you may select the latest one. Your results may vary slightly from the ones in this tutorial when you select a different version, but in general it is a good idea to select the latest version, as this will contain the most up-to-date SNP information.
> >
> > 2. Click on the **Get output** button to find a form similar to this:
> >
> >    ![`UCSC table browser` tool, second screen for SNPs](../../images/101_07.png)
> >
> >    Make sure that *"Create one BED record per"* is set to `Whole Gene` (Whole Gene here really means Whole Feature), and click on **Send query to Galaxy**. A second item will appear in your analysis history.
> >
> > 3. Now **rename** {% icon galaxy-pencil %} your new dataset to `SNPs` so we can easily remember what the file contains.
> >
> {: .hands_on}
>
{: .tip}

## Find exons with the most SNPs

Our objective is to find which exon contains the most SNPs. Therefore we have to intersect the file with the exon locations with the file containing the SNP locations (here "intersect" is just a fancy word for printing SNPs and exons that overlap side-by-side).

> <comment-title>Search bar</comment-title>
>
> Different Galaxy servers may have tools available under different sections, therefore it is often useful to use the **search bar** at the top of the tool panel to find your tool.
>
> Additionally different servers may have multiple, similarly named tools which accomplish similar functions. For these tutorials, you should select precisely the one that is described. However, in your real analyses, you'll need to search among the various options to find the one that works for you.
>
{: .comment}

> <hands-on-title>Finding Exons</hands-on-title>
>
> To find intersection we will be using `intersect intervals` {% tool [bedtools intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} from [BEDTools](https://bedtools.readthedocs.io/en/latest/content/overview.html) package.
>
> 1. {% tool [bedtools intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} find overlapping the intervals in various ways:
>
>    If you opened the tutorial on any usegalaxy.* instance, you can click on the tool name and will open the correct tool with correct version. Or, you can search for this tool yourself. Enter the word `intersect` in the search bar of the tool panel, and select the
>    tool named `bedtools Intersect intervals`
>
>    - *"File A to intersect with B"*: Select `Exons`
>    - *"Combined or separate output files"*: Select `One output file per 'input B' file"`
>    - *"File B to intersect with A"*: `SNPs`
>    - *"What should be written to the output file?"*: `Write the original entry in B for each overlap...`, which means that only matches are included in the output (i.e. only exons with SNPs in it and only SNPs that fall in exons)
>
>    The interface of the tool should look like this:
>    ![Bedtool intersect interval interface](../../images/101_bed_intersect_interface.png)
>
> 2. Click **Run Tool**.
> 3. Wait for the job to finish.
> 4. View the resulting file (with the {% icon galaxy-eye %} (eye) icon). If everything went okay, you should see a file that looks similar to this:
>
>    ![Contents of the `Join` output dataset](../../images/101_joined.png)
>
>    Here column 4 contains exon IDs (e.g., `ENST00000252835.5_cds_0_0_chr22_15528159_f`) and column 10 lists SNP IDs (e.g., `rs9617249`). Remember that variations are possible due to using different versions of UCSC databases: as long as you have similar looking columns you did everything right!
>    > <tip-title>Where do I find tool documentation?</tip-title>
>    > All Galaxy tools include documentation. If you scroll down on this page, you will find the help of the tool.
>    {: .tip}
>
>    > <comment-title>If things didn't work...</comment-title>
>    >
>    > Did the Intersect tool error with a memory failure? Or is this step executing for a long time? Most likely a setting was missed when extracting the data from the UCSC Table Browser. Try again, double checking that:
>    >
>    >  * For both SNP and EXON: *"region"* is actually changed to `position` with value `chr22`.
>    >  * For EXON: *"Create one BED record per"* `Coding Exons` is selected (*not* `Whole Gene` as for the SNP data).
>    >  * Carefully inspect the remaining Table Browser settings if these two most common reasons for problems were correct in your tool executions.
>    {: .comment}
>
{: .hands_on}

Let's take a look at this dataset. The first six columns correspond to the exons, and the last six columns correspond to the SNPs. Column 4 contains the exon IDs, and column 10 contains the SNP IDs. In our screenshot you see that the first lines in the file all have the same exon ID but different SNP IDs, meaning these lines represent different SNPs that all overlap the same exon.

> <question-title></question-title>
> For the first 3 exons in your file, what is the number of SNPs that fall into that exon?
>
> > <solution-title></solution-title>
> > At the time of writing, for hg38/GENCODE v29, joined with "Common dbSNPs(153)", using <kbd>ctrl+f</kbd> (<kbd>cmd+f</kbd> on Mac OS) to look for how many times each is used:
> >
> > Gene | Occurences
> > ---- | ----------
> > `ENST00000252835.5_cds_0_0_chr22_15528159_f` | 2
> > `ENST00000643195.1_cds_0_0_chr22_15528192_f` | 2
> > `ENST00000343518.11_cds_0_0_chr22_15690078_f` | 4
> {: .solution}
{: .question}


## Count the number of SNPs per exon

Since each line in our file represents a single overlap between SNP and exon, we can find the total number of SNPs in an exon, simply by counting the number of lines that have the same exon ID. However, to be more "proper" we will instead count the number of unique SNP IDs per exon. So let's do this for all the exons in our file:

> <hands-on-title>Counting SNPs</hands-on-title>
>
> 1. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.8+galaxy0) %} (operations on tabular data):
>
>    - *"Input tabular dataset"*: select the output dataset from **bedtools intersect intervals** {% icon tool %}
>    - *"Group by fields"*: `Column: 4` (the column with the exon IDs)
>
>    ![Datamash tool interface](../../images/101_datamash_1.png)
>
>    - Scroll tool interface down to *"Operation to perform on each group"*
>      - *"Type"*: `Count Unique values`
>      - *"On column"*: `Column: 10` (this column contains SNPs ids like `rs2236639`. This we will count occurrences of unique SNP ids for each exon)
>
>    ![Operation to perform on each group](../../images/101_datamash_2.png)
>
> 2. Click **Run Tool**. Your new output dataset will look something like this:
>
>    ![Contents of the `Group` output dataset](../../images/101_14.png)
>
{: .hands_on}

This file contains only two columns. The first contains the exon IDs, and the second the number of times that exon ID appeared in the file - in other words, how many SNPs were present in that exon.

> <question-title></question-title>
> How many exons are there in total in your file?
>
> > <solution-title></solution-title>
> > Each line now represents a different exon, so you can see the answer to this when you expand the history item, as in the image above. The exact number you see for your dataset may be slightly different due to the updates to the exon and SNPs information in UCSC. In our case the dataset contains 4,241 lines, which is equal to the number of exons overlapped by at least one SNP.
> >
> {: .solution }
{: .question}

## Sort the exons by SNPs count

Now that we have a list of all exons, and the number of SNPs they contain, we would like to know which exon has the *highest number* of SNPs. We can do this by sorting the file on the second column.

> <hands-on-title>Sorting</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1) %} data in ascending or descending order:
>
>    - *"Sort Query"*: Output from **Datamash** {% icon tool %}
>    - In *"Column selections"* set the following:
>       - *"on column"*: `Column: 2`
>       - *"in"*: `Descending order`
>       - *"Flavor"*: `Fast numeric sort`
>
> 2. Click **Run Tool**
>
> 3. Examine the output file.
>
>    ![Contents of the `Sort` output dataset](../../images/101_16.png)
>
>    The file should look similar to before sorting, but now the exons with the highest number of SNPs are at the top.
{: .hands_on}


> <question-title></question-title>
> Which exon has the highest number of SNPs in your file?
>
> > <solution-title></solution-title>
> > When this tutorial was last updated, `ENST00000253255.7_cds_0_0_chr22_46256561_r` had 27 SNPs.
> > Keep in mind this may depend on your settings when getting the data from UCSC.
> {: .solution}
{: .question}

## Select top five exons

Let's say we want a list with just the top-5 exons with highest number of SNPs.

> <hands-on-title>Select first</hands-on-title>
>
> 1. {% tool [Select first](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/9.5+galaxy0) %} lines from a dataset (head):
>
>    - *"File to select"*: The output from **Sort** {% icon tool %} (previous step of the analysis)
>    - "*Operation*": `Keep first lines`
>    - *"Number of lines"*: `5`
>
> 2. Click **Run Tool**.
>
> 3. Examine the output file, this should contain only the first 5 lines of the previous dataset.
>
{: .hands_on}

## Recover exon info

Congratulations! You have now determined which exons on chromosome 22 have the highest number of SNPs, but what else can we learn about them? One way to learn more about a genetic location is to view it in a genome browser. However, in the process of getting our answer, we have lost information about the location of these exons on the chromosome. But fear not, Galaxy saves all of your data, so we can recover this information quite easily.

> <hands-on-title>Compare two Datasets</hands-on-title>
>
> 1. {% tool [Compare two Datasets](comp1) %} to find common or distinct rows:
>
>    - *"Compare"*: `Exons`
>    - *"Using column"*: `Column: 4`
>    - *"against"*: the output from **Select first** {% icon tool %} (previous step of the analysis)
>    - *"and column"*: `Column: 1`
>    - *"to find"*: `Matching rows of 1st dataset`
>
> 2. Click **Run Tool**
> 3. Examine your output file. It should contain the locations of your top 5 exons:
>
>    ![Contents of the `Compare two Datasets` output dataset](../../images/101_19.png)
{: .hands_on}

## Display data in UCSC genome browser

A good way to learn about these exons is to look at their genomic surrounding. This can be done by using genome browsers. Galaxy can launch a genome browser such as IGV on your local machine, and it can connect to online genome browsers as well. An example of such an online genome browser is the UCSC genome browser.

> <hands-on-title>UCSC genome browser</hands-on-title>
>
> 1. First, check that the **database** of your latest history dataset is `hg38`. If not, click on the {% icon galaxy-pencil %} pencil icon and modify the **Database/Build:** field to `Human Dec. 2013 (GRCh38/hg38) (hg38)`.
>
>    {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="Human Dec. 2013 (GRCh38/hg38) (hg38)" %}
>
> 2. Second, check that the **format** of your latest history dataset is `bed`. If not, click on the {% icon galaxy-pencil %} pencil icon and modify the **Datatype** field to `bed`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
> 3. Click on the dataset in your history to expand it, then click on the {% icon galaxy-barchart %} (**Visualize**) icon
>
>    ![visualize button on the expanded history item](../../images/101_ucsc_visualize.png)
>
> 4. To **visualize the data in UCSC genome browser**, click on `display at UCSC (main)` now visible at the top (blue box) of the Visualize menu.
>
>    ![`display at UCSC main` link](../../images/101_displayucsc.png)
>
>    This will upload the data to UCSC as custom track. To see your data look at the `User Track` near the top. You can enter the coordinates of one of your exons at the top to jump to that location.
>
>    ![`User Track` shown in the UCSC genome browser](../../images/101_21.png)
{: .hands_on}

UCSC provides a large number of tracks that can help you get a sense of your genomic area, it contains common SNPs, repeats, genes, and much more (scroll down to find all possible tracks).

# Histories and workflows: A brief introduction

In Galaxy your analyses live in histories such as your current one. Histories can be very large, and you can have as many histories as you want. You can control your histories (switching, copying, sharing, creating a fresh history, etc.) in the {% icon galaxy-history-options %} **History Options** menu on the top of the history panel:

![History options menu](../../images/history_options_menu.png "'History options' allows for a variety of history operations")

## List your histories

You can create as many histories as you want. If you create a new history, your current history does not disappear. You can view your histories in two ways: (1) as a list or (2) side by side (also known as "Multiview"). Two Tip boxes below explain both of these approaches:

{% snippet faqs/galaxy/histories_list.md %}

{% snippet faqs/galaxy/histories_side_by_side_view.md %}


![Overview of histories with their datasets](../../images/histories_side_by_side_view.png "Histories side-by-side view: in this view you drag datasets between histories, switch between histories, create new histories etc.")

You can always return to your analysis view by clicking on Galaxy Icon in the top menu bar.

## Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step. But when you receive new data, or a new report is requested, it would be tedious to do each step over again. Wouldn't it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the `Extract workflow` option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking on the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click on "**History options**" dropdown at the top of your history panel and select **Extract workflow**.
>
>    ![`Extract Workflow` entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
>    ![Selection of steps for `Extract Workflow` from history](../../images/101_25.png)
>
> 3. Replace the **Workflow name** to something more descriptive, for example `Find exons with the highest number of features`.
>
>    While we created this workflow initially to analyse SNPs, if we had similarly formatted datasets we could use this workflow to find different features.
>
> 4. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes.
>
> 5. Click on the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created. But where did it go?
>
> 6. Click on **Workflows** in the left menu of Galaxy (Activity Bar). Here you have a list of all your workflows. Your newly created workflow should be listed at the top:
>
>    ![`Your workflows` list](../../images/workflow_cards.png "Workflow are listed in the center pane of the Galaxy interface as cards.")
{: .hands_on}

## The workflow editor

We can examine the workflow in Galaxy's workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. Click on the triangle to the right of your workflow name.
>
>    ![`Edit` option in the menu for the latest workflow](../../images/open_wf_editor.png "To open the interactive workflow editor click the 'Edit' button.")
>
> 2. Select **Edit** to launch the workflow editor. You should see something like this:
>
>    ![Workflow editor](../../images/wf_editor_open.png "Workflow editor interface. It can be used for creation and editing of workflows of any complexity.")
>
>    When you click on a workflow step, you will get a view of all the parameter settings for that tool on the right-hand side of your screen.
>
> 3. Re-arrange the boxes so you can clearly see the data flow. The default automatic layout hides some of the connections due to overlapping and box placement.
>
> 4. Make sure the  **check boxes** for `out_file1` in the `Select First` and `Compare two Datasets` tools are selected. Make sure that everything else is not selected.
>
>    Now, when we run the workflow, we will only see the final two outputs, i.e. the table with the top-5 exons and their SNP counts, and the file with exons ready for viewing in a genome browser.
>
> 5. The box named `Exons` is named ok, but we want to change `SNPs` since this workflow is not specific to SNPs
>
>    - **Click** on the box corresponding to the `SNPs` input dataset
>    - change the **Label** to `Features` on the right-hand side of your screen.
>
> 6. Let's also **rename the outputs**:
>
>    - Click on the `Select first` tool in the workflow editor
>    - In the menu on the right click on `Configure Output: 'out_file1'`
>        - Under `Rename dataset`, and enter a descriptive name for the output dataset like `Top 5 exon IDs`
>
>    ![Rename the output of the `Select first` tool step](../../images/configure_wf_output.png "The datasets generated by the workflow execution can be automatically renamed using  the 'Rename dataset' option")
>
> 7. **Repeat** this for the output of the `Compare two Datasets` tool, naming it `Top 5 exons`
>
> 8. **Save your workflow** (important!) by clicking on the {% icon galaxy-save %} icon at the top right of the screen.
>
> 9. **Return** to the analysis view by clicking on the Galaxy Icon at the top left of the menu bar.
>
{: .hands_on}

> <comment-title></comment-title>
> We could **validate** our newly built workflow by running it on the same input datasets that we used at the start of this tutorial, in order to make sure we do obtain the same results.
{: .comment}

## Run workflow on different data

Now that we have built our workflow, let's use it on some different data. For example, let's find out which exons have the highest number of repeat elements.

> <hands-on-title>Run workflow</hands-on-title>
>
> 1. Create a **new history** and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. We will need the list of exons again. We don't have to get this from UCSC again, we can just **copy** it from our previous history. To do this:
>
>    - Bring up a list of all you histories:
>
>    {% snippet faqs/galaxy/histories_list.md %}
>
>    - Display the "Galaxy 101" history in the Center pane by clicking "Open in Center pane" button:
>    ![Opening a history in the center pane](../../images/open_in_center_pane_button.png)
>
>    - Drag and drop `Exons` dataset into the empty history created at step 1
>    ![Drag and drop of `Exons` dataset in the history overview](../../images/drag_n_drop_dataset.gif)
> 3. Click on Galaxy at the left to return to the main analysis window
>
> 4. {% tool [Upload Data](upload1) %} the Repeats file from [Zenodo](https://doi.org/10.5281/zenodo.4104428)
>
>    ```
>    https://zenodo.org/record/4104428/files/UCSC-hg38-chr22-Repeats.bed
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    > <tip-title>Obtaining the Data from UCSC</tip-title>
>    >
>    > Again, for reproducibility we obtain the data from Zenodo ensuring that the results will never change, allowing us to do good science! However if you wish to obtain UCSC data:
>    >
>    > > <hands-on-title>Obtaining Exons from UCSC</hands-on-title>
>    > >
>    > > 1. **UCSC Main** {% icon tool %} table browser:
>    > >
>    > >     In the tool menu, navigate to `Get Data -> UCSC Main - table browser`
>    > >
>    > >     Now set the following parameters:
>    > >     - *"clade"*: `Mammal`
>    > >     - *"genome"*: `Human`
>    > >     - *"assembly"*: `Dec. 2013 (GRCh38/hg38)`
>    > >     - *"group"*: `Repeats`
>    > >     - {% icon param-text %} *"region"* should be changed to `position` with value `chr22`
>    > >     - *"output format"* should be changed to `BED - browser extensible data`
>    > >     - {% icon param-check %} *"Send output to"* should have the option `Galaxy` checked
>    > >
>    > >   Click on **get output** and then **Send query to Galaxy** on the next screen.
>    > {: .hands_on}
>    {: .tip}
>
> 5. **Rename** {% icon galaxy-pencil %} the dataset to `Repeats`
>
> 6. Open the **workflow menu** (top menu bar). Find the workflow you made in the previous section, and select the option `Run`.
>
>    ![`Run` option in the workflow menu](../../images/run_workflow_button.png)
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
>    The central panel will change to allow you to configure and launch the workflow.
>
> 7. Select appropriate datasets for the inputs as shown below, then scroll down and click `Run workflow`.
>    - Select `Expand to full workflow form`
>    - {% icon param-file %} *"Exons"*: the `Exons` file you copied from our previous history
>    - {% icon param-file %} *"Features"*: the `Repeats` file we downloaded from UCSC
>
>    ![Settings for running the workflow](../../images/workflow_setting_inputs.png)
>
>    > <comment-title>Potential workflow issues</comment-title>
>    >
>    >  * Galaxy validates the workflow inputs to ensure they're correct. It may show a validation error at the start, until you select `Exons` for the Exons input, and your repeats for the Features input.
>    >  * If you see an "Invalid column choice" error, you need to specify which column you want to use. If you have to type the column number, you need to type just the number e.g. `4` (not `Column 4` or anything else).
>    {: .comment}
>
> 8. Once the workflow has started, you will initially be able to see all its steps, but the unimportant intermediates may disappear after they complete successfully (based on the workflow settings):
>
>    ![Datasets appearing in the history](../../images/workflow_running.png)
{: .hands_on}

> <comment-title>Unhiding hidden datasets</comment-title>
> Because most intermediate steps of the workflow could be hidden, once it is finished you may only see the final two datasets. If we want to view the intermediate files after all, you can click the "## hidden" just below the history's name.
{: .comment}

> <question-title></question-title>
> Which exon had the highest number of repeats? How many repeats were there?
{: .question}

# Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

{% snippet faqs/galaxy/histories_sharing.md %}

> <hands-on-title>Share history and workflow</hands-on-title>
>
> 1. Share one of your histories with your neighbour.
> 2. See if you can do the same with your workflow!
> 3. Find the history and/or workflow shared by your neighbour. Histories shared with specific users can be accessed by those users in their {% icon galaxy-gear %} history menu under `Histories shared with me`.
{: .hands_on}

# Conclusion


{% icon trophy %} Well done! You have just performed your first analysis in Galaxy. You also created a workflow from your analysis so you can easily repeat the exact same analysis on other datasets. Additionally you shared your results and methods with others.


topic: introduction
tutorial: galaxy-reproduce
title: How to reproduce published Galaxy analyses
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 5af8ea90b32ca1168c9019c9dcfd427b
last_modified: 2026-02-26T21:12:17.041352
zenodo_link: https://zenodo.org/record/1319069
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-reproduce/tutorial.html

content:
This training will demonstrate how to reproduce analyses performed in the Galaxy framework. Before we start with the hands-on part, we would like to give you some information about Galaxy.

Galaxy is a scientific workflow, data integration and data analysis and publishing platform. Galaxy is an open-source platform for accessible, reproducible, and transparent computational research. While Galaxy was started to allow non-bioinformaticians to analyze DNA sequencing data, it nowadays enables analysis tasks of many different domains including machine learning, ecology, climate science and  omics-type of analyses. Galaxy is easy to use because it is accessible via a web-browser and provides a graphical user interface which enables access to pre-installed tools and large computational resources. In Galaxy, all analyses are stored in so-called histories. The history keeps track of all the tools, tool versions and parameters that were used in the analysis. From such a history, a workflow can be extracted; this workflow can be used to easily repeat the analysis on different data. Both, histories and workflows, can either be shared privately with colleagues or publicly, for example as part of a published manuscript.

For more background information about Galaxy, have a look into the Galaxy publication ({% cite Afgan_2018 %}). In depth technical details about technologies that enable reproducible analyses within Galaxy are described in {% cite Gr_ning_2018 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# What does Galaxy look like?

{% assign servers = nil | list_usegalaxy_servers_shuffle %}
Many different Galaxy servers exist. Some are public, some are private, some focus on a specific topic and others like the usegalaxy.\* servers cover a broad range of tools. To reproduce published results it is highly recommended to use the same Galaxy server that was used in the original study. In the case that this was a private server that is not accessible to you, you might want to use one of the main Galaxy servers: {% for server in servers %}[{{ server.name }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %}. To learn more about the different Galaxy servers visit the [slides: options for using Galaxy]({% link /topics/introduction/tutorials/options-for-using-galaxy/slides.html %}). The particular Galaxy server that you are using may look slightly different than the one shown in this training. Galaxy instance administrators can choose the exact version of Galaxy they would like to offer and can customize its look to some extent. The basic functionality will be rather similar across instances, so don’t worry! In this training we will use the European Galaxy server on which the original analysis was performed and shared.


> <hands-on-title>Log in or register</hands-on-title>
> 1. Open your favorite browser (Chrome/Chromium, Safari or Firefox, but not Internet Explorer/Edge!)
> 2. Browse to the [Galaxy Europe instance](https://usegalaxy.eu/) or to a Galaxy instance of your choosing
> 3. Choose *Login or Register* from the navigation bar at the top of the page
> 4. If you have previously registered an account with this particular instance of Galaxy (user accounts are *not* shared between public servers!), proceed by logging in with your registered *public name*, or email address, and your password.
>
>    If you need to create a new account, click on *Register here* instead.
>
{: .hands_on}


The Galaxy interface consists of three main parts:

1. The available tools are listed on the left
2. Your analysis history is recorded on the right
3. The central panel will let you run analyses and view outputs

![Galaxy interface with tools panel on left, main panel in center, and history panel on the right.]({% link shared/images/galaxy_interface.png %})


# Create a history and load data into it

Each analysis in Galaxy starts by creating a new analysis history and loading data into it. Galaxy supports a huge variety of data types and data sources. Different ways of bringing data into Galaxy are explained [in the interface slides]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}). To reproduce published results, the data needs to be loaded from the public repository where the authors have deposited the data. This is most often done by importing data via a web link.


> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. For instance, you can choose **Reproduction of published Galaxy results** as the name of your new history.
>
>    {% snippet faqs/galaxy/histories_rename.md name="Reproduction of published Galaxy results" %}
>
{: .hands_on}


> <comment-title>Background about the dataset</comment-title>
> The Iris flower data set, also known as Fisher’s or Anderson's Iris data set, is a multivariate dataset introduced by the British statistician and biologist Ronald Fisher in his 1936 paper ({% cite Fisher1936 %}).
> Each row of the table represents an iris flower sample, describing its species and the dimensions in centimeters of its botanical parts, the sepals and petals.
> You can find more detailed information about this dataset on its dedicated [Wikipedia page](https://en.wikipedia.org/wiki/Iris_flower_data_set).
{: .comment}


> <hands-on-title>Data upload</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `iris.csv` from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1319069/files/iris.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Check the **datatype**
>    - Click on the history item to expand it to get more information.
>    - The datatype of the iris dataset should be `csv`.
>    - **Change** {% icon galaxy-pencil %} the datatype *if* it is different than `csv`.
>      - Option 1: Datatypes can be **autodetected**
>      - Option 2: Datatypes can be **manually set**
>
>    {% snippet faqs/galaxy/datasets_detect_datatype.md datatype="datatypes" %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
>
> 4. Add an `#iris` tag {% icon galaxy-tags %} to the dataset
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    Make sure the tag starts with a hash symbol (`#`), which will make the tag stick not only to this dataset, but also to any results derived from it.
>    This will help you make sense of your history.
>
{: .hands_on}

> <details-title>Working with different types of datasets</details-title>
> Some input datasets might need more specialized treatment than explained here.
> A few data types contain more than one subfile. These are uploaded via the composite data function, which is a new tab on the right of regular upload. Then at the bottom set "composite type" to your file format. For each subfile a select box will appear with a description next to it, about which subfile has to be selected where.
> Some workflows require input files as dataset collections, in such cases "Input dataset collection" are shown as input when editing or viewing the workflow in the workflow menu. Collections contain several single dataset of the same type tied together. In case a workflow input requires a collection, you’ll need to build a collection out of your files after uploading them.
> [A specialized training explains how to use collections]({{site.baseurl}}/topics/galaxy-interface/tutorials/collections/tutorial.html).
>
> {% snippet faqs/galaxy/collections_build_list.md %}
>
> In case you want to run a published Galaxy workflow on your own data, you can find explanations about the options to upload your own data [in the interface slides]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}).
{: .details}


# Import and run a Galaxy workflow

Galaxy workflows may be published either directly via the Galaxy server or on public workflow repositories such as [WorkflowHub](https://workflowhub.eu/). Thus the workflow may be present in one of the three ways:

1. As a .ga file or url link, which needs to be imported into Galaxy
2. As a link from a personal Galaxy server account that needs to be added to the own Galaxy account
3. as a link that directly starts running the workflow in a specific Galaxy server, which is possible via the WorkflowHub website.

This tutorial follows option 1, but options 2 and 3 are no more difficult

> <tip-title>How to run workflows via options 2 and 3</tip-title>
> This is not part of the training, but information in case you received a workflow of interest via way 2) or 3).
>
> **Link from a personal Galaxy**:
> In case you received a link from a personal Galaxy user account
> 1. you need to log into exactly the same Galaxy server from where the workflow link is shared, which should be clear from the start of the link, e.g. "https://usegalaxy.eu/..."
> 2. Click on the link and on the upper right on to the plus symbol (import workflow)
> 3. Continue with Step 2 of the following hands-on box
>
> **Link from WorkflowHub**:
> In WorkflowHub there is an option to directly run a workflow.
> 1. Make sure that you prepared your input data on the same server as specified in the run button.
> 2. Click on "Run on usegalaxy.eu"
> 3. select inputs
> 4. "Run workflow"
{: .tip}

> <hands-on-title>Import and run workflow available as .ga file or link</hands-on-title>
>
> 1. **Import** the workflow either via url directly from [Zenodo](https://doi.org/10.5281/zenodo.5090049) or by uploading the .ga file
>
>    ```
>    https://zenodo.org/record/5090049/files/main_workflow.ga
>    ```
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Start** the workflow by clicking on the run symbol on the last column in the workflow overview list
>
> 3. Select the `iris` dataset as the input dataset.
>
> 4. **Run** the workflow by clicking on `run workflow`
>
>    > <question-title></question-title>
>    >
>    > How many history items do you have after running the workflow?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 12, out of which 4 are shown and 8 hidden (at top of history right ander the history name)
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <comment-title>Hidden datasets in history</comment-title>
> Some workflow outputs might be considered as not very important intermediate results and are thus marked as getting hidden in the analysis history after they turned green. This makes the history easier to navigate through the main results which are visible in the history. Hidden datasets can be made visible individually by clicking on the eye with a slash on top of the history and then clicking "Unhide" for the individual datasets. To unhide many dataset at once, click {% icon galaxy-selector %} "Select Items" at the top left of the history; then select all hidden datasets that you would like to unhide, then click "For n of N selected" and then "Unhide".
{: .comment}

By starting the workflow all jobs are sent to the Galaxy cluster for analysis. Sometimes it can take a bit until the datasets show up in your history. The jobs are processed one after the other or in parallel if the same input is used for several steps. Grey means waiting to run, yellow means running and green means finished. Red means there was an error.

{% snippet faqs/galaxy/analysis_troubleshooting.md %}

> <comment-title>The tool stays grey</comment-title>
> This scenario will likely not happen with this training analysis, but might happen with a real workflow. The tool runs are scheduled on the computing infrastructures according to their consumption of cores and memory. Thus, tools that are assigned lots of cores and/or memory need to wait until an appropriate computing spot is available. Several Galaxy server display the current computational load which gives you an idea how busy it currently is.
{: .comment}

Each dataset that turns green can already be inspected even though later datasets are still running. The second part of the training will focus on how to inspect datasets in a history.


# Inspecting the analysis history

Each history item represents one file or dataset, except when collections are used. History items are numbered, duplicates are not possible because any type of dataset manipulation will automatically generate a separate dataset in the history. Some tools produce several output files and therefore the number of history items can be larger than the number of steps in a workflow. Dataset names in the analysis history are not relevant for the tool run, therefore they can be adjusted in order to make the history easier to understand. The default name of a history item is composed of the tool name that was run and the history item number(s) of the input file(s), e.g. 'Unique on data 5'

> <hands-on-title>Inspect history</hands-on-title>
>
> 1. **Vizualize** the two scatter plots by clicking on their eye icons (view data)
>
>    {% snippet faqs/galaxy/features_scratchbook.md %}
>
> 2. **Show** all datasets by clicking on `hidden` on top of your history, right below the history name
>
> 3. **Compare** the `Convert csv to tabular` file with the `Datamash` file side by side using the scratchbook
>
> 4. **Track** how the `Datamash` results where obtained by clicking on the `Datamash` item in the history and then on its `i` icon (view details). The performed operations can be found in the section `Tool parameters`
>
>    > <question-title></question-title>
>    >
>    > 1. What are the different Iris species?
>    > 2. How many lines has the `Convert csv to tabular` file?
>    > 3. Which column was grouped in during the Datamash operation?
>    > 4. Which column of the `Remove beginning` file contains sepal length and which petal length?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1.  The 3 different Iris species are:
>    > >     - setosa
>    > >     - versicolor
>    > >     - virginica
>    > >
>    > > 2. 151 lines (by clicking on the file one can see the line count under its name)
>    > >
>    > > 3. Column 5 (details of Datamash tool: Group by fields - 5)
>    > >
>    > > 4. Column 1 and 3 (the dataset was generated by removing the header line from data 2, thus the content of the columns is the same as in data file 2)
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

If you got the same answers as written in the above solution box, then congratulations! You have imported and fully reproduced the results of a previous analysis.

However, sometimes you may wish to do more, like...

# Manipulating the analysis

Maybe you are interested in changing some of the original tool parameters and see how this changes the result. The parameter changes can be either done inside the workflow editor, which makes it easy to change many parameters at once (training on [Using Workflow Parameters]({% link topics/galaxy-interface/tutorials/workflow-parameters/tutorial.md %})) or directly in the history. To repeat a single analysis step with new parameters this can be done directly in the Galaxy history with the `re-run` option.

> <hands-on-title>Manipulate the analysis steps</hands-on-title>
>
> 1. **Rerun** the Scatterplot to plot Sepal length vs. Petal length
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
>   - *"Column to plot on x-axis"*: `1`
>   - *"Column to plot on y-axis"*: `3`
>
{: .hands_on}


# Additional Exercise: Import a published analysis history and explore it

Often not only workflows and raw data are published, but also the full Galaxy histories. These histories can either be inspected via their provided link, or imported in order to enable manipulating them in your own Galaxy account.

> <hands-on-title>Importing a history</hands-on-title>
>
>
> 1. {% tool [Import](upload1) %} a published history shared via an EU Server account
>
>    ```
>    https://usegalaxy.eu/u/annefou/h/galaxy101-for-everyone-diamond-dataset
>    ```
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
{: .hands_on}

The `diamonds` dataset comes from the well-known [ggplot2](https://ggplot2.tidyverse.org/reference/diamonds.html) package developed by Hadley Wickham and was initially collected from the Diamond Search Engine in 2008.
The original dataset consists of 53940 specimen of diamonds, for which it lists the prices and various properties.
For this training, we have created a simpler dataset from the original, in which only the five columns relating to the price and the so-called 4 Cs (carat, cut, color and clarity) of diamond characteristics have been retained. The same workflow as before was used to re-analysis this second dataset and create the analysis history, which highlights the re-usability of workflows across inputs.

> <comment-title>The 4 Cs of diamond grading</comment-title>
> According to the [GIA's (Gemological Institute of America) diamond grading system](https://4cs.gia.edu/wp-content/uploads/2013/03/All-Scales.jpg)
> - **Carat** refers to the weight of the diamond when measured on a scale
> - **Cut** refers to the quality of the cut and can take the grades *Fair*, *Good*, *Very Good*, *Premium* and *Ideal*
> - **Color** describes the overall tint, or lack thereof, of the diamond from colorless/white to yellow and is given on a letter scale ranging from D to Z (D being the best, known as colorless).
> - **Clarity** describes the amount and location of naturally occuring "inclusions" found in nearly all diamonds on a scale of eleven grades ranging from *Flawless* (the ideal situation) to *I3* (Included level 3, the worst quality).
{:.comment}

> <question-title></question-title>
>
> 1. What are the different Cut categories?
> 2. How many lines has the `diamonds.csv` file?
> 3. Is the color an important factor for the Diamond price?
>
> > <solution-title></solution-title>
> >
> > 1.  The 5 different Cut categories are:
> >     - Fair
> >     - Good
> >     - Ideal
> >     - Premium
> >     - Very Good
> >
> > 2. 53940 lines (by clicking on the file one can see the line count under its name)
> >
> > 3. We can create a new scatter plot and use color as a factor (Advanced, column differentiating the different groups: 3). Then, holding carat weight constant, we see on
> > the scatter plot that color is linked to the price of the diamond. So color also explains a lot of the variance found in price!
> > ![price-diamond-color-dependence](../../images/reproduce-diamond-colour.png)
> >
> {: .solution}
{: .question}

# Conclusion


{% icon trophy %} Well done! You have just reproduced your first analysis in Galaxy.


topic: introduction
tutorial: galaxy-intro-101-everyone
title: Galaxy Basics for everyone
difficulty: introductory
time_estimation: 1H30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 4dfccae356f990b5f07c9728c4eb82a9
last_modified: 2026-02-26T21:12:17.031147
zenodo_link: https://zenodo.org/record/1319069/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.html

content:
This practical aims at familiarizing you with the Galaxy user interface.
It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows and sharing your work.
Not everyone has the same background and that's ok!

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
> The Iris flower data set, also known as Fisher’s or Anderson's Iris data set, is a multivariate dataset introduced by the British statistician and biologist Ronald Fisher in his 1936 paper ({% cite Fisher1936 %}).
> Each row of the table represents an iris flower sample, describing its species and the dimensions in centimeters of its botanical parts, the sepals and petals.
> You can find more detailed information about this dataset on its dedicated [Wikipedia page](https://en.wikipedia.org/wiki/Iris_flower_data_set).
{: .comment}

## What does Galaxy look like?

> <hands-on-title>Log in or register</hands-on-title>
> 1. Open your favorite browser (Chrome/Chromium, Safari, or Firefox, but not Internet Explorer/Edge!)
> 2. Browse to a [Galaxy instance](https://galaxyproject.org/use/) of your choice
> 3. Choose *Login or Register* from the navigation bar at the top of the page
> 4. If you have previously registered an account with this particular instance of Galaxy (user accounts are *not* shared between public servers!), proceed by logging in with your registered *public name*, or email address, and your password.
>
>    If you need to create a new account, click on *Register here* instead.
>
>   > <comment-title>Different Galaxy servers</comment-title>
>   > The particular Galaxy server that you are using may look slightly different than the one shown in this training.
>   > Galaxy instance administrators can choose the exact version of Galaxy they would like to offer and can customize its look and feel to some extent.
>   > The basic functionality will be rather similar across instances, so don't worry!
>   {: .comment}
>
{: .hands_on}


The Galaxy interface is divided into four sections (panels):
* The Activity Bar on the left: _This is where you will navigate to the resources in Galaxy (Tools, Workflows, Histories etc.)_
* Currently active "Activity Panel" on the left: _By default, the {% icon tool %} **Tools** activity will be active and its panel will be expanded_
* Viewing panel in the middle: _The main area for context for your analysis_
* History of analysis and files on the right: _Shows your "current" history; i.e.: Where any new files for your analysis will be stored_

![Screenshot of the Galaxy interface with aforementioned structure](../../images/galaxy_interface.png)

The first time you use Galaxy, there will be no files in your history panel.




# Create a history

Galaxy allows you to create analysis histories. A **history** can be thought of as an electronic experimental lab book; it keeps track of all the tools and parameters you used in your analysis. From such a history, a **workflow** can be extracted; this workflow can be used to easily repeat the analysis on different data.

Think of a workflow as a cooking recipe with a list of ingredients (datasets) and a set of instructions
(pipeline of operations) that describes how to prepare or make something (such as a plot, or a new dataset).
The order of operations is important as very often the next operation takes as input the result of the previous operations. For instance, when baking
a cake, you would first sift the flour and then mix it with eggs as it would be impossible to sift the flour afterward.
That is what we call a pipeline. To make a full meal, we may need to combine multiple recipes (pipelines) together.

The finalized pipelines can be generalized as a workflow. If we use cooking as an analogy, a workflow could represent an entire menu with all the recipes for each meal.
In other words, using a workflow makes it possible to apply the same procedure to a different dataset, just by changing the input.

> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. For instance, you can choose **Galaxy 101 for everyone** as the name of your new history.
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Upload the Iris dataset

> <hands-on-title>Data upload</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `iris.csv` from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1319069/files/iris.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Check the **datatype**
>    - Click on the history item to expand it to get more information.
>    - The datatype of the iris dataset should be `csv`.
>    - **Change** {% icon galaxy-pencil %} the datatype *if* it is different than `csv`.
>      - Option 1: Datatypes can be **autodetected**
>      - Option 2: Datatypes can be **manually set**
>
>    {% snippet faqs/galaxy/datasets_detect_datatype.md datatype="datatypes" %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
>
> 4. Add an `#iris` tag {% icon galaxy-tags %} to the dataset
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    Make sure the tag starts with a hash symbol (`#`), which will make the tag stick not only to this dataset, but also to any results derived from it.
>    This will help you make sense of your history.
>
{: .hands_on}

# Pre-processing

Often, one or more data pre-processing step(s) may be required to proceed with the analysis.
In our case, the tools we will use require tab-separated input data and assume there is no header line. Since our data is comma-separated and has a header line, we will have to perform the following pre-processing steps to prepare it for the actual analysis:

- Format conversion
- Header removal


## Convert format

First, we will convert the file from comma-separated to tab-separated format. Galaxy has built-in format converters we can use for this.


> <hands-on-title>Converting dataset format</hands-on-title>
>
> 1. **Convert** {% icon galaxy-pencil %} the CSV file (comma-separated values) to tabular format (tsv; tab-separated values)
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="tabular (using csv-to-tabular)" %}
>
> 2. **Rename** {% icon galaxy-pencil %} the resulting dataset to `iris tabular`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. **View** the generated file by clicking on the {% icon galaxy-eye %} (eye) icon
>
>    > <question-title></question-title>
>    >
>    > How many header lines does our file have?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file has one header line, it contains the column names.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Remove header

Now it is time to run your first tool! We saw in the previous step that our file has 1 header line. This line does not contain any data, but the names of each column. We will now remove that line from our file before moving on to our analysis.

> <comment-title>Tip: Finding your tool</comment-title>
>
> Different Galaxy servers may have tools available under different sections, therefore it is often useful to use the **search bar** at the top of the tool panel to find your tool.
>
> Additionally different servers may have multiple, similarly named tools which accomplish similar functions. When following tutorials, you should use precisely the tools that they describe. For real analyses, however, you will need to search among the various options to find the one that works for you.
>
{: .comment}

> <hands-on-title>Removing header</hands-on-title>
>
> 1. {% tool [Remove Beginning](Remove beginning1) %} with the following parameters:
>    - *Remove first*: `1` (to remove the first line only)
>    - {% icon param-file %} *"from"*: select the **iris tabular** file from your history
>    - Click **Run Tool**
>
>    > <comment-title>Tip: search for the tool</comment-title>
>    >
>    > Use the **tools search box** at the top of the tool panel to find **Remove beginning** {% icon tool %}.
>    {: .comment}
>
>    ![Settings for the `Remove beginning` tool](../../images/101_foreveryone_remove_beginning.png)
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris clean`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Click on the new history item to expand it
>
>    > <question-title></question-title>
>    >
>    > 1. Which tags are present on this resulting dataset? (You may have to refresh the history panel to see the tags)
>    > 2. How many samples (lines) does our dataset contain?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The output of **Remove beginning** {% icon tool %} is also tagged with the label `iris`. Tags beginning with a hashtag (`#`) will propagate; they will appear on any datasets derived from your original tagged file.
>    > >
>    > > 2. There are 150 lines in our file (we can see this under the file name when we have expanded the history item). This means we have 150 samples.
>    > {: .solution}
>    {: .question}
>
> 4. **View** {% icon galaxy-eye%} the contents of the resulting file.
>    - You should see that the header line is now no longer present.
>
{: .hands_on}



# Data Analysis: What does the dataset contain?

Now we are going to inspect the dataset using simple tools in order to get used to the Galaxy interface and answer basic questions.

## How many different species are in the dataset?

In order to answer this question, we will have to look at column 5 of our file, and count how many different values (species) appear there. There are several ways we could do this in Galaxy. One approach might be to first extract this column from the file, and then count how many unique lines the file contains. Let's do it!

> <hands-on-title>Extract species</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} columns from a table with the following parameters:
>      - *"Cut columns"*: `c5`
>      - *"Delimited by"*: `Tab`
>      - {% icon param-file %} *"From"*: `iris clean` dataset
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris species column`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. **View** {% icon galaxy-eye %} the resulting file
>
> 4. {% tool [Unique](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0) %} occurrences of each record  with the following parameters:
>      - {% icon param-file %} *"File to scan for unique values"*: `iris species column` (the output from **Cut** {% icon tool %})
>
> 5. **Rename** {% icon galaxy-pencil %} the dataset to `iris species`
>
> 6. **View** {% icon galaxy-eye %} the resulting file
>
>    > <question-title></question-title>
>    >
>    > 1. How many different species are in the dataset?
>    > 2. What are the different Iris species?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 3 species.
>    > > 2. The 3 different Iris species are:
>    > >     - setosa
>    > >     - versicolor
>    > >     - virginica
>    > {: .solution}
>    {: .question}
{: .hands_on}

Now we have our answer! There are 3 different Iris species in our file.

Like we mentioned before, there are often multiple ways to reach your answer in Galaxy. For example, we could have done this with just a single tool, **Group** {% icon tool %} as well.
(This tool also can be searched for by the term "grouping" )


> <hands-on-title>Exercise: Grouping dataset</hands-on-title>
>
> 1. Try answering this question (how many Iris species are in the file?) again, using a different approach:
>    - Tool: **Group** data by a column and perform aggregate operation on other columns {% icon tool %}
>    - Input dataset: `iris clean` dataset to answer the same question.
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris species group`
>
> 3. Did you get the same answer as before?
>
> > <solution-title></solution-title>
> > 1. {% tool [Group](Grouping1) %}   with the following parameters:
> >   - *"Select data"* select `iris clean` dataset
> >   - *"Group by column"*: `Column: 5`
> >
> > 2. This approach should give the same answer. There are often multiple ways to do a task in Galaxy, which way you choose is up to you!
> {: .solution}
>
{: .hands_on}


## How many samples by species are in the dataset?

Now that we know that there are 3 different species in our dataset, our next objective is determining how many samples of each species we have. To answer this, we need to look at column 5 again, but instead of just determining how many unique values there are, we need to count how many times each of them occurs.

You may have noticed there were more parameters in the **Group** {% icon tool %} tool that we did not use. Let's have a closer look and see if any of them might help us answer this question.

> <comment-title>Tool Help</comment-title>
> To find out more about how a tool works, look at the help text (below the **Execute** button).
>
> Look at the tool help for the **Group** {% icon tool %}. Do you see any parameters that could help answer this question?
{: .comment}

Looking at the tool help for **Group** {% icon tool %}, we see that we can also perform aggregate operations such as mean, median, sum, max, min, count (and more). Counting sounds just like what we need, let's try it!

> <hands-on-title>Grouping dataset and adding information</hands-on-title>
>
> 1. **Re-run** {% icon galaxy-refresh %} the **Group** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `iris clean`
>    - {% icon param-select %} *"Group by column"*: `Column: 5`
>    - {% icon param-repeat %} *"Insert operation"*
>      - *"Type"*: `Count`
>      - *"On column"*: `Column: 1`
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris samples per species group`
>
> 3. **View** {% icon galaxy-eye %} the resulting file.
>
>    > <question-title></question-title>
>    > How many samples per species are in the dataset?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > We have 50 samples per species:
>    > >
>    > > | 1         | 2   |
>    > > |---------- | --- |
>    > > |setosa     | 50  |
>    > > |versicolor | 50  |
>    > > |virginica  | 50  |
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Analysis: How to differentiate the different Iris species?

Our objective is to find what distinguishes the different Iris species (Figure 1). We know that we have **3** species of iris flowers, with
**50** samples for each:
- setosa
- versicolor
- virginica

These species look very much alike as shown on the figure below.

![Three species of Iris flowers](../../images/iris_flowers.png "Three species of Iris flowers (Image attributions: versicolor by Danielle Langlois licensed under CC BY-SA 3.0, retrieved from <a href="https://commons.wikimedia.org/wiki/File:Iris_versicolor_3.jpg">WikiMedia</a>; virginica by Christer Johansson licensed under CC BY-SA 3.0, retrieved from <a href="https://commons.wikimedia.org/wiki/File:IMG_7911-Iris_virginica.jpg">WikiMedia</a>; setosa by and used with permission of Sonja Keohane, retrieved from <a href="http://www.twofrog.com/irissetosa.html">www.twofrog.com</a>)")

And our objective is to find out whether the features we have been given for each species can help us to highlight the differences between the 3 species.

In our dataset, we have the following features measured for each sample:
- Petal length
- Petal width
- Sepal length
- Sepal width

> <comment-title>petal and sepal</comment-title>
> The image below shows you what the terms sepal and petal mean.
> ![Iris flower image showing Petal and Sepal](../../images/iris_sepal_petal.png "Sepal and petal of Iris flowers")
{: .comment}

## Generate summary and descriptive statistics with

> <hands-on-title>Get the mean and sample standard deviation of Iris flower features</hands-on-title>
>
> 1. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `iris tabular`
>    - *"Group by fields"*: `5`
>    - *"Sort input"*: `Yes`
>    - *"Input file has a header line"*: `Yes`
>    - *"Print header line"*: `Yes`
>    - "Print all fields from input file": `No`
>    - *"Ignore case when grouping"*: `Yes`
>    - In *"Operation to perform on each group"*:
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c1`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c1`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c2`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c2`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c3`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c3`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c4`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c4`
>
> 2. Rename the dataset to `iris summary and statistics`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. **View** {% icon galaxy-eye %} the generated file
>
>
>   > <question-title></question-title>
>   >
>   > 1. Can we differentiate the different Iris flower species?
>   >
>   > > <solution-title></solution-title>
>   > >
>   > > 1. From the results, we can see that the average *Iris setosa* petal length is lower than 1.5 with a relatively small standard deviation (<0.2).
>   > > The same can be observed for *Iris setosa* petal widths. These numbers are much smaller (width and length) than *Iris versicolor* and *Iris virginica* petals.
>   > > We can then use these characteristics to differentiate *Iris setosa* from the two other species (*I. versicolor* and *I. virginica*). On the other hand,
>   > > we cannot easily differentiate *Iris Versicolor* from *Iris Virginica*. Further analysis is necessary.
>   > >
>   > {: .solution}
>   >
>   {: .question}
>
{: .hands_on}

## Visualize Iris dataset features with two-dimensional scatterplots

Let's visualize the Iris dataset to see how the features depend on each other, and
check whether we can spot any immediate patterns.

> <hands-on-title>Plot iris feature pairs in two dimensions</hands-on-title>
>
> 1. {% tool [Scatterplot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.3.5+galaxy0) %}   with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: **iris clean**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `Sepal length as a function of sepal width`
>    - *"Label for x axis"*: `Sepal length`
>    - *"Label for y axis"*: `Sepal width`
>    - In *"Advanced Options"*:
>        - *"Data point options"*: `User defined point options`
>            - *"relative size of points"*: `2.0`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>            - *"column differentiating the different groups"*: `5`
>            - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>    - In *"Output Options"*:
>        - *Additional output format*: `PDF`
>
> 2. **View** {% icon galaxy-eye%} the resulting plot:
>
>    ![Contents of the `Group` output dataset](../../images/101_foreveryone_scatter.png)
>
> 3. Rename the dataset to `iris sepal scatterplot`
>
> > <question-title></question-title>
>  >
>  > 1. What does this scatter plot tell us about Iris species?
>  > 2. Make a new scatter plot, this time with *Petal* length versus *Petal* width.
>  > 3. Can we differentiate between the three Iris species?
>  >
>  >    > <tip-title>Re-running the ggplot tool</tip-title>
>  >    > Instead of clicking on **Scatterplot w ggplot2** {% icon tool %} again, it is possible to recall the previous scatterplot parameters by clicking on re-run button and updating the parameters we wish to modify.
>  >    >    > <hands-on-title>Re-run the tool</hands-on-title>
>  >    >    > 1. Click on the {% icon galaxy-refresh %} icon (**Run this job again**) for the output dataset of **Scatterplot w ggplot2** {% icon tool %}
>  >    >    > ![rerun](../../images/rerun_scatterplot.png)
>  >    >    >
>  >    >    > This brings up the tool interface in the central panel with the parameters set to the values used previously to generate this dataset.
>  >    >    {: .hands_on}
>  >    {: .tip}
>  >
>  > > <solution-title></solution-title>
>  > >
>  > > 1. We get similar results than with Summary and statistics: *Iris setosa* can clearly be distinguished from *Iris versicolor* and
>  > > *Iris virginica*. We can also see that sepal width and length are not sufficient features to differentiate *Iris versicolor* from *Iris
>    > > virginica*.
>  > > 2. {% tool [Scatterplot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.3.5+galaxy0) %} with the following parameters:
>  > >     - {% icon param-file %} *"Input tabular dataset"*: `iris clean`
>  > >     - *"Column to plot on x-axis"*: `3`
>  > >     - *"Column to plot on y-axis"*: `4`
>  > >     - *"Plot title"*: `Petal length as a function of petal width`
>  > >     - *"Label for x axis"*: `Petal length`
>  > >     - *"Label for y axis"*: `Petal width`
>  > >     - In *"Advanced Options"*:
>  > >         - *"Data point options"*: `User defined point options`
>  > >             - *"relative size of points"*: `2.0`
>  > >         - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>  > >             - *"column differentiating the different groups"*: `5`
>  > >             - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>  > >
>  > > 3. Your new output dataset will look something like this:
>  > >
>  > >   ![Contents of the `Group` output dataset](../../images/101_foreveryone_scatter_petal.png)
>  > >
>  > > We can better differentiate between the 3 Iris species but for some samples the petal length versus width is still insufficient
>  > > to differentiate *Iris versicolor* from *Iris virginica*. And as before, *Iris setosa* can easily be distinguished from the two other species.
>  > {: .solution}
>  {: .question}
{: .hands_on}

# Galaxy management

## Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step. But when you receive new data, or a new report is requested, it would be tedious to do each step over again. Wouldn't it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the `Extract workflow` option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking on the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click on {% icon galaxy-history-options %} (**History options**) at the top of your history panel and select **Extract workflow**.
>
>    ![`Extract Workflow` entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
>    ![Selection of steps for `Extract Workflow` from history](../../images/101_foreveryone_workflow_create.png)
>
> 3. Replace the **Workflow name** to something more descriptive, for example: `Exploring Iris dataset with statistics and scatterplots`.
>
> 4. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes.
>
> 5. Click on the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created. But where did it go?
>
>    ![`Where workflows go` list](../../images/101_foreveryone_workflow.png)
>
> 6. Click on **Workflow** in the activity bar on the left side.
>    - Here you have a list of all your workflows.
>    - Your newly created workflow should be listed at the top:
>
>    ![`Your workflows` list](../../images/101_foreveryone_workflow_list.png)
{: .hands_on}

## The workflow editor


> <comment-title>Tip: Problems creating your workflow?</comment-title>
> If you had problems extracting your workflow in the previous step, we provide [a working copy for you]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/workflows/main_workflow.ga %}),
> which you can import to Galaxy and use for the next sections (see below how to import a workflow to Galaxy).
>
> {% snippet faqs/galaxy/workflows_run_trs.md path="topics/introduction/tutorials/galaxy-intro-101-everyone/workflows/main_workflow.ga" title="Main Workflow" %}
>
{: .comment}


We can examine the workflow in Galaxy's workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.

> <hands-on-title>Editing our workflow</hands-on-title>
>
> 1. Open the **workflow editor**
>    - Click on **Workflows** {% icon galaxy-workflows-activity %} at the interactive bar and locate your freshly created or imported workflow.
>
> 2. Select {% icon galaxy-wf-edit %} **Edit** to launch the workflow editor.
>    - You should see something like this:
>
>    ![Workflow editor](../../images/101_foreveryone_workflow_editor.png)
>
>    - When you click on a workflow step, you will get a view of all the parameter settings for that tool on the right-hand side of your screen (the **Details** section)
>    - You can also change the parameter settings of your workflow here, and also do more advanced configuration.
>
> 3. **Hiding intermediate outputs**
>    - We can tell Galaxy which outputs of a workflow are important and should be shown in our history when we run it, and which can be hidden.
>    - By default, all outputs will be shown
>    - Click the **checkbox** {% icon galaxy-selector %} next to the outputs to mark them as important:
>        - `outfile` in  **Unique** {% icon tool %}
>        - `out_file1` in **Group** {% icon tool%} step
>            - This should be the Group tool where we performed the counting, you can check which one that is by clicking on it and looking at the parameter settings in the **Details** box on the right.
>        - `png` in both **Scatterplot w ggplot2** {% icon tool %} steps
>    - To hide all other datasets in your history you will need to unselect the **eye** icon {% icon galaxy-eye %} in each tool box where you did not select the **checkbox** {% icon galaxy-selector %}
>    - Now, when we run the workflow, we will only see these final outputs
>        - i.e. the two dataset with species, the dataset with number of samples by species and the two scatterplots.
>
>    {% snippet faqs/galaxy/workflows_hide_intermediate_steps.md %}
>
> 4. **Renaming output datasets**
>    - When we performed the analysis manually, we often renamed output datasets to something more meaningful
>    - We can do the same in a workflow (see the tip box below)
>    - Let's rename the outputs we marked as important with the checkbox (and thus do not hide) to more meaningful names:
>        - **Unique** {% icon tool%}, output `outfile`: rename to `categories` tool
>        - **Group** {% icon tool %}, output `out_file1`: rename to `samples per category`
>        - Rename the scatterplot outputs as well, remember to choose a generic name, since we can now also run this on data other than iris plants.
>
>    {% snippet faqs/galaxy/workflows_rename_output.md %}
>
> 5. **Save your workflow** (important!) by clicking on the {% icon galaxy-save %} icon at the top right of the screen.
>
>
> 6. **Return** to the analysis view by clicking on **Galaxy** (or the Home icon {% icon galaxy-home %} or **Analyze Data** on older Galaxy versions) at the top menu bar.
>
{: .hands_on}

> <comment-title></comment-title>
> We could **validate** our newly built workflow by running it on the same input datasets that we used at the start of this tutorial, in order to make sure we do obtain the same results.
{: .comment}


## Run workflow on different data

Now that we have built our workflow, let's use it on some different data. For example, let us explore the `diamonds` R dataset with it.

> <hands-on-title>Create a new history and upload a new data</hands-on-title>
>
> 1. Create a **new history** and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. {% tool [Import](upload1) %} the file `diamonds.csv` from [Zenodo](https://zenodo.org/record/3540705/files/diamonds.csv) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3540705/files/diamonds.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the dataset to `diamonds`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Add a propagating tag {% icon galaxy-tags %} (e.g. `#diamonds`)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}

The `diamonds` dataset comes from the well-known [ggplot2](https://ggplot2.tidyverse.org/reference/diamonds.html) package developed by Hadley Wickham and was initially collected from the Diamond Search Engine in 2008.
The original dataset consists of 53940 specimen of diamonds, for which it lists the prices and various properties.
For this training, we have created a simpler dataset from the original, in which only the five columns relating to the price and the so-called 4 Cs (carat, cut, color and clarity) of diamond characteristics have been retained.

> <comment-title>The 4 Cs of diamond grading</comment-title>
> According to the [GIA's (Gemological Institute of America) diamond grading system](https://4cs.gia.edu/wp-content/uploads/2013/03/All-Scales.jpg)
> - **Carat** refers to the weight of the diamond when measured on a scale
> - **Cut** refers to the quality of the cut and can take the grades *Fair*, *Good*, *Very Good*, *Premium* and *Ideal*
> - **Color** describes the overall tint, or lack thereof, of the diamond from colorless/white to yellow and is given on a letter scale ranging from D to Z (D being the best, known as colorless).
> - **Clarity** describes the amount and location of naturally occuring "inclusions" found in nearly all diamonds on a scale of eleven grades ranging from *Flawless* (the ideal situation) to *I3* (Included level 3, the worst quality).
{:.comment}

As a further simplification, our training dataset has the qualities in the *color* and *clarity* columns re-encoded as integer values (1-23 for color qualities D-Z, and 1-11 for the clarity levels from *Flawless* to *I3*).
With this adjustment, we can reuse our workflow on the data, and analyze and visualize it following the same steps as we took for the Iris dataset.

> <hands-on-title>Run workflow</hands-on-title>
>
> To analyze the diamonds price/4 Cs dataset by reusing our workflow:
>
> 1. Open the **workflow menu** (from activity bar, left side).
>    - Find the workflow you made in the previous section,
>    - Select the option `Run` {% icon workflow-run %}.
>    - The central panel will change to allow you to configure and launch the workflow.
>    - Click on **Expanded workflow form**
>
> 2. Select the `diamonds` dataset as the input dataset.
>
> 3. Customize the first scatter plot:
>
>    This step is preconfigured to plot column 1 along the x and column 2 along the y axis, while grouping by column 5.
>    This is fine and will result in *price* getting plotted against *carat* with grouping by *cut*, but you would want to adjust the plot title and axis labels accordingly. To change any parameter click on the pencil icon {% icon galaxy-wf-edit %}:
>
>    - Change *"Plot title"* to `Diamond price as a function of carat with cut as a factor`
>    - Change *"Label for x axis"* to `Weight of the diamond (carat)`
>    - Change *"Label for y axis"* to `Price (US dollars)`
>
>    ![Customize scatter plot](../../images/customize_diamond_plot.png)
>
> 4. Customize the second scatter plot.
>
>    This one is preconfigured to plot column 3 along the x and column 4 along the y axis, which, for our new data, would plot *color* as a function of *clarity*. However, we would rather want to stick to plotting *price* against weight in *carat* as in the first plot, but group by *clarity* instead of by *cut* this time, so:
>
>    - Change *"Column to plot on x-axis"* to `1`
>    - Change *"Column to plot on y-axis"* to `2`
>    - Change *"Plot title"* to `Diamond price as a function of carat with clarity as a factor`
>    - Change *"Label for x axis"* to `Weight of the diamond (carat)`
>    - Change *"Label for y axis"* to `Price (US dollars)`
>    - And finally in *"Advanced Options"* change *"column differentiating the different groups"* to `4` (clarity).
>
> 6. Click `Run workflow`.
>
> 7. Once the workflow has started, you will initially be able to see all its steps, but the unimportant intermediates will disappear after they complete successfully:
>
>    > <question-title></question-title>
>    > 1. How many cut category are there in the Diamond dataset ?
>    > 2. How many samples are there in each cut category ?
>    > 3. What do you notice about the relationship between price and carat ?
>    > 4. Based on the plot showing Price vs. Carat with Clarity as a factor, do you think clarity accounts for some of the variance in price? Why ?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 5 different cut categories:
>    > > - Fair
>    > > - Good
>    > > - Ideal
>    > > - Premium
>    > > - Very Good
>    > >
>    > > 2. We have the following number of samples in each cut category:
>    > >
>    > >    | 1         | 2      |
>    > >    |---------- | ------ |
>    > >    | Fair      |	1610  |
>    > >    | Good	     |  4906  |
>    > >    | Ideal     |	21551 |
>    > >    | Premium   |	13791 |
>    > >    | Very Good |	12082 |
>    > >
>    > > 3. Using any of the scatter plots we made, we can see an obvious positive (non-linear) relationship between both variables: as carat size increases, price also increases. There is also very clear discrete values that carat size takes on, which are those vertical strips on the graph.
>    > >
>    > >    ![Price by Carat and Clarity](../../images/price_by_carat_and_clarity.png)
>    > >
>    > > 4. Holding carat weight constant, we see on the scatter plot shown above that diamonds with lower clarity are almost always cheaper than diamonds with better clarity: diamonds that are “Internally Flawless” are the most expensive whereas “I1” are the least expensive clarity types. So clarity explains a lot of the variance found in price!
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}


## Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

{% snippet faqs/galaxy/histories_sharing.md %}

> <hands-on-title>Share history</hands-on-title>
> 1. Share your history with your neighbour.
> 2. Find the history shared by your neighbour. Histories shared with specific users can be accessed by opening the history menu for the activity bar and selecting in the History menu the **Shared with Me** tab.
{: .hands_on}

# Conclusion


{% icon trophy %} Well done! You have just performed your first analysis in Galaxy. Additionally you can share your results and methods with others.


topic: introduction
tutorial: galaxy-intro-strands
title: Introduction to Genomics and Galaxy
difficulty: introductory
time_estimation: 2H30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 243b01123d2772691c7c562575405793
last_modified: 2026-02-26T21:12:17.039636
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-strands/tutorial.html

content:
This practical aims to familiarize you with the Galaxy user interface. It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows, and sharing your work.

> ###  {% icon comment %} Audience
> This tutorial is for those who are new to Galaxy, genomics, and bioinformatics.  If you aren't new to bioinformatics you can just do the items listed in the Hands-On boxes ({% icon hands_on %}), or you can try one of the [other introductory tutorials]({% link topics/introduction/index.md %}).
>
> This tutorial teaches the same basic content as [Galaxy 101]({% link topics/introduction/tutorials/galaxy-intro-101/tutorial.md %}), but requires less knowledge of biology to understand the questions this tutorial addresses.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction

> <comment-title>Requirements</comment-title>
>
> To run this practical you will need
>
> 1. An internet-connected computer.  Galaxy can run on your laptop without an internet connection, but this practical requires access to resources on the web.
> 1. A web browser. [Firefox](https://www.mozilla.org/firefox) and [Google Chrome](https://www.google.com/chrome/) work well, as does [Safari](https://www.apple.com/safari/).  Internet Explorer is known to have issues with Galaxy so avoid using that.
> 1. Access to a Galaxy instance.  Galaxy is available in many ways. If you are doing this practical as part of a workshop, the instructor will tell you which instance to use. If you are doing this on your own, you can use [usegalaxy.org](https://usegalaxy.org).
{: .comment}


> <question-title>Our Motivating Question</question-title>
> *I wonder if genes on opposite strands ever overlap with each other, and if so, how common is that?*
{: .question}

To explore this question we need a basic understanding of *genomes, chromosomes, strands,* and  *genes.*

> <comment-title>Definitions 1</comment-title>
>
> - **Genome**
>
>   The genome is the collection of all DNA native to an organism. For humans, the genome is all of a person's chromosomes.
>
> - **Chromosome**
>
>   The largest unit of DNA organization in an organism.  Humans have two copies of 23 chromosomes.  Chromosomes are *linear* in humans, and all animals and plants.  (Bacteria have *circular* chromosomes.)
>
> - **Strand**
>
>   Chromosomes are *double-stranded*.  One is the forward strand, is typically drawn on top, and moves from left to right. The other, reverse strand, is typically drawn on the bottom and moves from right to left.  Genes can occur on either strand.  A single gene will have parts on only one stand.
>
> - **Gene**
>
>   "What is a gene?" is actually a hotly debated question.  For our purposes, a gene is a section of DNA on chromosome strand that creates a molecule used by an organism.
>
> Graphically, the human genome can be shown as the chromosomes that are in it:
>
> ![Human genome consists of 23 (pairs of) chromosomes](../../images/human_genome_chroms.png)
>
> And here's a representation of a chromosome, strands, and genes on the chromosome.
>
> ![Chromosomes are double stranded and have forward and reverse strands; strands have genes on them](../../images/chromosome_strands_genes.png)
{: .comment}


The question we are asking looks like:

![Does a situation like this exist?](../../images/do_genes_on_opposite_strands_ever_overlap.png)

*Non-overlapping genes are common.  How common are overlapping genes?*

# Get human data

To answer this question we need to know where genes start and stop on human chromosomes.  That seems like a simple question, but if you are new to bioinformatics it's actually a hard question to answer.  Web searches will land you at any number of useful places on the web, but without a lot of background knowledge it's hard to know what you want:  *What's the difference between sequence and annotation?  What are FASTA, BED, GTF, GFF3, and VCF?  What are GRCh37, GRCh38, hg19, and hg38 (and what happened to hg20 through hg37 - are they okay)?*

It turns out that for this particular question (and for many others), most **Galaxy** instances can help us find this information.

> <hands-on-title>Log in to Galaxy</hands-on-title>
> In your web browser, *go to* **your Galaxy instance** and *log in or register*.
{: .hands_on}

The Galaxy interface consists of three main parts. The available tools are listed on the left, your analysis history is recorded on the right, and the central panel will show the home page, tool forms, and dataset content.

![Galaxy interface showing history panel on the right, tools panel on the left and main panel at the center](../../images/galaxy_interface.png)

> <hands-on-title>Start with an empty history</hands-on-title>
>
> - At the start of the tutorial you should ensure that you are working in a new history, i.e. one without any datasets
>
>   {% snippet faqs/galaxy/histories_create_new.md %}
>
{: .hands_on}

## Get data into Galaxy

There are [many ways to get data into a Galaxy instance]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}). We are going to use the **Get Data** toolbox in the **Tools** panel on the left.

> <hands-on-title>Open Get Data toolbox</hands-on-title>
>
> 1. *Click* on the **Get Data** toolbox to expand it.
>
>    ![The Get Data toolbox](../../images/getdata.PNG)
>
{: .hands_on}

The **Get Data** toolbox contains a list of data sources that this Galaxy instance can get data directly from.  **Upload file** is quite useful for getting data from your computer or from the web (see the [Getting data into Galaxy slides]({% link topics/galaxy-interface/tutorials/get-data/slides.html %})).  Today we are going to use the **UCSC Main table browser**.

### Get genes

> <hands-on-title>Go to UCSC</hands-on-title>
>
> 1. *Click* on the tool {% tool UCSC Main table browser %} to go to UCSC.
>
>    ![Click on UCSC Main table browser](../../images/101_01.png)
>
{: .hands_on}

This will take you to the UCSC Table Browser:

 ![UCSC table browser tool, first screen for genes](../../images/ucsc_table_browser_w_default_settings.png)

The [UCSC Table Browser](https://genome.ucsc.edu/cgi-bin/hgTables) provides access to all the data that is shown in the [UCSC Genome Browser](https://genome.ucsc.edu/cgi-bin/hgGateway) (see box below). If you are working on a species that UCSC supports (like human) then the Table Browser is a great place to get genomic data.

The Table Browser has a daunting number of options. Fortunately, they are all set to commonly used defaults, greatly simplifying things, and most of the options are already set to what we want:

* **clade:** `Mammal`
* **genome:** `Human`
* **assembly:** `Dec. 2013 (GRCh38/hg38)`
* **group:** `Genes and Gene Predictions`
* **track:** `GENCODE v41`

**clade** and **genome** seem pretty clear.  **assembly** asks which version/definition of the human genome we want.  (Any will do for our question, but UCSC is suggesting `hg38`, which is also the most recent.)  **group** is set to `Genes and Gene Predictions` which sounds like what we want. So far so good.

**track** has a bewildering list of options. UCSC suggests `GENCODE v41`.  A web search leads us to the [GENCODE web site](https://www.gencodegenes.org/) which prominently states:

> <warning-title>ALL GENCODE is different from GENCODE</warning-title>
> **Warning** The Table browser only provides the most recent release of GENCODE which is updated several times per year. ALL GENCODE does not contain the same data as GENCODE and you should select the GENCODE track even if the version number is wrong.
{: .warning}

>The goal of the GENCODE project is to identify and classify all gene features in the human and mouse genomes with high accuracy based on biological evidence...
{: .quote}

Time for a few more definitions.

> <comment-title>Definitions 2</comment-title>
>
> - **Reference genome**
>
>    A reference genome is the *genome of a single individual* that has been thoroughly studied, to the point that we know exactly what most of that individual's DNA is.  In practice a reference genome is used as shared map by researchers working on that organism. Reference genomes are updated periodically as techniques improve.
>
> - **Sequence**
>
>   A genome's sequence describes the DNA in that genome, down to the A, C, T, and G (single nucleotide) level including the exact location where each is.  Given a reference genome, you can ask questions like, "What's the DNA on chromosome 2 between positions 1,678,901 and 1.688,322?"
>
> - **Genome/Gene annotation**
>
>   The sequence tells us what DNA is where, but it doesn't tell us anything about the function of that DNA.  *Annotation* is additional information about particular regions of the genome like where genes, repeats, promotors, and centromeres are, or how active a particular gene is.
{: .comment}

The **track** option asks us which set of annotations do we want to get?  There are so many choices because annotation is the result of analysis and interpretation, and there are many ways to do this. (And in this case, many of the options aren't even genes or gene predictions.)

GENCODE is "high-quality" and  "gene annotation." That sounds like a good thing to use.  Lets stay with the default: `GENCODE V41`.

So far we haven't changed *anything* from the defaults.  Lets change something.  The default  **region** is the whole genome, which can be done, but it's a lot of information. For this exercise lets use just one (small) chromosome.

> <hands-on-title>Limit the region and get the data.</hands-on-title>
>
> 1. Say that we just want chromosome 22
>    - For **region** select `position`.
>    - In the text box next to `position` enter `chr22` (case matters).
>
>    ![Change the region](../../images/ucsc_tb_set_region_chr22.png)
>
> 2. *Click* the **get output** button.
>
>    And, that doesn't actually get us the output.  It sends us to a second UCSC page that asks us exactly what we want.
>
>    ![UCSC Table Browser 2nd page](../../images/ucsc_tb_2nd_page_whole_gene.png)
>
> 3. Under **Create one BED record per** make sure that **Whole Gene** is selected.
>
> 4. *Click* the **Send query to Galaxy** button at the bottom of the form.
{: .hands_on}

This returns us to Galaxy, first displaying a big green box (that's good!) and then returning us to the view we started with.  Except that we now have an item in our history, the dataset from UCSC.


### History Item Status

Watch your new history item.  It will go through three statuses before it's done.

| Color | Icon | Meaning | |
| ---- | ---- | ---- |
| **Grey** | Clock | Item is waiting to start (waiting for data transfer to start) | ![Status: Queued](../../images/status_queued.png) |
| **Yellow** | Spinner | Item is running (data is actively being transferred). | ![Status: Running](../../images/status_running.png) |
| **Green** | None | Item has finished successfully (data transfer complete). | ![Status: Successfully finished](../../images/status_finished_success.png) |
| **Red**  | Cross | The job has failed. There can be [many reasons](https://galaxyproject.org/support/tool-error/). | ![Status: Failed](../../images/status_failed.png) |
| ---- | ---- | ---- |

You can find more information in the [Undestanding Galaxy history system]({% link topics/galaxy-interface/tutorials/history/tutorial.md %}) training.

## Examine the data

> <hands-on-title>Look at the data.</hands-on-title>
> Once the dataset is green, *click* on the dataset name (something like **UCSC Main on Human...**)
{: .hands_on}

This expands the dataset and shows you information about it, and a preview of its contents.

![Preview of the data from UCSC Table Browser](../../images/genes_human_chr22_preview.png)

From this preview we can obtain a huge amount of useful information:
1. The dataset has over 4000 regions, meaning that there are over 4000 genes on chromosome 22.
1. The dataset is in **BED** format, one of several standards for representing genome annotation.  We got BED format because BED was preselected as the output format in the UCSC table browser.
1. The dataset's "database" is **hg38**.  This says which revision of the reference genome this data maps too.  hg38 is the latest human reference genome.  hg38 was also selected by default in UCSC.
1. Finally, it shows us the first 5 rows in the dataset.

The dataset preview is informative, but you can't see much of the actual dataset.  Lets use one of the dataset icons to see the whole dataset:

> <hands-on-title>Look at all the data.</hands-on-title>
> 1. Click on the {% icon galaxy-eye %} (eye) icon to view the contents of the dataset. This displays all of the data.
>
>    ![Full dataset in central panel](../../images/genes_human_chr22_dataset_view.png)
>
> 2. Use the side scroll bars to move through the file.
>
> > <question-title></question-title>
> >
> > How many columns include the BED file?
> >
> > > <solution-title></solution-title>
> > >
> > > Our BED file includes 12 columns, on which the information about the genes distribution along the genome in the chromosome 12 is summarized.
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

> <details-title>Additional information about BED files</details-title>
>
>[BED](https://genome.ucsc.edu/FAQ/FAQformat.html#format1) is one of several well-established tabular formats for genomic data.  Other formats include GFF3 and GTF.  For the type of analysis we are doing today, BED format is easiest to work with.  BED was created to power the UCSC Genome Browser.  BED files contain between 3 and 15 columns.  Our example BED file describes genes and contains 12 columns.
>
>We care about columns 1, 2, 3, and 6:
>
>| # | Column Name | Meaning |
>| ---- | ---- | ---- |
>| 1 | Chromosome | The name of the chromosome this gene is on. |
>| 2 | Start | Where on the chromosome the gene starts. |
>| 3 | End | Where on the chromosome the gene ends. |
>| 6 | Strand | Which strand the gene is on.  `+` means forward (top, left to right), `-` means reverse (bottom, right to left) |
>
>See the [BED format description at UCSC](https://genome.ucsc.edu/FAQ/FAQformat.html#format1) for a full description of all the columns.
>
>
{: .details}


## Naming

Galaxy allows you to name your analyses (your histories) and your datasets.  We only have one history ("Unnamed history") and one dataset ("UCSC Main on Human:...") so far, but it's a good idea to

1. Always name your histories
2. Name your input and final output datasets, and any significant intermediate datasets.

You don't have to do this.  Galaxy is quite happy for you to have an infinite number of "Unnamed history" histories, and to have all your datasets be obscurely named.  However, once you've run your first 5 unnamed analyses, all with obscurely named datasets, you'll might wish you would have named everything.

> <hands-on-title>Name your stuff</hands-on-title>
>
> 1. **Name your history** to be meaningful and easy to find.
>    - *Click* on the title of the history and enter something like **Intro - Strands** as the name.  Hit the `enter` key on your keyboard to save it.
>   ![Rename the history]({% link shared/images/rename_history.png %})
> 1. **Rename your dataset**
>    - *Click* on the **pencil icon** to edit the dataset attributes.
>    - In the next screen change the name of the dataset to something like `Genes` or `Genes chr22`.
>    - *Click* the **Save** button at the bottom of the screen.
>
>    Your history should now look something like this:
>
>    ![Rename dataset](../../images/genes_human_chr22_rename.png)
{: .hands_on}

# We've got the data - what's our plan for answering the question?

You have to know what's possible, before you can build a plan.  If you don't have experience with data analysis then you might not have any idea how you would answer our question.  Before we dive in using a particular solution, think about how you might solve this.  If you don't have any experience with tools, then think about how you might solve it manually, using pencil and paper (it may help to assume you have an infinite supply of helpers to do the pencil and paper work).

Here's how we'll answer this question:

1. Split the genes dataset in two: one for genes on the forward strand, and one for genes on the reverse strand.
1. Compare the two datasets to see which ones, if any, overlap.
1. Check how many (or what percentage) of our genes overlap with another gene.

It turns out that all of these steps are easy in Galaxy!

## Split the genes into forward and reverse datasets

How might we do this?  Column 6 contains the strand information.  Can we split genes into two datasets based on the value of Column 6.  How?  Lets take a look at our available tools.  And *whoa! There are over 40 toolboxes, and several hundred tools.* How are we going to find a tool that can do the split?


> <hands-on-title>Searching for and launching a tool</hands-on-title>
>
> 1. Try the **tools search box**.  Think of terms that might describe what we want to do and type them in the search box.  Do you see anything promising?  Explore a little.
> 2. If you haven't already searched with it, *enter* `filter` in the **tool search box**.  Among the results you should see
>    - {% tool [Filter](Filter1) %} data on any column using simple expressions.
>    - That might work.
> 3. *Click* on {% tool [Filter](Filter1) %} to open the Filter tool in the central panel.
> 4. Take a look at the **Syntax** and **Example** sections to understand what the tool does.
>
> ![The Filter Tool form](../../images/filter_tool_defaults.png)
{: .hands_on}

It doesn't say anything about **Filter** being able to split a file into multiple files.  It does look like we can use Filter to get only genes on the forward strand, or only genes on the reverse strand.  We would have to run Filter twice, once for forward strand genes, and once for reverse strand genes. Let's do that.

> <hands-on-title>Run the Filter tool to get genes on the forward strand.</hands-on-title>
>
> * The filter tool has 3 fields:
>
>   1. **Dataset**: This pulldown will list any dataset from your history that this tool can work on.  In your case that's probably only one dataset. {% icon param-files %} *"filter"*: `Genes chr22` .
>   2. **Condition**: this free text field is where we specify which records we want in the output dataset. Put in *"Condition"*:  `c6 == "+" ` .
>       This specifies that column 6 (the strand) must be equal to (`==` is Python for *is equal to*) a plus sign.
>   3.  *"Header lines to skip"*:  `0 `
>
>       Our dataset does not have any header lines.
>
> * Finally, *click* the **Execute** button.
>
>   ![Run the Filter tool to get only the forward strand genes](../../images/filter_tool_forward_strand_genes_only.png)
>
{: .hands_on}

This adds another dataset to your history. This one should contain only genes on the forward strand.  Once the dataset is green, click the {% icon galaxy-eye %} (eye) icon to confirm this. We also recommend that you rename this dataset to something like `Genes, forward strand` (remember how?).

Now we want to get the genes on the reverse strand.  There are actually many ways to get this.  Here are two of them.

> <hands-on-title>Get genes on the reverse strand</hands-on-title>
>
> **Method 1**
>
> 1. Open the dataset preview by *clicking* on the name of the `Genes, forward strand` dataset.  This shows an icon than the uploaded `Genes chr22` dataset did not: a looping arrow.
> 1. *Click* the **looping arrow** ("Run this job again") icon.  This won't actually run the job again.  What it will do is bring up the Filter tool form with *the exact same settings that were used to produce this dataset.*
> 1. Rerun {% tool [Filter](Filter1) %} but with
>    - *"Condition"*:  `c6 == "-" `
> 1. *Click* the **Run Tool** button.
>
> **Method 2**
> 1. *Click* on {% tool [Filter](Filter1) %} in the tool panel to open the Filter tool in the central panel.
> 1. *Fill* the form as before, *except*:
>    * Make sure the **Dataset** pulldown is set to the `Genes chr22` dataset.
>    * *Set* **Condition** to `c6 == "-"`.
> 1. *Click* the **Run Tool** button.
>
> **Both Methods**
> 1. *Rename* your new dataset to something like `Genes, reverse strand`
{: .hands_on}

The rerun button can be a huge help as you run more complex tools.

> <comment-title>Empty result?</comment-title>
>
> If you used Method 2 and didn't explicitly set the dataset, then you ran Filter on the `Genes, forward strand` dataset. None of the genes in the forward strand dataset have "-" in column 6 so all of them were filtered out from the result.
>
> Try again and set the dataset to your `Genes chr22` dataset.
{: .comment}


Your history should now have (at least) 3 datasets in it, with names like:

* `Genes chr22`
* `Genes, forward strand`
* `Genes, reverse strand`

The number of genes in the `forward` plus `reverse` datasets should be the same as in the `Genes chr22` dataset.  If they aren't can you figure out why?

## Check for overlaps

Genes are an example of a *genomic interval*.

> <comment-title>Definitions 3</comment-title>
>
> - **Genomic interval**
>   In Galaxy, a *genomic interval* is a something that spans part of a chromosome (or some other linear frame of reference like a contig).  Genes are a common example of a genomic interval.  Even a chromosome is a genomic interval, albeit a very long one.
{: .comment}

Galaxy excels at answering questions about genomic intervals and different sets of genomic intervals relate to each other.  Lets take a look.

> <hands-on-title>Genomic Interval Tools</hands-on-title>
>
> * The next step is finding overlapping intervals, so type `interval` in the tool search box. There are many results from this search, so you might want to click the *Show sections* button to see which sections the tools are organised into.
> * *Explore* the tools in this toolbox, looking for something that we can use to see which genes on opposite strands overlap.
{: .hands_on}

Although there are the bedtools (section **BED**) and a section **Operate on Genomic Intervals** offering promising tools, the simple **Intersect** tool under **Text Manipulation** appears sufficient.  Let's try **Intersect**.

> <hands-on-title>Genomic Interval Tools</hands-on-title>
>
> 1. {% tool [Intersect](toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0) %} with the following parameters:
>     - *"Return"*:  `Overlapping Intervals`.
>       This looks like it might return whole genes, while `Overlapping pieces` may return only the parts that overlap.  We suspect that whole genes might be more useful.
>     - {% icon param-files %}*"of"*:  `Genes, forward strand` (the first dataset)
>     - {% icon param-files %} *"that intersect"* : `Genes, reverse strand` (the second dataset)
>     - *"for at least"*: `1`
>
>       This will return genes with even just one position overlapping.
>     - *Click* **Run Tool**.
>
>     ![Run Intersect](../../images/genes_human_intersect_strands.png)
>
> 3. Now repeat the intersect, but make the first dataset be the reverse genes, and the second be the forward genes.
>
> 4. Finally give both of the new datasets meaningful names, like `Overlapping forward genes` and `Overlapping reverse genes`
{: .hands_on}


# Results and final steps.

At this point we *could* say that we have answered our question. Using dataset previews in the history panel, we can compare the number of genes in the `Overlapping forward` and `Overlapping reverse` datasets with the number of genes in the full `Genes chr22` dataset, and *conclude that overlapping genes on opposite strands are actually pretty common.*

However, before we rush off to publish our conclusions, let's

1. Get both the forward and reverse overlapping genes into a single dataset (one link will look better in our publication), and
2. *Visualize* our new dataset, just to double-check our results.

## Combine forward and reverse overlapping genes into one dataset.

What tool can we use to combine the two datasets into one?  Try *searching* for `combine` or `join` or `stack` in the tool search box.  You'll find lots of tools, but none of them do what we want to do. *Some times you just have to manually look through toolboxes to find what you need.*  Where should we look?  It's probably not **Get Data** or **Send Data**, but it could easily be in any of the next 4 toolboxes: **Lift-Over, Collection Operations, Text Manipulation, or Datamash**.

It turns out that **Lift-Over** and **Collection Operations** are not what we want.  (But, take a look at them: if you are going to work with genomic data it won't take long before you'll need both.)

> <hands-on-title>Concatenate two datasets</hands-on-title>
>
> 1. *Open* the **Text Manipulation** toolbox.
> 2. Near the top of the toolbox is **Concatenate datasets tail-to-head**. *Click* on it.  Lets try that tool.
> 3. {% tool [Concatenate](cat1) %} with the following parameters:
>   - {% icon param-files %} *"Concatenate Dataset"*: `Overlapping reverse genes`.
>   - *"Dataset*"
>      - Click on {% icon param-repeat %} *"Insert Dataset"*
>
>         This adds a second dataset pull-down menu to the form.
>
>      - In *"1: Dataset"*
>         - {% icon param-files %} *"Select"*: `Overlapping forward genes` as the second dataset.
> 4. *Click* **Run Tool**
> 5. *Rename* the resulting dataset something informative like `Overlapping genes`
{: .hands_on}

Once the concatenate operation is finished, preview the dataset in your history panel.  Does it have the expected number of genes in it?  If not, see if you can figure out what happened.

## Visualize the overlapping genes

Galaxy knows about several visualization options for lots of different dataset types, including BED.  Whenever you preview a dataset in the history panel, Galaxy provides links to these visualizations.  For BED files (which is the format we have), options include **IGB, IGV,** and **UCSC main.**  IGB and IGV are widely used desktop applications and eventually you may want to install one or both of them.  For now, let's visualize the data at UCSC, using the UCSC *Genome* Browser.

> <hands-on-title>Display data in Genome Browser</hands-on-title>
>
> 1. Click on your `Overlapping genes` dataset in your history panel. This will show the dataset preview in the history panel.
> 2. Click to expand the dataset, if it isn't already, so that you can see the dataset metadata and additional actions like Visualize.
> 3. Click on the {% icon galaxy-barchart %} (**Visualize**) icon
> 4. Click on the **display at UCSC (main)** link that appears in the blue box at the top of the screen.
>    ![visualisation options are shown in Galaxy's middle panel]({% link topics/introduction/images/101_displayucsc.png %})
>
> This will launch a new window, showing UCSC's Genome Browser with our dataset shown right at the top.  UCSC figures out that our first overlapping gene is ~11 million bases into chromosome 22, and it has landed us there.
>
>   ![UCSC Genome Browser](../../images/ucsc_gb_genes_strands_overlapping.png)
{: .hands_on}


> <comment-title>Background: UCSC Genome Browser</comment-title>
>
> * *Genome browsers* are software for viewing genomic information graphically.  The [UCSC Genome Browser](https://genome.ucsc.edu/cgi-bin/hgGateway) (and most genome browsers) typically display different types of *annotation* about a region of a genome.  This is displayed as a stack of *tracks* and each track contains a different type of information.
>
> * Genome browsers are useful for seeing information in context and for seeing (and discovering) correlations between different types of information.  (They are also useful for visually checking results, which is what we are doing now.)
>
> * The UCSC Genome Browser has information on over 100 animals, and their [Archaeal Genome Browser](http://archaea.ucsc.edu/cgi-bin/hgGateway?db=pyrFur2) has genomic information on well over 100 microbial species.
{: .comment}

Now, take a look at one of our results.  (Any pair of overlapping genes will do.)  Our data is in the second to top track (**User Track / User Supplied Track**). That track shows a line of small black boxes, sometimes connected with a line.

> <hands-on-title>Zoom in on an area of the chromosome that shows a set of linked black boxes</hands-on-title>
> To zoom in,
> * *Click* on the **Scale** track (the top track) just to the left of the start of the black boxes.
> * Now *drag* the mouse across the Scale track to just to the right of the  black boxes and let go.
>   ![Zoom by clicking and dragging on the Scale track](../../images/ucsc_gb_zoom_by_drag.png)
> * A window pops up describing several ways to interact with the browser.  Just *click* the **Zoom In** button at the bottom.
> * This redraws the window, this time zoomed in to what you highlighted.
> * Continue to zoom in until you have the set of linked black boxes you picked centered on the screen.
> * Once you are as zoomed as you want to be, click on one of the linked boxes.  This will expand the track:
>   ![One gene, fully zoomed and expanded](../../images/ucsc_gb_zoom_to_1_gene.png)
{: .hands_on}

The black boxes connected by lines represent genes, and each set of connected boxes is a single gene (actually, a single transcript of a gene).  Take a close look at the top several tracks.

* It looks like we preserved the gene definitions just fine.
* It looks like, sure enough, there are overlapping genes here, and they are on opposite sides.  The arrows on the connecting lines tell us which strand the gene is on.

Um, *what's up with the boxes and the lines connecting them?*

> <comment-title>Definitions 4</comment-title>
>
> - **Exon**
>   In humans (and in all plants and animals) the molecules that are built from genes are often only built from a part of the DNA in the gene. The sections of DNA that can produce the molecules are called *exons*.
{: .comment}

As you may have guessed (or already knew): The black boxes are exons.  *Genes* are defined as *covering the entire area from the first black box to the last connected black box.*

## Do we have a problem?

*Maybe.*

Our *ad hoc* review of identified overlapping genes in the UCSC Genome Browser has *(or should have!)* confirmed that every gene we said has an overlapping gene on the opposite strand does in fact have that.  So, our conclusion appears solid: A significant percentage of genes *do* overlap with other genes on the opposite strand.

But, our conclusion may not be as *significant* as we had hoped. If only parts of genes, the exons, make stuff in our bodies, then should we have run this analysis on just the exons rather then the entire genes?  Probably.

Let's refine our question slightly

> <question-title>Our Revised Motivating Question</question-title>
> *I wonder if **exons** on opposite strands ever overlap with each other, and if so, how common is that?*
{: .question}


### So, I have to do this all over again, but with exons?

*Well, yes and no.*  We will have to run the analysis again, this time on exons instead of whole genes. But we won't have to manually recreate every step of our analysis. Galaxy enables us to create a reusable *workflow* from the analysis we just did, and then we can rerun the analysis, as one step, anytime we want to, and on any dataset.

## Walk through the workflow tutorial

Run the [Create a reusable workflow from a history]({% link topics/galaxy-interface/tutorials/history-to-workflow/tutorial.md %}) tutorial for how to do this, *and then come back here to run your newly created workflow with the exon data.*

# Rerun analysis with exon data

We want to run the same analysis, but this time only look for overlaps that happen in *exons*, the parts of genes that produce stuff our body uses.  Before we start looking at exons, let's start a new history, one that contains only the genes file we got from UCSC. We could go back to UCSC and refetch the file, but there is an easier way.

> <hands-on-title>Create a new history that contains some data from current history</hands-on-title>
>
> 1. *Click* on the **gear icon** at the top of the current history.
> 2. *Select* the **Copy Datasets** option from the pull down menu.  This launches the copy datasets form.
> 3. Under **Source History:**, *select* the dataset(s) you want to copy to the new history.
>    - In our case, we only want the `Genes chr22` dataset.
> 4. Under **Destination History** enter an informative history name in the **New history named:** box.
>    - For example, `Exon overlaps on opposite strands`
>      ![Copy only the Genes chr22  dataset to the new history](../../images/copy_datasets_to_new_history.png)
> 5. *Click* the **Copy History Items** button to create your new history.
>    - This creates a new history (with the copied dataset) and throws up a green box saying:
>      "1 dataset copied to 1 history: `name you gave your new history`."
> 6. The history name is a link.  *Click* on it.
{: .hands_on}

## Get the exon data

And your new history appears in the history panel with the copied *genes* dataset.  What we need is *exons.* How can we get the exon information?  There are two relatively easy ways to get this information, one of which will be very familiar.

The first method involves going back to the UCSC Table Browser.  Everything on the first form would stay the same: We still want human, hg38, GENCODE v41, and just `chr22`. The second form is what changes.  Instead of getting the **Whole Gene**, we need to retrieve the **Coding Exons** only.

The second method is to use the **Gene BED To Exon/Intron/Codon BED expander** tool in the **Operate on Genomic Intervals** toolbox to extract the exon information from the genes BED file we already have. (*Of course!* you say.  Umm, there is no way that you should have known that you already had this information in the genes file, or that this tool existed.)

> <hands-on-title>Get the exon data</hands-on-title>
>
> 1. Get the exon information, either by revisiting UCSC, or by using the {% tool [Gene BED To Exon/Intron/Codon BED expander](gene2exon1) %} tool.  If you use the expander tool select **Coding Exons only** from the **Extract** pull-down.
> 2. Give the resulting dataset a meaningful name.
{: .hands_on}


If you got the data from UCSC it will look something like this:
![Contents of the exons dataset from UCSC](../../images/101_exons.png)


Your history should now have two datasets: one describing entire genes, and one describing just the exons.

## Rerun the analysis, this time on exons.

When you did the *History to Workflow* tutorial you created a new workflow that was then added to your list of defined workflows.

> <hands-on-title>Run the workflow on the exon data.</hands-on-title>
>
> 1. *Click* the **Workflow** tab in the menu at the top of the Galaxy page.
>
>    This lists all of your defined workflows, including the one you just created.
> 2. *Click* on the {% icon workflow-run %} (Run workflow) button next to your workflow.
>
>    This launches the workflow run form (which you also saw when testing this workflow)
> 3. *Select* **No** under **Send results to a new history**
>
>    If you do not see this option displayed, *click* on **Expand to full workflow form** to reveal it.
> 4. *Set* the first (and only) input dataset to the `Exons` dataset, using the pull-down menu.
> 5. *Click* the **Run workflow** button.
{: .hands_on}

And Galaxy launches the workflow and says (in a nice big green box) something like:

> You can check the status of queued jobs and view the resulting data by refreshing the History pane.
{: .quote}

Which in this case *is the truth*.  You can refresh the history panel by either reloading the whole page, of by clicking the looping arrow icon at the top of the history panel.  What you'll see is a stack of history steps that will go from queued to running to done as you watch them.

All steps in the history will be green when the workflow is done. Once that happens compare your output dataset with your input dataset?  What percentage of exons overlap with other exons on the opposite strand, and is it common or rare?  Is it a smaller percentage than we saw for whole genes?


## Are we done?

Probably.  Note that we can no longer say what percentage of *genes* overlap.  We *can* say what percentage of exons overlap, and that is probably close enough for our goals.  If it isn't and we actually need to say what percentage of genes overlap, then we will have to do some extra work.  This can be done is several ways, but the Galaxy 101 tutorial may give you some ideas on how to follow this question all the way to genes.

# Conclusion

{% icon trophy %} Well done! You have just performed your first analysis in Galaxy!

{% snippet faqs/galaxy/why_not_excel.md %}


topic: transcriptomics
tutorial: rna-seq-analysis-clustering-viz
title: RNA-Seq data analysis, clustering and visualisation tutorial
difficulty: introductory
time_estimation: 6h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 7d3dd813120b5b59b30fe2102cf3e3f6
last_modified: 2026-02-26T21:12:18.564999
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-analysis-clustering-viz/tutorial.html

content:
In this computer assignment, we will analyze the results of an RNA-seq dataset. This is one type of high-dimensional -omics data that biomedical scientists frequently use. The raw RNA-seq data are sequence reads, but we will use the processed data from an RNA-seq experiment, often referred to as a count matrix. A count matrix contains for every sample (in the columns of the matrix) and every gene or transcript (in the rows of the matrix) the number of sequencing reads representing that gene or transcript (an integer, i.e., 0, 1, 2, 3, ...). The expression level of that gene is derived from the read count by correcting (normalization) for the total number of sequencing reads in a particular sample.

RNA-seq is often used to explore the function of a gene or the effect of a genetic variant. Since RNA-seq can survey the expression levels of all genes, it is well suited to obtain a global idea of the effect on the physiology of a cell and the biological processes affected. In this computer assignment, we will explore a dataset from mouse B-cells from TP53-/- knockout and wild-type mice. It is a two-factor experiment, because cells of both genotypes have been analyzed with or without being exposed to ionizing radiation (IR); thus, genotype and exposure to IR represent different factors in the experiment. The research paper describing this dataset is {% cite Tonelli2015 %} ([direct link](http://dx.doi.org/10.18632/oncotarget.5232)). The TP53 gene coding for the p53 protein is a well-known tumor suppressor gene. p53 is involved in the induction of apoptosis, senescence, cell cycle arrest and metabolic reprogramming. p53 is activated by double strand DNA breaks and routes cells to apoptosis when the DNA damage cannot be repaired anymore. TP53 mutation is one the most common driver of mutations in cancer, because TP53-deficient cells continue to proliferate, even after significant DNA damage. IR induces double strand DNA breaks and cell death (apoptosis) and is a well-known activator of p53. The authors used TP53-/- cells and wild-type cells to explore which part of the response of B-cells to IR is dependent on p53.

> <comment-title>Full data</comment-title>
>
> The original data are available at NCBI Gene Expression Omnibus (GEO) under accession number [GSE71176](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE71176). The raw RNA-Seq reads have been extracted from the Sequence Read Archive (SRA) files and converted into FASTQ files and counts.
>
{: .comment}

# Data upload

In the first part of this tutorial we will upload the count matrices to your Galaxy environment.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-Seq exercise
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the count files from our [github repository](https://github.com/pacthoen/BMW2_RNA_clust_vis) using the Data Upload menu (top left) and the button _Paste/Fetch Data_.
>
>    ```
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_2.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_3.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_4.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_2.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_3.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_4.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_mock_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_mock_2.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_IR_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_IR_2.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    See also [this screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20115748.png)
>
>    *Note:* The files with null in the filename are for the p53 knockout (p53-/-) cells and the files with p53 in the filename are for the wild-type cells
>
>    **Alternative:** If this results in an error, you may download the zip file with all the input data from the [github site](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/data/input_data_Bcell_dataset.zip) to your local computer, unzip and import all the files using the _Choose local file_ option from the Upload menu.
>
> 3. **Inspect** {% icon galaxy-eye %} some of the files by hitting they eye button
> 4. Upload also the gene annotation file:
>
>    ```text
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/Mus_musculus.NCBIM37.65.gtf.gz
>    ```
>    **Alternative** Like before, if this results in an error, you can download the gtf.gz file from the [github site](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/data/Mus_musculus.NCBIM37.65.gtf.gz) to your local computer and import the gtf.gz file (no need for unzipping) using _Choose local file_ option from the Upload menu
>
>    _Note:_ The numbers in the count files are the number of sequence reads to a specific gene. Only mRNAs that were detected in all samples were selected during the preprocessing of the data.
>
> 5. **Set the datatype** {% icon galaxy-pencil %} of the annotation file to `gtf.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="gtf.gz" %}
>
>    > <question-title></question-title>
>    >
>    > 1. How many mRNAs were detected in all samples?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1.By clicking on the dataset you can see the attributes of the dataset. There are 6,882 lines and a comment line. Thus, there are 6,882 detectable mRNAs in the dataset
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

# Analysis of the differential gene expression

## Identification of the differentially expressed features

To be able to identify differential gene expression between wild-type and p53-null cells and between IR- and mock-treated cells, we need to normalise our data first. In the lecture you have seen TPM and RPKM normalization. Here we use a bit more advance way of normalizing data using the Variance Stabilization and Transformation (VST) method from the DESeq2 package and tool. (see [DESeq2 documentation](http://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#collapsing-technical-replicates)). DESeq2 also runs the Differential Gene Expression (DGE) analysis, which has two basic tasks:

- Estimate the biological variance using the replicates for each condition
- Estimate the significance of expression differences between any two conditions

This expression analysis is estimated from read counts and attempts are made to correct for variability in measurements using replicates, that are absolutely essential for accurate results.

> <details-title>Technical vs biological replicates</details-title>
>
> A technical replicate is an experiment which is performed once but measured several times (e.g. multiple sequencing of the same library). A biological replicate is an experiment performed (and also measured) several times.
>
> In our data, we have 2-4 biological replicates (here called samples) per condition.
>    > <question-title></question-title>
>    >
>    > 1. How many conditions are there in this experiment and what is the number of replicates in each of the conditions?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1.There are 4 conditions
>    > > - Wild-type cells not exposed to IR - 4 replicates
>    > > - Wild-type cells exposed to IR - 4 replicates
>    > > - TP53-/- cells not exposed to IR - 2 replicates
>    > > - TP53-/- cells exposed to IR - 2 replicates
>    > >
>    > {: .solution}
>    {: .question}
{: .details}

Multiple factors with several levels can then be incorporated in the analysis describing known sources of variation (e.g. treatment, tissue type, gender, batches), with two or more levels representing the conditions for each factor. After normalization we can compare the response of the expression of any gene to the presence of different levels of a factor in a statistically reliable way.

In our example, we have samples with two varying factors that can contribute to differences in gene expression:

- Genotype (either wild-type or knock-out)
- Treatment (either IR or mock)

The effect of Ionizing Radiation (IR) is our primary interest. Later on, we will explore the differences in IR response between wild-type and knock-out cells and which genes are dependent on p53 for their response to IR.
We will now assign each sample to the right group

We can now run **DESeq2**:

> <hands-on-title>Determine differentially expressed features</hands-on-title>
>
> 1. {% tool [DESeq2](toolshed.g2.bx.psu.edu/repos/iuc/deseq2/deseq2/2.11.40.8+galaxy0) %} with the following parameters:
>    - *"how"*: `Select datasets per level`
>        - In *"Factor"*:
>           - *"Specify the factor name"*: `Treatment`
>           - In *"1: Factor level"*:
>               - *"Specify the factor level"*: `IR`
>               - In *"Count file(s)"*: `Select all the IR treated count files`. _Note1:_ Use the _Switch to column select_ option to select the files. _Note2:_ The first factor level is compared to the second factor level. In this case IR vs. mock. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20123708.png)
>           - In *"2: Factor level"*:
>               - *"Specify the factor level"*: `Mock`
>               - In *"Count file(s)"*: `Select all the mock treated count files`
>       - {% icon param-repeat %} *"Insert Factor"*
>           - *"Specify the factor name"*: `Genotype`
>               - In *"Factor level"*:
>                    - {% icon param-repeat %} *"Insert Factor level"*
>                        - *"Specify the factor level"*: `KO`
>                        - In *"Count file(s)"*: `Select all the KO count files.`  i.e. those with null in the file name
>                    - {% icon param-repeat %} *"Insert Factor level"*
>                        - *"Specify the factor level"*: `WT`
>                        - In *"Count file(s)"*: `Select all the WT count files` i.e. those with p53 in the file name
>    - *"Files have header?"*: `Yes`
>    - *"Choice of Input data"*: `Count data (e.g. from HTSeq-count, featureCounts or StringTie)`
>    - In *"Advanced options"*:
>        - *"Use beta priors"*: `Yes`
>    - In *"Output options"*:
>        - *"Output selector"*: `Generate plots for visualizing the analysis results`, `Output normalised counts`, `Output VST normalized table`
>
{: .hands_on}


**DESeq2** {% icon tool %} generated 4 outputs:

- A table with the normalized counts for each gene (rows) in the samples (columns)
- A graphical summary of the results, useful to evaluate the quality of the experiment:

    1. A plot of the first 2 dimensions from a principal component analysis

    2. Heatmap of the sample-to-sample distance matrix (with clustering) based on the normalized counts.

        The heatmap gives an overview of similarities and dissimilarities between samples: the color represents the distance between the samples. Dark blue means shorter distance, i.e. closer samples given the normalized counts.


    3. Dispersion estimates: gene-wise estimates (black), the fitted values (red), and the final maximum a posteriori estimates used in testing (blue)

        This dispersion plot is typical, with the final estimates shrunk from the gene-wise estimates towards the fitted estimates. Some gene-wise estimates are flagged as outliers and not shrunk towards the fitted value. The amount of shrinkage can be more or less than seen here, depending on the sample size, the number of coefficients, the row mean and the variability of the gene-wise estimates.

    4. Histogram of *p*-values for the genes in the comparison between the 2 levels of the 1st factor

    5. An [MA plot](https://en.wikipedia.org/wiki/MA_plot):

        This displays the global view of the relationship between the expression change of conditions (log ratios, M), the average expression strength of the genes (average mean, A), and the ability of the algorithm to detect differential gene expression. The genes that passed the significance threshold (adjusted p-value < 0.1) are colored in blue.

- A summary file with the following values for each gene:

    1. Gene identifiers
    2. Mean normalized counts, averaged over all samples from both conditions
    3. Fold change in log2 (logarithm base 2)

       The log2 fold changes are based on the primary factor level 1 vs factor level 2, hence the input order of factor levels is important. Here, DESeq2 computes fold changes of 'treated' samples against 'untreated' from the first factor 'Treatment', *i.e.* the values correspond to up- or downregulation of genes in treated samples.

    4. Standard error estimate for the log2 fold change estimate
    5. [Wald](https://en.wikipedia.org/wiki/Wald_test) statistic
    6. *p*-value for the statistical significance of this change
    7. *p*-value adjusted for multiple testing with the Benjamini-Hochberg procedure, which controls false discovery rate ([FDR](https://en.wikipedia.org/wiki/False_discovery_rate))

    > <tip-title>What are p-values and what are they used for?</tip-title>
    >
    > The p-value is a measure often used to determine whether or not a particular observation possesses statistical significance. Strictly speaking, the p-value is the probability that the data could have arisen randomly, assuming that the null hypothesis is correct. In the concrete case of RNA-Seq, the null hypothesis is that there is no differential gene expression. So a p-value of 0.13 for a particular gene indicates that, for that gene, assuming it is not differentially expressed, there is a 13% chance that any apparent differential expression could simply be produced by random variation in the experimental data.
    >
    > 13% is still quite high, so we cannot really be confident differential gene expression is taking place. The most common way that scientists use p-values is to set a threshold (commonly 0.05, sometimes other values such as 0.01) and reject the null hypothesis only for p-values below this value. Thus, for genes with p-values less than 0.05, we can feel safe stating that differential gene expression plays a role. It should be noted that any such threshold is arbitrary and there is no meaningful difference between a p-value of 0.049 and 0.051, even if we only reject the null hypothesis in the first case.
    >
    > Unfortunately, p-values are often heavily misused in scientific research, enough so that Wikipedia provides a [dedicated article](https://en.wikipedia.org/wiki/Misuse_of_p-values) on the subject. See also [this article](https://fivethirtyeight.com/features/not-even-scientists-can-easily-explain-p-values/) (aimed at a general, non-scientific audience).
    {: .tip}

For more information about **DESeq2** and its outputs, you can have a look at the [**DESeq2** documentation](https://www.bioconductor.org/packages/release/bioc/manuals/DESeq2/man/DESeq2.pdf).

- An output file with Normalized counts
- An output file with VST-Normalized counts (VST stands for Variance Stabilization and Transformation; this is the preferred normalization method in DESeq2 and we will use these counts later on for PCA and clustering

Use the **filter** {% icon tool %} tool on the DESeq2 result file to identify the genes with absolute fold-change greater than 2. Use the condition: abs(c3)>2. Number of header lines to skip: 1. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20130228.png). And use the **sort** tool to identify the genes with the largest fold-change (most upregulated). See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20132050.png)

> <question-title></question-title>
>
> 1. What does it mean when an mRNA has an absolute 2logFC greater than 2?
> 2. How many mRNAs have an absolute 2logFC greater than 2?
> 3. What is the mRNA that is most upregulated in IR- vs mock-treated cells and what is its false discovery rate (the chance that this is a false positive)?
>
> > <solution-title></solution-title>
> >
> > 1. That the genes is 2^2 = 4 times higher or lower expressed in the cells treated with IR than in mock-treated cells
> > 2. 666 (!)
> > 3. ENSMUSG00000000308; FDR is in p-adj column. It is 1.3*10^-26
> >
> {: .solution}
{: .question}

## Annotation of the DESeq2 results

The ID for each gene is something like ENSMUSG00000026581, which is an ID from the Ensembl gene database. These IDs are unique but sometimes we prefer to have the gene names, even if they may not reference an unique gene (e.g. duplicated after re-annotation). But gene names may hint already to a function or they help you to search for desired candidates. We would also like to display the location of these genes within the genome. We can extract such information from the annotation file which we uploaded before.

> <hands-on-title>Annotation of the DESeq2 results</hands-on-title>
>
>
> 1. {% tool [Annotate DESeq2/DEXSeq output tables](toolshed.g2.bx.psu.edu/repos/iuc/deg_annotate/deg_annotate/1.1.0) %} with:
>    - {% icon param-file %} *"Tabular output of DESeq2/edgeR/limma/DEXSeq"*: the `DESeq2 result file` (output of **DESeq2** {% icon tool %})
>    - *"Input file type"*: `DESeq2/edgeR/limma`
>    - {% icon param-file %} *"Reference annotation in GFF/GTF format"*: imported gtf `Annotation file`
>    - See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20132922.png)
{: .hands_on}

The generated output is an extension of the previous file:

1. Gene identifiers
2. Mean normalized counts over all samples
3. Log2 fold change
4. Standard error estimate for the log2 fold change estimate
5. Wald statistic
6. *p*-value for the Wald statistic
7. *p*-value adjusted for multiple testing with the Benjamini-Hochberg procedure for the Wald statistic
8. Chromosome
9. Start
10. End
11. Strand
12. Feature
13. Gene name

> <question-title></question-title>
>
> 1. What is the Gene Symbol of the most overexpressed gene
> 2. On which chromosome is the most over-expressed gene located?
>
> > <solution-title></solution-title>
> >
> > 1. ENSMUSG00000000308 (the top-ranked gene with the highest positive log2FC value)
> > 2. It is located on the reverse strand of chromosome 2, between 121183449 bp and 121189473 bp.
> {: .solution}
{: .question}

The annotated table contains no column names, which makes it difficult to read. We would like to add them before going further.

> <hands-on-title>Add column names</hands-on-title>
>
> 1. In the Upload menu, create a new file using the _Paste/Fetch data_ option form the Upload menu, by directly pasting in the line below in the white area, and call it `header` instead of `New File`
>
>    ```text
>    GeneID	Base mean	log2(FC)	StdErr	Wald-Stats	P-value	P-adj	Chromosome	Start	End	Strand	Feature	Gene name
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="header" format="tabular" %}
>    See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20134800.png)
>
> 3. {% tool [Concatenate datasets](cat1) %} to add this header line to the **Annotate** output:
>    - {% icon param-file %} *"Concatenate Dataset"*: the `header` dataset
>    - *"Dataset"*
>       - Click on {% icon param-repeat %} *"Insert Dataset"*
>         - {% icon param-file %} *"select"*: output of **Annotate** {% icon tool %}
>
> 4. Rename the output to `Annotated DESeq2 results`
{: .hands_on}

## Create Volcano plot to visualise differentially expressed genes

> <hands-on-title></hands-on-title>
> 1. Select the tool **Volcano Plot**
>    In a Volcano plot, the 2logFC (x-axis) is plotted against the -log10(P-value).
>    - Choose the right column numbers. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20162602.png). Change the significant threshold to 0.05 and LogFC 2 as as thresholds to colour.
>    - Put in correct titles under _Plot options_.
>
> 2. Repeat the DESeq2 analysis but now with `Genotype` as first factor (contrasting KO and WT cells). Repeat the Volcano plot.
>
> > <question-title></question-title>
> >
> > 1. Is IR causing more up- or more downregulation of gene expression?
> > 2. Does the Treatment or the Genotype have a bigger effect on gene expression
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are more mRNA upregulated than downregulated. The upregulated genes can be found on the right side of the plot.
> > > 2. There are more mRNA with high FC and strong significance in the comparison between IR and mock-treated cells than between KO and WT cells.
> > {: .solution}
> {: .question}
{: .hands_on}



# Functional enrichment analysis of the DE genes

We have extracted genes that are differentially expressed in IR- vs. mock-treated samples. Now, we would like to know if the differentially expressed genes are enriched transcripts of genes which belong to more common or specific categories in order to identify biological functions that might be impacted.

## Gene Ontology analysis

[Gene Ontology (GO)](http://www.geneontology.org/) analysis is widely used to reduce complexity and highlight biological processes in genome-wide expression studies. However, standard methods give biased results on RNA-Seq data due to over-detection of differential expression for long and highly-expressed transcripts.

[**goseq**](https://bioconductor.org/packages/release/bioc/vignettes/goseq/inst/doc/goseq.pdf) ({% cite young2010gene %}) provides methods for performing GO analysis of RNA-Seq data while taking length bias into account. **goseq** could also be applied to other category-based tests of RNA-Seq data, such as KEGG pathway analysis, as discussed in a further section.

**goseq** needs 2 files as inputs:

- A tabular file with the differentially expressed genes from all genes assayed in the RNA-Seq experiment with 2 columns:
  - the Gene IDs (unique within the file), in uppercase letters
  - a boolean indicating whether the gene is differentially expressed or not (`True` if differentially expressed or `False` if not). We will use abs(2LogFC) > 2 as a criterion
- A file with information about the length of a gene to correct for potential length bias in differentially expressed genes

> <hands-on-title> Prepare input for goseq </hands-on-title>
>
> 1. Use the **Gene length and GC content** tool on the _Annotation file_ (gtf format). See this [screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20210701.png)
>    - *"Analysis to perform"*: `gene lengths only`
> 2. Merge the gene length and the annotated DESeq2 ouput file **DESeq2 result file** with primary factor Treatment using the **Join two datasets side by side** tool. In *"Keep the header lines"*: `No`. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20214135.png)
> 3. Use the **Compute on rows** tool to create a column with TRUE and FALSE using the following expression: `bool(float(c3)>2)`. In the *"Error handling"* choose in *"Autodetect column types"* `No` and *"Fail on references to non-existent columns"* `No` and *"If an expression cannot be computed for a row"* choose `Fill in a replacement value` and Replacement value `False`. See this [screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20222441.png)
> 4. {% tool [Cut](Cut1) %} columns from a table with the following parameters:
>    - *"Cut columns"*: `c1,c9`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the output generated in step 3
> 5. Rename the output to `Gene IDs and differential expression - IR`
> 6. {% tool [Cut](Cut1) %} columns from a table with the following parameters:
>    - *"Cut columns"*: `c1,c8`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the output generated in step 3
> 7. Rename the output to `Gene length differential expression - IR`
{: .hands_on}

We have now the two required input files for goseq.

> <hands-on-title>Perform GO analysis</hands-on-title>
>
> 1. {% tool [goseq](toolshed.g2.bx.psu.edu/repos/iuc/goseq/goseq/1.50.0+galaxy0) %} with
>    - *"Differentially expressed genes file"*: `Gene IDs and differential expression`
>    - *"Gene lengths file"*: `Gene IDs and length`
>    - *"Gene categories"*: `Get categories`
>       - *"Select a genome to use"*: `mouse (mm10)`
>       - *"Select Gene ID format"*: `Ensembl Gene ID`
>       - *"Select one or more categories"*: `GO: Cellular Component`, `GO: Biological Process`, `GO: Molecular Function`
>    - In *"Output Options"*
>      - *"Output Top GO terms plot?"*: `Yes`
>      - *"Extract the DE genes for the categories (GO/KEGG terms)?"*: `Yes`
>
{: .hands_on}

**goseq** generates with these parameters 3 outputs:

1. A table (`Ranked category list - Wallenius method`) with the following columns for each GO term:

    1. `category`: GO category
    2. `over_rep_pval`: *p*-value for over-representation of the term in the differentially expressed genes
    3. `under_rep_pval`: *p*-value for under-representation of the term in the differentially expressed genes
    4. `numDEInCat`: number of differentially expressed genes in this category
    5. `numInCat`: number of genes in this category
    6. `term`: detail of the term
    7. `ontology`: MF (Molecular Function - molecular activities of gene products), CC (Cellular Component - where gene products are active), BP (Biological Process - pathways and larger processes made up of the activities of multiple gene products)
    8. `p.adjust.over_represented`: *p*-value for over-representation of the term in the differentially expressed genes, adjusted for multiple testing with the Benjamini-Hochberg procedure
    9. `p.adjust.under_represented`: *p*-value for under-representation of the term in the differentially expressed genes, adjusted for multiple testing with the Benjamini-Hochberg procedure

    To identify categories significantly enriched/unenriched below some p-value cutoff, it is necessary to use the adjusted *p*-value.

    > <question-title></question-title>
    >
    > 1. How many GO terms are over-represented with an adjusted P-value < 0.05? How many are under-represented?
    > 2. Do you find evidence that p53 plays a role in the response to IR?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 5 GO terms (4 from the Biological Processes category, 1 from the Molecular Function category) are over-represented.
    > > 2. Signal transduction by p53 class mediator is second in rank and contains 19 differentially expressed genes.
    > >    {% tool [Filter data on any column using simple expressions](Filter1) %} on c8 (adjusted p-value for over-represented GO terms) and c9 (adjusted p-value for under-represented GO terms)
    > >
    > > 2. For over-represented, 50 BP, 5 CC and 5 MF and for under-represented, 5 BP, 2 CC and 0 MF
    > >
    > >    {% tool [Group data](Grouping1) %} on column 7 (category) and count on column 1 (IDs)
    > >
    > {: .solution}
    {: .question}

2. A graph with the top 10 over-represented GO terms

    > <question-title></question-title>
    >
    > What is the x-axis? How is it computed?
    >
    > > <solution-title></solution-title>
    > >
    > > The x-axis is the percentage of genes in the category that have been identified as differentially expressed: $$100 \times \frac{numDEInCat}{numInCat}$$
    > >
    > {: .solution}
    {: .question}

3. A table with the differentially expressed genes (from the list we provided) associated to the GO terms (`DE genes for categories (GO/KEGG terms)`)

> <comment-title>Advanced tutorial on enrichment analysis</comment-title>
>
> In this tutorial, we covered GO enrichment analysis with **goseq**. You can also use the same method to identify metabolic and signal transduction pathways from the KEGG database (do not select plotting the Top GO pathways in that case). To learn other methods and tools for gene set enrichment analysis, please have a look at the ["RNA-Seq genes to pathways"]({% link topics/transcriptomics/tutorials/rna-seq-genes-to-pathways/tutorial.md %}) tutorial.
{: .comment}


# Visualisation of high-dimensional datasets
In the second part of this computer assignment we will use commonly used techniques to identify the main sources of variation and to evaluate which samples and sample groups are most similar. We will continue using the same dataset but first filter for the most variable genes. This is for computational efficiency and to reduce the effect of noise.


## Filter for most variable genes
For filtering the most variable genes, we will compute the coefficient of variation (CV). This is defined as the standard deviation divided by the mean.
> <hands-on-title> Filter for most variable genes </hands-on-title>
>
> 1. Use the the **Table compute** tool to calculate the standard deviation (std)
>    - *"Table"*: `VST normalized data`
>    - *"Input data has"*: `Column names on the first row` and `Row names on the first column`
>    - *"Type of table operation "*: `Compute expression across rows or columns`
>       - *"Calculate"*: `Custom`
>       - *"Custom function on vec"*: `vec.std()`
>       - *"For each"*: `row`
>       - *"Output formatting options"*: keep everything checked. Output should have column and row headers
>    - See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20085500.png)
> 2. Use the the **Table compute** tool to calculate the mean (mean)
>    - *"Table"*: `VST normalized data`
>    - *"Input data has"*: `Column names on the first row` and `Row names on the first column`
>    - *"Type of table operation "*: `Compute expression across rows or columns`
>       - *"Calculate"*: `Custom`
>       - *"Custom function on vec"*: `vec.mean()`
>       - *"For each"*: `row`
>       - *"Output formatting options"*: keep everything checked. Output should have column and row headers
> 3. Use the **Join two files** tool to merge the normalised count data with the standard deviation and mean datasets (two steps)
>    -  *"First line is a header line"*: `Yes`
> 4. Use the **Compute on row** tool to calculate the CV. Expression `c14/c15`. Label the new row as `CV`
> 5. Use the **Sort** tool to sort the genes based on CV in descending order. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20104444.png)
>    - *"Number of header lines"*: `1`
>    - *"Column selections"*
>       - *"on column"*: `16`
>       - *"in"*: `decsending order`
>       - *"Flavour"*: `Fast numeric sort (-n)`
> 6. Select the first 2000 rows using the **Select first** tool. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20105700.png)
> 7. Select only the columns with data (and the first column with the gene identifiers; columns 1-13) using the **Cut** tool
> 8. Relabel the file as `Input data for PCA and clustering`
>
{: .hands_on}


## Principal Component Analysis (PCA)
PCA is a dimension reduction technique to identify the main sources of variation.
For this part of the assignment we will use the _multivariate_ tool{% icon tool %}. There is a separate [tutorial](https://training.galaxyproject.org/topics/metabolomics/tutorials/lcms-dataprocessing/tutorial.html#bibliography) for the **multivariate** tool applied to metabolomics data.

The tool needs a sampleMetadata file where the different groups / factors are defined. You will be able to colour the PCA score plots according to those groups.
The tool needs a featureMetadata file with the names (and optional characteristics) of the genes.

### Upload metadata

> <hands-on-title> Upload metadata</hands-on-title>
>
> 1. Create the featureMetadata file using the **cut** tool on the `Input data for PCA and clustering` object
>    - *"Cut columns"*: c1
> 2. Import the metadata files from our [github repository](https://github.com/pacthoen/BMW2_RNA_clust_vis) using the Data Upload menu (top left) and the button _Paste/Fetch Data_.
>
>    ```text
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/GSE71176_sample_metadata.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    *Note*: Make sure that the order of the sample names in the data matrix and the sampleMetadata file are the same!
>
> 3. **Set the datatype** {% icon galaxy-pencil %} to `Tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="Tabular" %}
>
> 4. Run {% tool [Multivariate](toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10) %} with the following parameters, according to this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20124140.png):
>    - {% icon param-file %} *"Data matrix file"*: `Input data for PCA and clustering`
>    - {% icon param-file %} *"Sample metadata file"*: Your uploaded metadata
>    - {% icon param-file %} *"Variable metadata file"*: `Filtered_variableMetadata` (output of the last _cut_ job)
>    - *"Number of predictive components"*: `2`
>    - *"Advanced graphical parameters"*: `Full parameter list`
>        - *"Sample colors"*: `Treatment`
>        - *"Amount by which plotting text should be magnified relative to the default"*: `0.4`
>    - *"Advanced computational parameters"*: `Use default`
>
{: .hands_on}

> <tip-title>What if I get an error?</tip-title>
>
> If the multivariate analysis tool gives an error, it may have to do with the fact that the column headers do not exactly match with the metadata or that there is another problem with the input files. You may solve this by first running the _*Check_format*_ tool and run the analysis again with the output files from that tool.
{: .tip}

Among the four output files generated by the tool, the one we are interested in is the *Multivariate_figure.pdf* file.
The PCA scores plot representing the projection of samples on the two first components is located at the bottom left corner. The PCA loadings plot is located at the bottom right corner.

> <question-title></question-title>
>
> 1. What is the cumulative percentage explained by the first two components.
> 2. Which one of the two factors in the experiment has the largest effect on the gene expression levels: the genotype (TP53-/- or wild-type) or the exposure to IR? Play with the plot colors and shapes and create a figure that nicely visualizes this.
>
>
> > <solution-title></solution-title>
> >
> > 1.  The score plots contains the percentage of variance explained in the axis titles. Principal component t1  explains 51% of total variability and t2 18% of total variability. The cumulative variance explained by the first two components is 51+18 = 69%.
> > 2. The t1 nicely separates the IR_treated cells from the mock-treated cells. t1 explains most of the variation thus IR treatment explains most of the variation. The t2 mainly separates the KO cells treated with IR, suggesting that KO cells respond differently to IR than WT cells, which makes sense because p53 is important for the response to IR.
> {: .solution}
{: .question}

> <hands-on-title> Repeat the analysis</hands-on-title>
>
> 1. Repeat the analysis but now with four principal components and plot principal component 3 at the x-axis (abscissa) and component 4 at the y-axis (ordinate)
> 2. Optional: one can also OPLS (Orthogonal Partial Least Squares) which is a supervised form of multivariate analysis
{: .hands_on}

## Clustering

We will use the **heatmap2** tool to perform the clustering of samples based on their mRNA expression patterns and evaluate the influence of different clustering algorithms (average linkage vs single linkage) and distance measures (Euclidean distance or Pearson correlation). See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20152150.png)

> <hands-on-title>Clustering with Heatmap2 </hands-on-title>
>
> 1. {% tool [heatmap2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1) %} with the following parameters:
>    - *"Input"*: `Input data for PCA and clustering`
>    - *"Plot title"*: indicate the clustering algorithm and the distance measure used
>    - *"Data transformation"*: `Plot the data as is`. Because the data has already been VST normalized
>    - *"Compute Z-scores prior to clustering"*: `Compute on rows`. This brings all the genes on the same scale (mean zero and standard deviation. If you do not this (you can check!) the clustering will group high expressed genes together and low expressed together, whereas it is more interesting to cluster genes based on the differences seen between samples.
>    - *"Distance method"*: `Euclidean` OR `Pearson`
>    - *"Clustering method"*: `Average (=UPGMA)` OR `Single`
>    - *"Type of colormap to use"*: `Gradient with 3 colors`
>
>
> > <question-title></question-title>
> >
> > 1. How does the clustering of samples relate to the PCA results?
> > 2. Which differences do you observe between different clustering algorithms and different distance measures?
> > 3. Indicate the cluster of genes for which their IR response is p53-dependent. Are these genes mostly up- or downregulated after IR?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. Again, the IR-exposed cells cluster separately from the mock-exposed cells.
> > >   There is not much difference between the TP53-/- and wild-type mock-exposed cells, whilst the IR-exposed TP53-/- and wild-type cells cluster separately.
> > > 2. The clustering is not very dependent on the Distance measure used. However, with single linkage, it is much more difficult to identify discrete clusters, because the single, closest feature is added each time to an already existing cluster. Average and complete linkage methods usually start with finding the closest features that make up distinct clusters.
> > > 3. There is a cluster of genes that is only upregulated in the cells that are wild-type for p53 and not in IR-treated knockout cells. These mRNAs are dependent on p53 expression for their upregulation after exposure to IR.
> > {: .solution}
> {: .question}
>
{: .hands_on}

# Conclusion

In this computer assignment, we have analyzed real RNA sequencing data to extract useful information, such as which genes are up or downregulated by IR. Your complete workflow can be extracted using the _Worklows_ button. Do this according to this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20155239.png).

{% snippet faqs/galaxy/workflows_extract_from_history.md %}

You can always rerun and adapt your entire workflow. A workflow as you could have obtained it can be found [on this tutorial's workflow page](./workflows/)
The output of this computer assignment can also be found in [this folder](https://github.com/pacthoen/BMW2_RNA_clust_vis/upload/main/results) on the github site.


topic: transcriptomics
tutorial: rna-seq-counts-to-viz-in-r
title: RNA Seq Counts to Viz in R
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 84557990c01a68c2e44f34706a00ce4e
last_modified: 2026-02-26T21:12:18.571729
zenodo_link: https://zenodo.org/record/3477564/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-counts-to-viz-in-r/tutorial.html

content:
This tutorial will show you how to visualise RNA Sequencing Counts with R

{% include topics/data-science/tutorials/r-basics/tutorial_origin.md %}

With RNA-Seq data analysis, we generated tables containing list of DE genes, their expression, some statistics, etc. We can manipulate these tables using Galaxy, as we saw in some tutorials, e.g. ["Reference-based RNA-Seq data analysis"]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), and create some visualisations.

Sometimes we want to have some customizations on visualization, some complex table manipulations or some statistical analysis. If we can not find a Galaxy tools for that or the right parameters, we may need to use programming languages as R or Python.

> <tip-title>Prerequisites</tip-title>
> It is expected that you are already somewhat familiar with the R basics (how to create variables, create and access lists, etc.) If you are not yet comfortable with those topics, we recommend that you complete the requirements listed at the start of this tutorial first, before proceeding.
>
> If you are starting with this tutorial, you will need to import a dataset:
>
> > <hands-on-title>Using datasets from Galaxy</hands-on-title>
> >
> > 1. Upload the following URL to Galaxy:
> >    ```
> >    https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular
> >    ```
> >
> > 2. Note the history ID of the dataset in your Galaxy history
> >
> > 3. {% tool [RStudio](interactive_tool_rstudio) %} in Galaxy provides some special functions to import and export from your history:
> >
> >    ```R
> >    annotatedDEgenes <- read.csv(gx_get(2), sep="\t") # will import dataset number 2 from your history, use the correct ID for your dataset.
> >    ```
> >
> >
> {: .hands_on}
>
> > <hands-on-title>Using datasets without Galaxy</hands-on-title>
> > 1. Read the tabular file into an object called `annotatedDEgenes`:
> >
> >    ```R
> >    ## read in a CSV file and save it as 'annotatedDEgenes'
> >    annotatedDEgenes <- read.csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep="\t")
> >    ```
> >
> >  {: .tip}
> {: .hands_on}
{: .tip}

In this tutorial, we will take the list of DE genes extracted from DESEq2's output that we generated in the ["Reference-based RNA-Seq data analysis" tutorial]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), manipulate it and create some visualizations.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

# Visualization

In RNA-seq data analysis and other omics experiments, visualization are an important step to check the data, their quality and represent the results found for publication.

`ggplot2` is a plotting package that makes it simple to create complex plots from data in a data frame. It provides a more programmatic interface for specifying what variables to plot, how they are displayed, and general visual properties. Therefore, we only need minimal changes if the underlying data change or if we decide to change from a bar plot to a scatter plot. This helps in creating publication quality plots with minimal amounts of adjustments and tweaking.

`ggplot2` graphics are built step by step by adding new elements, i.e. layers:

```R
ggplot(data = <DATA>, mapping = aes(<MAPPINGS>)) +
  <GEOM_FUNCTION>() +
  <GEOM_FUNCTION>() +
  <GEOM_FUNCTION>()
```

Adding layers in this fashion allows for extensive flexibility and customization of plots.

## Volcano plot

> <hands-on-title>First plot, step by step</hands-on-title>
>
> 1. Load `ggplot2`
>
>     ```R
>     library(ggplot2)
>     ```
>
>    > <tip-title>Didn't work?</tip-title>
>    > You might need to install ggplot2 first
>    >
>    > ```R
>    > > install.packages("ggplot2")
>    > ```
>    {: .tip}
>
> 3. Bind a plot to a specific data frame
>
>    ```R
>    ggplot(data = annotatedDEgenes)
>    ```
>
>    A new window will open in the bottom-right panel
>
> 4. Define a mapping (using the `aes` aesthetic function), by selecting the variables to be plotted and specifying how to present them in the graph, e.g. as x/y positions or characteristics such as size, shape, color, etc.
>
>     ```R
>     ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = P.value))
>     ```
>
>     X-axis is now the log2 FC and and Y-axis the p-value
>
>     > <comment-title>Data format</comment-title>
>     >
>     > Some `ggplot2` functions need data in the 'long' format:
>     > - a column for every dimension
>     > - a row for every observation
>     >
>     > Well-structured data will save you lots of time when making figures with `ggplot2`.
>     {: .comment}
>
> 5. Add information about the graphical representations of the data in the plot (points, lines, bars) using `geoms` function added via the `+` operator
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = P.value)) +
>          geom_point()
>    ```
>
>    > <comment-title>Graphical representations of the data in the plot</comment-title>
>    >
>    > `ggplot2` offers many different `geoms`; we will use some common ones today, including:
>    > - `geom_point()` for scatter plots, dot plots, etc.
>    > - `geom_boxplot()` for, well, boxplots!
>    > - `geom_line()` for trend lines, time series, etc.
>    {: .comment}
{: .hands_on}

We should obtain our first ggplot2 plot:

![Our first ggplot2 plot](../../images/rna-seq-counts-to-viz-in-r/firstPlot.png)

This plot is called a volcano plot, a type of scatterplot that shows statistical significance (P value) versus magnitude of change (fold change). The most upregulated genes are towards the right, the most downregulated genes are towards the left, and the most statistically significant genes are towards the top. With this plot, we can then quickly identify genes with large fold changes that are also statistically significant, i.e. probably the most biologically significant genes.

The current version of the plot is not really informative, mostly due to the high number of p-values close to zero. In order to resolve this, we will apply `-log10()` to the y-axis values.

> <hands-on-title>Volcano plot with log values on y-axis</hands-on-title>
>
> 1. Create volcano plot with log values on the y-axis
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value))) +
>         geom_point()
>    ```
{: .hands_on}

![Volcano Plot version 0](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v0.png)

> <question-title>Categorising expression levels</question-title>
>
> 1. Why are there no points with log2 FC between -1 and 1?
>
> > <solution-title></solution-title>
> >
> > The input file we used are the 130 genes with a significant adjusted p-values (below < 0.05) and an absolute fold change higher than 2 (log2 FC < -1 or log 2 FC > 1).
> >
> {: .solution}
{: .question}

> <comment-title>The `+` sign</comment-title>
>
> The `+` in the `ggplot2` package is particularly useful because it allows you to modify existing `ggplot` objects. This means you can easily set up plot templates and conveniently explore different types of plots, so the above plot can also be generated with code like this:
>
> ```R
> # Assign plot to a variable
> de_genes_plot <- ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value)))
>
> # Draw the plot
> de_genes_plot +
>     geom_point()
> ```
>
> The `+` sign used to add new layers must be placed at the end of the line containing the previous layer. If, instead, the `+` sign is added at the beginning of the line containing the new layer, `ggplot2` will not add the new layer and will return an error message.
>
> ```R
> # This is the correct syntax for adding layers
> > de_genes_plot +
>       geom_point()
>
> # This will not add the new layer and will return an error message
> > de_genes_plot
>     + geom_point()
> ```
{: .comment}

> <comment-title>Some extra comments</comment-title>
>
> Anything you put in the `ggplot()` function can be seen by any `geom` layers that you add (i.e., these are universal plot settings). This includes the `x-` and `y-axis` mapping you set up in `aes()`.
>
> You can also specify mappings for a given `geom` independently of the mappings defined globally in the `ggplot()` function.
{: .comment}

Building plots with `ggplot2` is typically an iterative process. We start by defining the dataset we'll use, lay out the axes, and choose a `geom`. We can now modify this plot to extract more information from it.

> <hands-on-title>Format plot</hands-on-title>
>
> 1. Add transparency (`alpha`) to avoid overplotting
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value))) +
>         geom_point(alpha = 0.5)
>    ```
>
>    ![Volcano Plot version 1](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v1.png)
>
> 2. Add colors for all the points
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value))) +
>        geom_point(alpha = 0.5, color = "blue")
>    ```
>
>    ![Volcano Plot version 2](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v2.png)
>
> 3. Color point based on their strand
>
>     ```R
>     ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>         geom_point(alpha = 0.5)
>     ```
>
>     ![Volcano Plot version 3](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v3.png)
>
>     We use a vector as an input to the argument color and `ggplot2` provides a different color corresponding to different values in the vector.
>
> 4. Add axis labels
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>        geom_point(alpha = 0.5) +
>          labs(x = "log2(Fold Change)",
>               y = "-log10(P-Value)")
>    ```
{: .hands_on}

We have now a nice Volcano plot:

![Volcano Plot version 5](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v5.png)

> <question-title>Standard error over the `-log10` of the adjusted `P-value`</question-title>
>
> Create a scatter plot of the standard error over the `-log10` of the adjusted `P-value` with the chromosomes showing in different colors. Make sure to give your plot relevant axis labels.
>
> > <solution-title></solution-title>
> >
> > ```
> > ggplot(data = annotatedDEgenes, aes(x = -log10(P.adj), y = StdErr, color = Chromosome)) +
> >   geom_point() +
> >      labs(x = "-log10(P.adj)",
> >           y = "StdErr")
> > ```
> >
> > ![Exercise Plot 1](../../images/rna-seq-counts-to-viz-in-r/ggplotExercise01.png)
> {: .solution}
{: .question}

## Volcano plot by chromosomes

`ggplot2` has a special technique called faceting that allows us to split one plot into multiple plots based on a factor included in the dataset. We will use it to split our volcano plot into five panels, one for each chromosome.

> <hands-on-title>Volcano plot by chromosomes</hands-on-title>
>
> 1. Split volcano plot by chromosome using `facet_grid`
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(. ~ Chromosome)
>    ```
>
>    ![Volcano Plot version 6](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v6.png)
>
> 2. Split volcano plot by chromosome using `facet_grid` with plots stacked vertically
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(Chromosome ~ .)
>    ```
>
>    ![Volcano Plot version 7](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v7.png)
>
>    The `facet_grid` geometry allows to explicitly specify how we want your plots to be arranged via formula notation (`rows ~ columns`). The `.` can be used as a placeholder that indicates only one row or column).
>
> 3. Add white background using `theme_bw()` to make the plot more readable when printed
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(Chromosome ~ .) +
>      theme_bw()
>    ```
>
>    > <details-title>`ggplot2` themes</details-title>
>    >
>    > In addition to `theme_bw()`, which changes the plot background to white, `ggplot2` comes with several other themes which can be useful to quickly change the look of your visualization.
>    >
>    > - `theme_minimal()` and `theme_light() `are popular
>    > - `theme_void()` can be useful as a starting point to create a new hand-crafted theme
>    >
>    > The complete list of themes is available on the [ggplot2 website](https://ggplot2.tidyverse.org/reference/ggtheme.html). The `ggthemes` package provides a wide variety of options (including an Excel 2003 theme). The [`ggplot2` extensions website](https://www.ggplot2-exts.org/) provides a list of packages that extend the capabilities of `ggplot2`, including additional themes.
>    {: .details}
>
> 4. Remove the grid
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(Chromosome ~ .) +
>      theme_bw() +
>      theme(panel.grid = element_blank())
>    ```
>
{: .hands_on}

![Volcano Plot version 8](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v8.png)

> <question-title>Standard error over the `-log10` of the adjusted `P-value`</question-title>
>
> Create a scatter plot of the standard error over the `-log10` of the adjusted `P-value` with the chromosomes showing in different colors and one facet per strand. Make sure to give your plot relevant axis labels.
>
> > <solution-title></solution-title>
> >
> > ```
> > ggplot(data = annotatedDEgenes, aes(x = -log10(P.adj), y = StdErr, color = Chromosome)) +
> >  geom_point() +
> >  labs(x = "-log10(P.adj)",
> >       y = "StdErr") +
> >  facet_grid(Strand ~ .)
> > ```
> >
> > ![Exercise Plot 2](../../images/rna-seq-counts-to-viz-in-r/ggplotExercise02.png)
> >
> {: .solution}
{: .question}

## Barplot of the number of DE genes

We would like now to make a barplot showing the number of differentially expressed genes.

> <hands-on-title>Barplot of the number of DE genes</hands-on-title>
>
> 1. Create a barplot with `geom_bar` function of the number of DE genes for each feature with one plot per chromosome
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = Feature, fill = Chromosome)) +
>    geom_bar() +
>    facet_grid(Chromosome ~ .)
>    ```
{: .hands_on}

![Bar plot 1](../../images/rna-seq-counts-to-viz-in-r/barPlot_v1.png)

> <question-title>Remove the legend</question-title>
>
> The chromosome is labeled on the individual plot facets so we don't need the legend. Use the help file for `geom_bar` and any other online resources to remove the legend from the plot.
>
> > <solution-title></solution-title>
> >
> > ```
> > ggplot(data = annotatedDEgenes, aes(x = Feature, fill = Chromosome)) +
> >  geom_bar(show.legend = F) +
> >  facet_grid(Chromosome ~ .)
> > ```
> > ![Bar plot 2](../../images/rna-seq-counts-to-viz-in-r/barPlot_v2.png)
> >
> {: .solution}
{: .question}


> <question-title>Create a R script with plots</question-title>
>
> Take another few minutes to either improve one of the plots generated in this exercise or create a beautiful graph of your own (using the [RStudio `ggplot2` cheat sheet](https://raw.githubusercontent.com/rstudio/cheatsheets/main/data-visualization-2.1.pdf) for inspiration).
>
> Here are some ideas:
> 1. Change the size or shape of the plotting symbol
> 2. Change the name of the legend
> 3. Change the label of the legend
> 4. Use a different color palette (see the [cookbook here](http://www.cookbook-r.com/Graphs/Colors_(ggplot2)/)).
{: .question}

# Conclusion


Data manipulation and visualization are important parts of any RNA-Seq analysis. Galaxy provides several tools for that as explained in several tutorials:

- [Reference-based RNA-Seq]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %})
- [Visualization of RNA-seq results with heatmap2]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-heatmap2/tutorial.md %})
- [Visualization of RNA-seq results with Volcano Plot]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %})

But sometimes we need more customization and then need to use programming languages as R or Python.

Working with a programming language (especially if it's your first time) often feels intimidating, but the rewards outweigh any frustrations. An important secret of coding is that even experienced programmers find it difficult and frustrating at times – so if even the best feel that way, why let intimidation stop you? Given time and practice you will soon find it easier and easier to accomplish what you want.

Finally, we won't lie; R is not the easiest-to-learn programming language ever created. So, don't get discouraged! The truth is that even with the modest amount of R covered today, you can start using some sophisticated R software packages, and have a general sense of how to interpret an R script. Get through these lessons, and you are on your way to being an accomplished R user!


topic: transcriptomics
tutorial: goenrichment
title: GO Enrichment Analysis
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: eba748051f52eacd124e5fc8efa4f6f5
last_modified: 2026-02-26T21:12:18.549570
zenodo_link: https://zenodo.org/record/2565417
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/goenrichment/tutorial.html

content:
When we have a large list of genes of interest, such as a list of differentially expressed genes obtained from an RNA-Seq experiment, how do we extract biological meaning from it?

One way to do so is to perform functional enrichment analysis. This method consists of applying statistical tests to verify if genes of interest are more often associated to certain biological functions than what would be expected in a random set of genes. In this tutorial you will learn about enrichment analysis and how to perform it.

## What is the Gene Ontology?
The [Gene Ontology](http://www.geneontology.org/) (GO) is a structured, controlled vocabulary for the classification of gene function at the molecular and cellular level. It is divided in three separate sub-ontologies or GO types: biological process (e.g., signal transduction), molecular function (e.g., ATPase activity) and cellular component (e.g., ribosome). These sub-ontologies are structured as directed acyclic graphs (a hierarchy with multi-parenting) of GO terms.

![GO Example](../../images/transcriptomics_images/goenrichment_GOexample.png "QuickGO - http://www.ebi.ac.uk/QuickGO")

The GO Ontology, like other ontologies, are usually coded in the [OBO](http://www.geneontology.org/faq/what-obo-file-format) or the [OWL](http://www.geneontology.org/faq/what-owl-file) formats. It can be downloaded from the [Gene Ontology website](http://geneontology.org/page/download-ontology) or from the [OBO foundry](http://www.obofoundry.org/). You can also find in Galaxy tools that allow you to manipulate and extract information from OBO files, but this is outside the scope of this tutorial.

> <comment-title></comment-title>
> Take note of when and where you obtained your ontology file, as these are constantly being updated.
>
{: .comment}


## What are GO annotations?
Genes are associated to GO terms via GO annotations. Each gene can have multiple annotations, even of the same GO type. An important notion to take into account when using GO is that, according to the **true path rule**, a gene annotated to a term is also implicitly annotated to each ancestor of that term in the GO graph. GO annotations have evidence codes that encode the type of evidence supporting them: only a small minority of genes have experimentally verified  annotations; the large majority have annotations inferred electronically based on sequence homology or known patterns.

GO annotations can be obtained from the [Gene Ontology website](http://geneontology.org/page/download-go-annotations), or from species-specific databases. One useful resource to obtain GO annotations is [Ensembl biomart](http://www.ensembl.org/biomart/martview). Again, take note to when and from where you obtained your annotations. For example, if you obtained your data from Ensembl, record the release you used.

> ### Overview
>
> In this tutorial, we will deal with:
>
> 1. Functional Enrichment Analysis
> 2. Methods to simplify the results
> 3. Interpretation of the results
> {:toc}
>
{: .agenda}


# Functional Enrichment Analysis

To perform functional enrichment analysis, we need to have:
- A set of genes of interest (e.g., differentially expressed genes): **study set**
- A set with all the genes to consider in the analysis: **population set** (which must contain the study set)
- **GO annotations**, associating the genes in the population set to GO terms
- The **GO ontology**, with the description of GO terms and their relationships

For each GO term, we need to count the frequency (**k**) of genes in the study set (**n**) that are associated to the term, and the frequency (**K**) of genes in the population set (**N**) that are associated to the same term. Then we test how likely would it be to obtain at least **k** genes associated to the term if **n** genes would be randomly sampled from the population, given the frequency **K** and size **N** of the population.

The appropriate statistical test is the one-tailed variant of Fisher’s exact test, also known as the hypergeometric test for over-representation. When the one-tailed version is applied, this test will compute the probability of observing at least the sample frequency, given the population frequency.  The [hypergeometric distribution](https://en.wikipedia.org/wiki/Hypergeometric_distribution) measures precisely the probability of **k** successes in **n** draws, without replacement, from a finite population of size **N** that contains exactly **K** successful objects:

![Go Enrichment Formula](../../images/transcriptomics_images/goenrichment_formula.png)

For this first exercise we will use data from [Trapnell et al. 2014](https://www.ncbi.nlm.nih.gov/pubmed/22383036 "Trapnell et al. data"). In this work, the authors created an artificial dataset of gene expression in *Drosophila melanogaster*, where 300 random genes were set (insilico) to be differentially expressed between two conditions.

> <hands-on-title></hands-on-title>
>
> The data for this tutorial is available at [Zenodo](https://zenodo.org/record/2565417) to download. For convenience and reproducibility of results, we already added the GO ontology and annotations in the Zenodo repository.
>
> 1. **Create a new history**
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Upload to the Galaxy** the following files:
>    - go.obo
>    - drosophila_gene_association.fb
>    - trapnellPopulation.tab
>
>    > <tip-title>Upload data to Galaxy</tip-title>
>    > * **Click** on the upload button in the upper left of the interface.
>    >
>    > ![Galaxy Upload](../../images/transcriptomics_images/goenrichment_galaxy_upload.png)
>    >
>    > * Press **Choose local file** and search for your file.
>    > * Press **Start** and wait for the upload to finish.
>    {: .tip}
>
> 3. **Rename** the *go.obo* file to `GO` and *drosophila_gene_association.fb* file to `GO annotations Drosophila melanogaster`.
>
>    After you upload the files, and if you press the {% icon galaxy-eye %} (eye) icon of `trapnellPopulation.tab` it should look something like this:
>    ![Trapnell File](../../images/transcriptomics_images/goenrichment_trapnellFile.png "Trapnell file")
>
>
>    > <comment-title></comment-title>
>    > The study set represents the differentially expressed genes. These were chosen as having an adjusted p-value for the differential expression test (last column) smaller than a given threshold. In this case, we want to select the genes with an adjusted p-value < 0.05.
>    {: .comment}
>
> 4. **Filter data on any column using simple expressions** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `trapnellPopulation.tab`
>    - {% icon param-text %} *"With following condition"*: `c7 < 0.05`
> ![Galaxy Filter](../../images/transcriptomics_images/goenrichment_galaxyFilter.png)
>
> 5. This generates one file. **Rename** it to `trapnellStudy`.
>
>    > <comment-title></comment-title>
>    > Both files have the same type of information, the difference between them being the number of genes, as the genes in the study sample are a subset of the population.
>    {: .comment}
>
> 6. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `Go Annotations Drosophila melanogaster`
>    - {% icon param-file %} *"Study set File"*: `trapnellStudy`
>    - {% icon param-file %} *"Population set File"*: `trapnellPopulation.tab`
>    - Use the default options for the rest.
> ![GO Enrichment Trapnell](../../images/transcriptomics_images/goenrichment_galaxyTrapnell.png)
>
>
>    > <question-title></question-title>
>    >
>    > What were the results from running GOEnrichment?
>    >
>    > > <solution-title></solution-title>
>    > > This will generate 6 files with the respective default names: `goenrichment on trapnellStudy MF Table', `goenrichment on trapnellStudy BP Table', `goenrichment on trapnellStudy MF Table', `goenrichment on trapnellStudy MF Graph', `goenrichment on trapnellStudy BP Graph' and `goenrichment on trapnellStudy CC Graph'. The three Table files list the results of the statistical test for each GO Term, ordered by p-value, and the three Graph files are image files displaying a graph view of the enriched GO terms.
>    > {: .solution}
>    {: .question}
>
>    > <comment-title></comment-title>
>    > For each GO term we obtain a p-value corresponding to a single, independent test. Since we are making multiple similar tests, the probability of at least one of them being a false positive increases. Therefore we need to make a correction for multiple testing.
>    {: .comment}
>
>    > <question-title></question-title>
>    >
>    > How many significant terms do we get?
>    >
>    > > <solution-title></solution-title>
>    > > When we ask how many significant terms, we want to see GO terms that have a p-value < 0.05. According to the results, for Molecular Function we have 5 GO terms, Biological Process we have 43 GO terms and Component Cellular we have 10 GO terms.
>    > {: .solution}
>    {: .question}
>
> If you press the {% icon galaxy-eye %} (eye) icon of the *Molecular Function* file (`MF Trapnell`) you should see something like this:
>
> ![GO Enrichment Trapnell MF](../../images/transcriptomics_images/goenrichment_mfTrapnell.png)
>
>    > <question-title></question-title>
>    >
>    > Did you expect to see significant terms?
>    >
>    > > <solution-title></solution-title>
>    > > The ~300 genes should be random, so we wouldn't expect to see any enriched term. Nonetheless we still have significant terms.
>    > {: .solution}
>    {: .question}
{: .hands_on}

> <comment-title></comment-title>
>
> Let's go back a little bit, and reopen the `trapnellPopulation.tab` file. If you go through the file, you'll see genes with 'NA' as an adjusted p-value. This means that there are genes in our background population for which the differential expression test was not even performed (usually genes that were not expressed in any sample). These genes are irrelevant for this functional enrichment analysis. We also need to note that the study set does not include those genes!
{: .comment}

Let's remove the irrelevant genes from the background population (`trapnellPopulation.tab`), to see the differences in results.

> <hands-on-title></hands-on-title>
>
> 1. **Filter data on any column using simple expressions** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `trapnellPopulation.tab`
>    - {% icon param-text %} *"With following condition"*: `c7 != 'NA'`
> ![Galaxy Filter NA](../../images/transcriptomics_images/goenrichment_galaxyFilterNA.png)
>
> 2. This generates one file. **Rename** to `trapnellFilteredPopulation`.
> 3. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `Go Annotations Drosophila melanogaster`
>    - {% icon param-file %} *"Study set File"*: `trapnellStudy`
>    - {% icon param-file %} *"Population set File"*: `trapnellFilteredPopulation`
>    - Use the default options for the rest.
> ![GO Enrichment Trapnell new Population](../../images/transcriptomics_images/goenrichment_galaxyTrapnellNewPop.png)
>
> 4. **Rename** all 6 output files by appending `FilteredPop` to the name, to distinguish them from the previous outputs.
> Let's check the new graph **goenrichment on trapnellStudy MF Graph FilteredPop**.
>
> ![GO Enrichment Trapnell new Population MF](../../images/transcriptomics_images/goenrichment_mfTrapnellNew.png)
>
>
>    > <question-title></question-title>
>    >
>    > 1. How many significant terms do we get now?
>    > 2. Why do you see these differences?
>    >
>    > > <solution-title></solution-title>
>    > > - According to the results, in *Molecular Function* and *Biological Process* we have 0 GO terms and *Component Cellular* just 1 GO term.
>    > > - The background population genes that we removed are not random, they are usually genes that are expressed in specific conditions, tissues or time points. If they are included in the test, we will obtain false enrichments, as we saw.
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Simplification of graphs

Graphical views are essential, but sometimes the graph view can become overwhelming due to the size of the results. To exemplify this issue, we will next perform functional enrichment analysis using a more realistic dataset from a study using the mouse model organism. The original dataset can be found [on NCBI](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE30352). In this [study](https://www.nature.com/articles/nature10532), the authors compared the gene expression of several tissues. Here, we will use results from the comparison between heart and brain.

> <hands-on-title></hands-on-title>
>
> For the first exercise we will use as a study set the differential genes (padjusted<0.05).
> 1. **Upload to Galaxy** the mouse_brain_vs_heart.txt, Mus_musculus_annotations_biomart_e92.tab and mouse_brain_vs_heart.difgenes.txt files.
> 2. **Rename** the *mouse_brain_vs_heart.txt* file to `Mouse population`, *Mus_musculus_annotations_biomart_e92.tab* file to `GO annotations Mus musculus` and *mouse_brain_vs_heart.difgenes.txt* file to `Mouse diff`.
> 3. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus`
>    - {% icon param-file %} *"Study set File"*: `Mouse diff`
>    - {% icon param-file %} *"Population set File"*: `Mouse population`
>    - Select **'No'** in the Summarize Output option.
> ![GO Enrichment Mouse Diff](../../images/transcriptomics_images/goenrichment_galaxyMouseDiff.png)
>
> 4. This will generate 6 files with the names: `goenrichment on Mouse diff MF Table`, `goenrichment on Mouse diff BP Table`, `goenrichment on Mouse diff CC Table`, `goenrichment on Mouse diff MF Graph`, `goenrichment on Mouse diff BP Graph` and `goenrichment on Mouse diff CC Graph`.
> 5. Analyze the table and graph from *Biological Process*.
>
>
> ![GO Enrichment Mouse Diff BP](../../images/transcriptomics_images/goenrichment_bpMouseDiff.png)
>
>    > <comment-title></comment-title>
>    > As you can see the three graphs are very complex and difficult to analyze.
>    {: .comment}
{: .hands_on}

As you see, the number of enriched GO Terms is very high, with graphs that are too extensive to analyze manually. And this is despite the fact that GOEnrichment ignores singletons and skips dependent tests by default, precisely to avoid enrichment results that are too extensive and not informative.

The Summarize Output option in the GOEnrichment tool addresses this problem by conflating branches/families of enriched GO terms and selecting the most representative term(s) from them (usually 1-2 term per family). The greatly simplifies the results while retaining branch information, and thus ensuring that every enriched family of functions is present in the results. Some specificity is necessarily lost, but the trade-off is that the results become easier and more intuitive to analyze.

> <hands-on-title></hands-on-title>
>
> 1. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus`
>    - {% icon param-file %} *"Study set File"*: `Mouse diff`
>    - {% icon param-file %} *"Population set File"*: `Mouse population`
>    - Use the default options for the rest (notice that by default the Summarize Option is on).
> ![GO Enrichment Mouse Diff Summarize](../../images/transcriptomics_images/goenrichment_galaxyMouseDiffSum.png)
> 2. Analyze again the table and graph from *Biological Process*.
>
>
> ![GO Enrichment Mouse Diff Summarize BP](../../images/transcriptomics_images/goenrichment_bpMouseDiffSum.png)
>
>    > <question-title></question-title>
>    > 1. Are there differences in complexity comparing the graph with and without the summarize output option?
>    > > <solution-title></solution-title>
>    > > 1. Yes, there are differences. As you can see, the activation of the Summarize option reduces the size of the graph because this parameter causes families of GO terms to be conflated. Each major branch in the full results is still present in the summarized results, but now is reduced to 1 or 2 most representative terms, leading to a graph that is much easier to interpret while still containing all the key functional information.
>    > {: .solution}
>    {: .question}
{: .hands_on}

Another approach to reduce the complexity of the results is to use a shallower version of GO, the GO slims. GO slims are transversal cuts of GO that cover all key branches but lack specific terms. Thus, using them leads to much simpler results than using the full GO, but also leads to a substantial loss in specificity, which is greater than that of the Summarize Output option. You can download slimmed versions of GO from the [Gene Ontology website](http://geneontology.org/page/go-subset-guide).

To test the GO slim approach, let us use the mouse dataset again. First, however, we need to use GOSlimmer tool to convert the annotations file from full GO to GO slim (as GO annotations are typically made to terms that are too specific to be in the GO slim, and thus need to be extended by the true path rule).

> <hands-on-title></hands-on-title>
>
> 1. **Upload to the Galaxy** the goslim_generic.obo file.
> 2. **Rename** the *goslim_generic.obo* file to `GO Slim`.
> 3. Run **GOSlimmer** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Full Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"GOSlim File"*: `GO Slim`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus`
>
> ![Galaxy GO Slimmer](../../images/transcriptomics_images/goenrichment_galaxySlimer.png)
>
> This will generate one file called `Slim Annotations`.
{: .hands_on}

Now we will go use the GOEnrichment tool with the new Slim Annotations file and the same study set.

> <hands-on-title></hands-on-title>
>
> 1. **GOEnrichment** {% icon tool %} with the following parameters:
>    - Use the `GO Slim`, `Slim Annotations`, `Mouse diff` and `Mouse population` files.
>    - Use the default options.
> ![GO Enrichment Mouse Slim](../../images/transcriptomics_images/goenrichment_galaxyMouseSlim.png)
>
>    > <question-title></question-title>
>    >
>    > 1. What differences do you observe when comparing the results obtained with the GO Slim to those obtained with the full GO, with the Summarize Output option?
>    >
>    > *Component Cellular* with full GO
>    > ![GO Enrichment Mouse Summarize CC](../../images/transcriptomics_images/goenrichment_ccSum.png)
>    > *Component Cellular* with GO Slim
>    > ![GO Enrichment Mouse Slim CC](../../images/transcriptomics_images/goenrichment_ccSlimSum.png)
>    >
>    > > <solution-title></solution-title>
>    > > 1. The differences that you observe are due to the ontology used. When we apply the summarize option with the full GO, the GOEnrichment tool will return a summarized output (as we have seen previously). When we opted for GO Slim, the original annotation was already summarized, resulting in an even more summarized output, but with a consequent loss of specificity.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


# Interpretation of the results
The interpretation of the results will depend on the biological information that we intend to extract. Enrichment analysis can be used in validation (e.g., of a protocol for extracting membrane proteins), characterization (e.g., of the effects of a stress in a organism) and elucidation (e.g., of the functions impacted by the knock-out of a transcription factor).

There is one important point to keep in mind during the analysis: statistically significant is different from biologically meaningful. That said, it is typically possible to obtain some biological or technical insight about the underlying experiment from statistically enriched terms, even if it isn’t readily apparent.

Terms that are very generic tend to be difficult to interpret, because the meaning they convey is shallow. On the other hand, very specific terms are generally not integrative and thus not useful in interpreting a gene set collectively. The interesting terms are those that are sufficiently specific to transmit substantial biological meaning, yet generic enough to integrate multiple genes.

For the second exercice, we will continue to work with the same study set as before but now we analyze separately genes that are over- and under-expressed, and see the enriched GO terms presents in the brain and heart from the mouse.

> <hands-on-title></hands-on-title>
>
> 1. **Upload to Galaxy** the mouseOverexpressed.txt and the mouseUnderexpressed.txt files.
>
>    > <comment-title></comment-title>
>    >
>    > The differentially expressed genes can be identified using the adjusted p-value (also known as FDR). The logFC values indicate whether genes are more expressed (logFC>0) or less expressed (logFC<0) in one condition when comparing with another condition.
>    >
>    {: .comment}
>
> 2. **GOEnrichment** {% icon tool %} with the following parameters for the both study files (*mouseOverexpressed.txt* and the *mouseUnderexpressed.txt*).
>    - Use the `GO`, the `GO annotations Mus musculus` and the `Mouse population` files.
>    - Use the default options.
> ![GO Enrichment Mouse Overexpress](../../images/transcriptomics_images/goenrichment_galaxyMouseOver.png)
> ![GO Enrichment Mouse Underexpress](../../images/transcriptomics_images/goenrichment_galaxyMouseUnder.png)
>
> 3. This will generate 12 files, 6 for each sample file, like in previous cases.
>
>    > <question-title></question-title>
>    >
>    > Analyze both Biological Process tables. According to the study, which tissues are over- and underexpressed?
>    >
>    > > <solution-title></solution-title>
>    > > The samples correspond to the expressions that occur in the tissues referring to the brain and heart, so the results in the tables (and also in the graphs) should correspond to the specific functions of each organ. When we analyze the tables of enriched functional terms, we can see that the results from underexpressed genes reveal functions related to the brain. While in the case of the genes overexpressed, we identify functions related to muscle / heart function.
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Conclusion

Functional enrichment is a good way to look for patterns in gene lists, but interpretation of results can become a complicated process. One way to reduce this complexity is to use the GOEnrichment tool. This tool not only performs the GO Enrichment test, showing us enriched GO terms from our sets, but also contains functionality to simplify the results and make them more easily interpretable. Independently of this, we need to be careful when choosing our genes of interest, but also the background set of genes against which we want to compare.


topic: transcriptomics
tutorial: full-de-novo
title: De novo transcriptome assembly, annotation, and differential expression analysis
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 5c64eb561e4ce1ccd3c97347dceb34bb
last_modified: 2026-02-26T21:12:18.548891
zenodo_link: https://zenodo.org/record/3541678
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/full-de-novo/tutorial.html

content:
As a result of the development of novel sequencing technologies, the years between 2008 and 2012 saw a large drop in the cost of sequencing. Per megabase and genome, the cost dropped to 1/100,000th and 1/10,000th of the price, respectively. Prior to this, only transcriptomes of organisms that were of broad interest and utility to scientific research were sequenced; however, these developed in 2010s high-throughput sequencing (also called next-generation sequencing) technologies are both cost- and labor- effective, and the range of organisms studied via these methods is expanding.

Examining non-model organisms can provide novel insights into the mechanisms underlying the "diversity of fascinating morphological innovations" that have enabled the abundance of life on planet Earth. In animals and plants, the "innovations" that cannot be examined in common model organisms include mimicry, mutualism, parasitism, and asexual reproduction. De novo transcriptome assembly is often the preferred method to studying non-model organisms, since it is cheaper and easier than building a genome, and reference-based methods are not possible without an existing genome. The transcriptomes of these organisms can thus reveal novel proteins and their isoforms that are implicated in such unique biological phenomena.

[(source)](https://en.wikipedia.org/wiki/De_novo_transcriptome_assembly)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Read cleaning (20 minutes)

Known sequencing biases:
- Unknown nucleotides (Ns)
- Bad quality nucleotides
- Hexamers biases (Illumina. Now corrected ?)

Why do we need to correct those?
- To remove a lot of sequencing errors (detrimental to the vast majority of assemblers)
- Because most de-bruijn graph based assemblers can’t handle unknown nucleotides

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 12 `fq.gz` into a `List of Pairs` collection named `fastq_raw`
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from Zenodo using the URLs given below
>
>      [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3541678.svg)](https://doi.org/10.5281/zenodo.3541678)
>
>    ```
>    https://zenodo.org/record/3541678/files/A1_left.fq.gz
>    https://zenodo.org/record/3541678/files/A1_right.fq.gz
>    https://zenodo.org/record/3541678/files/A2_left.fq.gz
>    https://zenodo.org/record/3541678/files/A2_right.fq.gz
>    https://zenodo.org/record/3541678/files/A3_left.fq.gz
>    https://zenodo.org/record/3541678/files/A3_right.fq.gz
>    https://zenodo.org/record/3541678/files/B1_left.fq.gz
>    https://zenodo.org/record/3541678/files/B1_right.fq.gz
>    https://zenodo.org/record/3541678/files/B2_left.fq.gz
>    https://zenodo.org/record/3541678/files/B2_right.fq.gz
>    https://zenodo.org/record/3541678/files/B3_left.fq.gz
>    https://zenodo.org/record/3541678/files/B3_right.fq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true collection_type="List of Pairs" collection_name="fastq_raw" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to ...
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

## Quality control

> <hands-on-title>Task description</hands-on-title>
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>   - *"Short read data from your current history"*: `fastq_raw` (collection)
>
{: .hands_on}

<!-- ## Quality control with **MultiQC** - step 2/2

> <hands-on-title>Task description</hands-on-title>
>
> 1. **MultiQC** {% icon tool %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `FastQC`
>                - In *"FastQC output"*:
>                    - *"Type of FastQC output?"*: `Raw data`
>                    - *"FastQC output"*: `data XX, data XX, and others (flattened)`
>
>    > <comment-title></comment-title>
>    >
>    > We agree that it's not comfortable. The wrapper of MultiQC must be improved
>    {: .comment}
>
{: .hands_on} -->

## Read cleaning with **Trimmomatic**

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trimmomatic** {% icon tool %} with the following parameters:
>    - *"Single-end or paired-end reads?"*: `Paired-end (as collection)`
>    - *"Select FASTQ dataset collection with R1/R2 pair"*: `fastq_raw`
>    - *"Perform initial ILLUMINACLIP step?"*: `Yes`
>    - *"Adapter sequences to use"*: `TruSeq3 (additional seqs) (paired-ended, for MiSeq and HiSeq)`
>    - In *"Trimmomatic Operation"*:
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Cut bases off end of a read, if below a threshold quality (TRAILING)`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Cut bases off start of a read, if below a threshold quality (LEADING)`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Sliding window trimming (SLIDINGWINDOW)`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Drop reads with average quality lower than a specific level (AVGQUAL)`
>                - *"Minimum length of reads to be kept"*: `25`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Drop reads below a specified length (MINLEN)`
>                - *"Minimum length of reads to be kept"*: `50`
>    - *"Output trimmomatic log messages?"*: `Yes`
> 2. **Rename** the Dataset Collection
>    - `Trimmomatic on collection XX: paired` -> `fastqc_cleaned`
>
>    > <comment-title></comment-title>
>    >
>    > You can check the Trimmomatic log files to get the number of read before and after the cleaning
>    > ```
>    > Input Read Pairs: 10000
>    > Both Surviving: 8804 (88.04%)
>    > Forward Only Surviving: 491 (4.91%)
>    > Reverse Only Surviving: 456 (4.56%) Dropped: 249 (2.49%)
>    > ```
>    {: .comment}
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
{: .hands_on}

## Quality control after cleaning

> <hands-on-title>Task description</hands-on-title>
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>   - *"Short read data from your current history"*: `fastqc_cleaned` (collection)
>
{: .hands_on}

# Assembly (120 minutes - computing)

## Assembly with **Trinity**

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinity** {% icon tool %} with the following parameters:
>    - *"Are you pooling sequence datasets?"*: `Yes`
>        - *"Paired or Single-end data?"*: `Paired-end collection`
>            - *"Strand specific data"*: `No`
>    - *"Run in silico normalization of reads"*: `No`
>    - In *"Additional Options"*:
>        - *"Use the genome guided mode?"*: `No`
> 2. **Rename** the Trinity output
>    - `Trinity on data 52, data 51, and others: Assembled Transcripts` -> `transcriptome_raw.fasta`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Assembly assessment / cleaning

## Checking of the assembly statistics

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinity Statistics** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_raw.fasta`
>
>    > <comment-title></comment-title>
>    > This step, even with this toy dataset, will take around 2 hours
>    {: .comment}
>
{: .hands_on}

## Remapping on the raw transcriptome

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Align reads and estimate abundance** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_raw.fasta`
>    - *"Paired or Single-end data?"*: `Paired`
>        - *"Left/Forward strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `left`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Right/Reverse strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `right`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Strand specific data"*: `Yes`
>    - *"Abundance estimation method"*: `Salmon`
>    - In *"Additional Options"*:
>        - *"Trinity assembly?"*: `Yes`
> 2. **Rename** the 6 `* isoforms counts` :(
>    - Check in the information panel (**i** icon) the lineage of your file (ex: `A1_left.fq.gz` ... )
>    - Rename the datasets: `A1_raw`, `A2_raw`, `A3_raw`, `B1_raw`, `B2_raw`, `B3_raw`.
>
>    > <comment-title></comment-title>
>    >
>    > If you check at the Standard Error messages of your outputs. You can get the `Mapping rate`
>    > 1. Click on one dataset
>    > 2. Click on the little **i** icon
>    > 3. Click on *Tool Standard Error:	stderr*
>    > ```
>    > [2019-11-14 15:44:21.500] [jointLog] [info] Mapping rate = 44.4358%
>    > ```
>    {: .comment}
>
>    > <comment-title></comment-title>
>    >
>    > At this stage, you can now delete some useless datasets
>    > - `Trimmomatic on collection XX: unpaired`
>    > - `Align reads and estimate abundance on *: genes counts`
>    > Note that the dataset are just hidden. You can delete them permanently and make some room in the history options (the little wheel icon)
>    {: .comment}
>
>
{: .hands_on}

## Merge the mapping tables and compute normalizations

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Build expression matrix** {% icon tool %} with the following parameters:
>    - *"Abundance estimates"*: `A1_raw`, `A2_raw`, `A3_raw`, `B1_raw`, `B2_raw`, `B3_raw`
>    - *"Abundance estimation method"*: `Salmon`
>
{: .hands_on}

> <question-title></question-title>
>
> What are the three tables?
>
> > <solution-title></solution-title>
> >
> > 1. `estimated RNA-Seq fragment isoform counts (raw counts)``
> > 2. `matrix of isoform TPM expression values (not cross-sample normalized)`
> > 3. `matrix of TMM-normalized expression values`
> >
> {: .solution}
>
{: .question}

## Compute contig Ex90N50 statistic and Ex90 transcript count

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Compute contig Ex90N50 statistic and Ex90 transcript count** {% icon tool %} with the following parameters:
>    - *"Expression matrix"*: `Build expression matrix: matrix of TMM-normalized expression values`
>    - *"Transcripts"*: `transcriptome_raw.fasta`
> 2. Click on the visulization icon on the dataset `Compute contig Ex90N50 statistic and Ex90 transcript count: ExN50 statistics`
>    1. **Scatterplot - Creates a 2D-scatterplot from tabular datapoints**
>    2. *"X Column"*: select the Columns `1`
>    3. *"Y Column"*: select the Columns `2`
>
{: .hands_on}

### What we get
![ExN50_plot_toy_dataset](../../images/full-de-novo/ExN50_plot_toy_dataset.png)

### What we should get with a real dataset
![ExN50_plot](../../images/full-de-novo/ExN50_plot.png)
[(source)](https://github.com/trinityrnaseq/trinityrnaseq/wiki/Transcriptome-Contig-Nx-and-ExN50-stats)

## Transcriptome annotation completeness

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Busco** {% icon tool %} with the following parameters:
>    - *"Sequence to analyse"*: `transcriptome_raw.fasta`
>    - *"Mode"*: `transcriptome`
>    - *"Lineage"*: `eukaryota_odb9`
>
{: .hands_on}

![RNASeq samples quality check Graphs](../../images/full-de-novo/rnaseq_samples_quality_check.png)

## Filter low expression transcripts

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Filter low expression transcripts** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_raw.fasta`
>    - *"Expression matrix"*: `Build expression matrix: matrix of isoform TPM expression values (not cross-sample normalized)`
>    - *"Minimum expression level required across any sample"*: `1.0`
>    - *"Isoform filtering method"*: `Keep all isoforms above a minimum percent of dominant expression`
>        - *"Minimum percent of dominant isoform expression"*: `1`
>
>    > <comment-title></comment-title>
>    >
>    > If you check at the Standard Error messages of your outputs. You can get the `Retained` rate
>    > 1. Click on one dataset
>    > 2. Click on the little **i** icon
>    > 3. Click on *Tool Standard Error:	stderr*
>    > ```
>    > 	Retained 2096 / 2102 = 99.71% of total transcripts.
>    > ```
>    {: .comment}
>
> 2. **Rename** the output
>    - `Filter low expression transcripts on data 42 and data 14: filtered low expression transcripts` -> `transcriptome_filtered.fasta`
>
{: .hands_on}

## Checking of the assembly statistics after cleaning

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinity Statistics** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_filtered.fasta`
>
{: .hands_on}


# Annotation
## Generate gene to transcript map

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Generate gene to transcript map** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_filtered.fasta`
>
{: .hands_on}

## Peptide prediction

> <hands-on-title>Task description</hands-on-title>
>
> 1. **TransDecoder** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_filtered.fasta`
>    - In *"Training Options"*:
>        - *"Select the training method"*: `Train with the top longest ORFs`
>
{: .hands_on}

## Similarity search

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Diamond** {% icon tool %} with the following parameters:
>    - *"What do you want to align?"*: `Align amino acid query sequences (blastp)`
>    - *"Input query file in FASTA or FASTQ format"*: `TransDecoder on data XXX: pep`
>    - *"Select a reference database"*: `Uniprot Swissprot`
>    - *"Format of output file"*: `BLAST Tabular`
>    - In *"Method to restrict the number of hits?"*: `Maximum number of target sequences`
>        - *"The maximum number of target sequence per query to report alignments for"*: `1`
> 3. **Rename** the Diamond output
>    - `Diamond on data XXX` -> `Diamond (blastp)`
> 2. **Diamond** {% icon tool %} with the following parameters:
>    - *"What do you want to align?"*: `Align DNA query sequences (blastx)`
>    - *"Input query file in FASTA or FASTQ format"*: `transcriptome_filtered.fasta`
>    - *"Select a reference database"*: `Uniprot Swissprot`
>    - *"Format of output file"*: `BLAST Tabular`
>    - In *"Method to restrict the number of hits?"*: `Maximum number of target sequences`
>        - *"The maximum number of target sequence per query to report alignments for"*: `1`
> 4. **Rename** the Diamond output
>    - `Diamond on data XXX` -> `Diamond (blastx)`
>
>    > <comment-title></comment-title>
>    >
>    > Note that you can both use **Diamond** {% icon tool %} or the **NCBI BLAST+ blastp** {% icon tool %} and **NCBI BLAST+ blast** {% icon tool %}
>    {: .comment}
>
{: .hands_on}

## Find signal peptides

> <hands-on-title>Task description</hands-on-title>
>
> 1. **SignalP 3.0** {% icon tool %} with the following parameters:
>    - *"Fasta file of protein sequences"*: `TransDecoder on data XXX: pep`
>
{: .hands_on}

## Find transmembrane domains

> <hands-on-title>Task description</hands-on-title>
>
> 1. **TMHMM 2.0** {% icon tool %} with the following parameters:
>    - *"FASTA file of protein sequences"*: `TransDecoder on data XXX: pep`
>
{: .hands_on}

## Search again profile database

> <hands-on-title>Task description</hands-on-title>
>
> 1. **hmmscan** {% icon tool %} with the following parameters:
>    - *"Sequence file"*: `TransDecoder on data XXX: pep`
>
{: .hands_on}

## Transcriptome annotation using **Trinotate**

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinotate** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_filtered.fasta`
>    - *"Peptides"*: `TransDecoder on data XXX: pep`
>    - *"Genes to transcripts map"*: `Generate gene to transcript map on data XXX: Genes to transcripts map`
>    - *"BLASTP: Peptides vs Uniprot.SwissProt"*: `Diamond (blastp)`
>    - *"BLASTX: Transcripts vs Uniprot.SwissProt"*: `Diamond (blastx)`
>    - *"HMMER hmmscan: Peptides vs PFAM"*: `Table of per-domain hits from HMM matches of TransDecoder on data XXX: pep against the profile database`
>    - *"TMHMM on Peptides"*: `TMHMM results`
>    - *"SignalP on Peptides"*: `SignalP euk results`
>    - *"Let Galaxy downloading the Trinotate Pre-generated Resource SQLite database"*: `Yes`
>
{: .hands_on}

# Differential Expression (DE) Analysis

## Remapping on the filtered transcriptome using

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Align reads and estimate abundance** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_filtered.fasta`
>    - *"Paired or Single-end data?"*: `Paired`
>        - *"Left/Forward strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `left`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Right/Reverse strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `right`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Strand specific data"*: `Yes`
>    - *"Abundance estimation method"*: `Salmon`
>    - In *"Additional Options"*:
>        - *"Trinity assembly?"*: `Yes`
> 2. **Rename** the 6 `* isoforms counts` :(
>    - Check in the information panel (**i** icon) the lineage of your file (ex: `A1_left.fq.gz` ... )
>    - Rename the datasets: `A1`, `A2`, `A3`, `B1`, `B2`, `B3`.
>
>    > <comment-title></comment-title>
>    >
>    > If you check at the Standard Error messages of your outputs. You can get the `Mapping rate`
>    > 1. Click on one dataset
>    > 2. Click on the little **i** icon
>    > 3. Click on *Tool Standard Error:	stderr*
>    > ```
>    > [2019-11-14 15:44:21.500] [jointLog] [info] Mapping rate = 44.4358%
>    > ```
>    {: .comment}
>
>    > <comment-title></comment-title>
>    >
>    > At this stage, you can now delete some useless datasets
>    > - `Align reads and estimate abundance on *: genes counts`
>    > Note that the dataset are just hidden. You can delete them permanently and make some room in the history options (the little wheel icon)
>    {: .comment}
>
{: .hands_on}

## Merge the mapping tables and compute a TMM normalization

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Build expression matrix** {% icon tool %} with the following parameters:
>    - *"Abundance estimates"*: `A1`, `A2`, `A3`, `B1`, `B2`, `B3`
>    - *"Abundance estimation method"*: `Salmon`
> 2. **Describe samples and replicates**  {% icon tool %} with the following parameters:
>    - *"Samples"*
>        - *"1: Samples"*:
>            - *"Full sample name"*: `A1`
>            - *"Condition"*: `A`
>        - *"2: Samples"*:
>            - *"Full sample name"*: `A2`
>            - *"Condition"*: `A`
>        - ...:
>        - *"6: Samples"*:
>            - *"Full sample name"*: `B3`
>            - *"Condition"*: `B`
>
{: .hands_on}

## RNASeq samples quality check

> <hands-on-title>Task description</hands-on-title>
> 1. **RNASeq samples quality check** {% icon tool %} with the following parameters:
>    - *"Expression matrix"*: `Build expression matrix: estimated RNA-Seq fragment isoform counts (raw counts)`
>    - *"Samples description"*: `Describe samples`
>
{: .hands_on}

## Differential expression analysis

> <hands-on-title>Task description</hands-on-title>
> 1. **Differential expression analysis** {% icon tool %} with the following parameters:
>    - *"Expression matrix"*: `Build expression matrix: estimated RNA-Seq fragment isoform counts (raw counts)`
>    - *"Sample description"*: `Describe samples` (the last one)
>    - *"Differential analysis method"*: `DESeq2`
>
{: .hands_on}

## Extract and cluster differentially expressed transcripts

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Extract and cluster differentially expressed transcripts** {% icon tool %} with the following parameters:
>    - In *"Additional Options"*:
>        - *"Expression matrix"*: `Build expression matrix: estimated RNA-Seq fragment isoform counts (raw counts)`
>        - *"Sample description"*: `Describe samples`
>        - *"Differential expression results"*: `Differential expression results on data XXX and data XXX`
>        - *"p-value cutoff for FDR"*: `1`
>        - *"Run GO enrichment analysis"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > *"p-value cutoff for FDR"*: `1`
>    > Don't do this at home! It's because we have a Toy Dataset. The cutoff should be around `0.001`
>    {: .comment}
>
{: .hands_on}

## Partition genes into expression clusters

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Partition genes into expression clusters** {% icon tool %} with the following parameters:
>    - *"RData file"*: `Extract and cluster differentially expressed transcripts: RData file`
>    - *"Method for partitioning genes into clusters"*: `Cut tree based on x percent of max(height) of tree`
>
{: .hands_on}

# Conclusion


Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.


topic: transcriptomics
tutorial: rna-seq-bash-star-align
title: RNA-seq Alignment with STAR
difficulty: intermediate
time_estimation: 90M
tools_json: []
requirements_json: []
tags_json: []
content_hash: eb9fe90a771b482110043a9a536a50ca
last_modified: 2026-02-26T21:12:18.570505
zenodo_link: https://zenodo.org/record/6457007
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-bash-star-align/tutorial.html

content:
In recent years, RNA sequencing (in short RNA-Seq) has become a very widely used technology to analyze the continuously changing cellular transcriptome, i.e. the set of all RNA molecules in one cell or a population of cells. One of the most common aims of RNA-Seq is the profiling of gene expression by identifying genes or molecular pathways that are differentially expressed (DE) between two or more biological conditions. This tutorial demonstrates a computational workflow for counting and locating the genes in RNA sequences. The first and most critical step in an RNA-seq analysis.

In the study of {% cite brooks2011conservation %}, the authors identified genes and pathways regulated by the *Pasilla* gene (the *Drosophila* homologue of the mammalian splicing regulators Nova-1 and Nova-2 proteins) using RNA-Seq data. They depleted the *Pasilla* (*PS*) gene in *Drosophila melanogaster* by RNA interference (RNAi). Total RNA was then isolated and used to prepare both single-end and paired-end RNA-Seq libraries for treated (PS depleted) and untreated samples. These libraries were sequenced to obtain RNA-Seq reads for each sample. The RNA-Seq data for the treated and the untreated samples can be compared to identify the effects of *Pasilla* gene depletion on gene expression.

In this tutorial, we illustrate the analysis of the gene expression data step by step using 7 of the original datasets:

- 4 untreated samples: [GSM461176](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461176), [GSM461177](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461177), [GSM461178](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461178), [GSM461182](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461182)
- 3 treated samples (*Pasilla* gene depleted by RNAi): [GSM461179](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461179), [GSM461180](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461180), [GSM461181](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461181)

Each sample constitutes a separate biological replicate of the corresponding condition (treated or untreated). Moreover, two of the treated and two of the untreated samples are from a paired-end sequencing assay, while the remaining samples are from a single-end sequencing experiment.

> <comment-title>Full data</comment-title>
>
> The original data are available at NCBI Gene Expression Omnibus (GEO) under accession number [GSE18508](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE18508). The raw RNA-Seq reads have been extracted from the Sequence Read Archive (SRA) files and converted into FASTQ files.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title></comment-title>
>
> This tutorial is significantly based on Galaxy's ["Reference-based RNA-Seq data analysis"]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial.
>
{: .comment}

# Data Upload

At first, we need to put our hands on our data. In this tutorial we will use the files for 2 out of the 7 samples to demonstrate how to calculate read counts (a measure of the gene expression) from FASTQ files (quality control, mapping, read counting). The FASTQ files for the other 5 samples, are provided, in case you want to reproduce the whole analysis later.
The "Data Upload" process is the only one in this tutorial that takes place directly into Galaxy's interface.


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the FASTQ file pairs from [Zenodo]({{ page.zenodo_link }}) or a data library:
>    - `GSM461177` (untreated): `GSM461177_1` and `GSM461177_2`
>    - `GSM461180` (treated): `GSM461180_1` and `GSM461180_2`
>
>    ```text
>    {{ page.zenodo_link }}/files/GSM461177_1.fastqsanger
>    {{ page.zenodo_link }}/files/GSM461177_2.fastqsanger
>    {{ page.zenodo_link }}/files/GSM461180_1.fastqsanger
>    {{ page.zenodo_link }}/files/GSM461180_2.fastqsanger
>    ```
>
> 2. {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <comment-title></comment-title>
>    >
>    > Note that these are the full files for the samples and ~1.5Gb each so it may take some minutes to import.
>    >
>    > For a quicker run-through of the FASTQ steps a small subset of each FASTQ file (~5Mb) can be found here on [Zenodo]({{ page.zenodo_link }}):
>    >
>    > ```text
>    > {{ page.zenodo_link }}/files/GSM461177_1_subsampled.fastqsanger
>    > {{ page.zenodo_link }}/files/GSM461177_2_subsampled.fastqsanger
>    > {{ page.zenodo_link }}/files/GSM461180_1_subsampled.fastqsanger
>    > {{ page.zenodo_link }}/files/GSM461180_2_subsampled.fastqsanger
>    > ```
>    >
>    {: .comment}
>
> 4. Change the datatype from `fastqsanger` to `fastq`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastq" %}
>
> 5. We also need to import two more files, essential for the alignment operation (and basically every alignment procedure): the organism's reference genome (here *D. melanogaster*) and the organism's gene annotation.
>    Those can be aquired directly via link and Galaxy's data library as described above. For this tutorial we are going to use the files [dm6.fa.gz](https://hgdownload.soe.ucsc.edu/goldenPath/dm6/bigZips/dm6.fa.gz) and [Drosophila_melanogaster.BDGP6.87.gtf (dm6)](https://usegalaxy.eu/libraries/folders/F30cab321d898d2fb/dataset/02c5f7fcdb6bf41f). Note that it is essential to convert genome's file from `*.fa.gz` to `*.fa`. That is easy now that we have already used the same method to convert `fastqsanger` to `fastq`. Remember to change the name of the file, too, in your working history as Galaxy will not do it automatically. Doing so will prevent any confusions later on.
>
> {% snippet faqs/galaxy/datasets_rename.md name="dm6.fa" %}
>
{: .hands_on}

{% include topics/sequence-analysis/tutorials/quality-control/fastq_question.md %}

Keep in mind that the reads are raw data from the sequencing machine without any pretreatments. They need to be assessed for their quality.

# Tools to use

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Software to use</hands-on-title>
>
> The R Console and other interactive tools like RStudio are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. This is one of those cases and, in order to do that, we will use the terminal provided by the RStudio itself.
> We go to "Tools" and pick the "Shell..." option and we are good to go. Our workspace is the left, terminal window that just opened.
>
> Fortunately, [miniconda](https://docs.conda.io/en/latest/miniconda.html) is already installed. Miniconda is a package manager that simplifies the installation processes. We can and will use it to install every essential package for our tutorial. However, it is of critical importance that we do that in an new environment within our existing base and install our packages in said environment.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ conda create -n name_of_your_env fastqc cutadapt star samtools subread
> > $ conda activate name_of_your_env
> > ```
> {: .code-in}
>
>
> | Software | Version | Manual | Available for | Description |
> | -------- | ------------ | ------ | ------------- | ----------- |
> | [fastqc](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) | 0.12.1 | [fastqc Manual](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/) | Linux, MacOS, Windows | A quality control tool for high throughput sequence data. |
> | [Cutadapt](https://cutadapt.readthedocs.io/en/stable/index.html) | 4.4 | [Cutadapt Manual](https://cutadapt.readthedocs.io/en/stable/guide.html) | Linux, MacOS | Finds and removes adapter sequences, primers, poly-A tails and other types of unwanted sequence from your high-throughput sequencing reads. |
> | [STAR](https://github.com/alexdobin/STAR) | 2.7.10b | [STAR Manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf) | Linux, MacOS | Universal RNA-seq aligner. |
> | [SAMtools](http://samtools.sourceforge.net/) | 1.17 | [SAMtools Manual](http://www.htslib.org/doc/samtools.html) | Linux, MacOS | Utilities for manipulating alignments in the SAM format. |
> | [Subread](https://subread.sourceforge.net/) | 2.0.3 | [featureCounts Manual](https://subread.sourceforge.net/featureCounts.html) | Linux, MacOS, Windows | Subread is a general-purpose read aligner which can align both genomic DNA-seq and RNA-seq reads and it includes **featureCounts**, a highly efficient general-purpose read summarization program that counts mapped reads for genomic features.  |
>
{: .hands_on}

# Data from Galaxy to RStudio 

Okay, our data are imported into our Gaxaly History we are working on, but we are gonna need to access them through RStutio in order to work with them.
Fortunately, this step is pretty simple with Galaxy:

> <hands-on-title>Get Data</hands-on-title>
> > <code-in-title> The gx_get() Function</code-in-title>
> > ```R
> > gx_get(number_of_file_you_want_to_import_as_given_in_Galaxy_History)
> > ```
> {: .code-in}
> The above function will import the files selected in RStudio and it will return the path where they will be stored. We will execute it multiple times, one for every distinct file (6 in our occasion). And now every time we are going to need one of the files imported we are going to use the path returned by the function.
>
{: .hands_on}

# Quality Control

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. Adapters may also be present if the reads are longer than the fragments sequenced and trimming these may improve the number of reads mapped.

Sequence quality control is therefore an essential first step in your analysis. We will use similar tools as described in the ["Quality control" tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}): [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) to create a report of sequence quality and [Cutadapt](https://cutadapt.readthedocs.io/en/stable/guide.html) ({% cite marcel2011cutadapt %}) to improve the quality of sequences via trimming and filtering.

> <hands-on-title>FastQC</hands-on-title>
>
> When using fastqc to check the quality of our reads, it would be nice to have a folder where our results will be saved. From there we can access them and assess the quality of the reads given.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ mkdir qualityRaw
> > $ fastqc /import/number_of_dataset_1_as_described_above -o qualityRaw/
> > $ fastqc /import/number_of_dataset_2_as_described_above -o qualityRaw/
> > $ fastqc /import/number_of_dataset_3_as_described_above -o qualityRaw/
> > $ fastqc /import/number_of_dataset_4_as_described_above -o qualityRaw/
> > ```
> {: .code-in}
>
{: .hands_on}

> <hands-on-title>Cutadapt</hands-on-title>
>
>  Now, it would be a good idea to trim the reads in order to get rid of bases that were sequenced with high uncertainty (i.e. low quality bases) at the read ends and also remove the reads of overall bad quality. We can also use this opportunity to restore the trimmed files names to make our job easier down the road. 
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ mkdir trimmedData
> > $ cutadapt /import/2 /import/3_reverse_file_of_2 -o /trimmedData/GSM461177_R1 -p /trimmedData/GSM461177_R2
> > ```
> {: .code-in}
> The same trimming procedure should take place for the second pair of reads (forward and reverse as above). After that, the files we are going to work with are the ones located in the **trimmedData** folder (4 in our case). 
>
{: .hands_on}

> <hands-on-title>FastQC on trimmed data</hands-on-title>
>
> A second quality check can take place, same as before, only this time applied on the trimmed data files located in the **trimmedData** folder.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ mkdir qualityTrimmed
> > $ fastqc /trimmedData/GSM641177_R1 -o qualityRaw/
> > $ fastqc /trimmedData/GSM641177_R2 -o qualityRaw/
> > $ fastqc /trimmedData/GSM641180_R1 -o qualityRaw/
> > $ fastqc /trimmedData/GSM641180_R2 -o qualityRaw/
> > ```
> {: .code-in}
>
{: .hands_on}

# Alignment to a reference genome

![Simplified workflow that starts with the raw sequence reads and ends up to genome alignment](../../images/bash-star-rnaseq/workflow_align.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

We perform read alignment or mapping to determine where in the genome our reads originated from. There are a number of tools to choose from and, while there is no gold standard, there are some tools that are better suited for particular NGS analyses. We will be using the [STAR Aligner (STAR)](https://github.com/alexdobin/STAR), which is a universal aligner for mapping sequences against a large reference genome.

The alignment process consists of two steps:

1. Indexing the reference genome
2. Aligning the reads to the reference genome

# Setting up

> <hands-on-title>Index the reference genome</hands-on-title>
> Our first step is to index the reference genome for use by STAR. Indexing allows the aligner to quickly find potential alignment sites for query sequences in a genome, which saves time during alignment. Indexing the reference only has to be run once. The only reason you would want to create a new index is if you are working with a different reference genome or you are using a different tool for alignment.
>
> > <code-in-title>Indexing with `STAR`</code-in-title>
> > ```bash
> > $ mkdir index
> > $ STAR --runThreadN 16 --runMode genomeGenerate --genomeDir ~/index --genomeFastaFiles /import/14 --sjdbGTFfile /import/15 --sjdbOverhang 100 --genomeSAindexNbases 12
> > ```
> {: .code-in}
>
> While the index is created and saved in the **index** folder, you will see output that looks something like this:
>
> > <code-out-title></code-out-title>
> > ```
> > STAR version: 2.7.10b   compiled: 2022-11-01T09:53:26-04:00 :/home/dobin/data/STAR/STARcode
> > STAR.master/source
> > Date Time ..... started STAR run
> > Date Time ... starting to generate Genome files
> > Date Time ..... processing annotations GTF
> > Date Time ... starting to sort Suffix Array. This may take a long time...
> > Date Time ... sorting Suffix Array chunks and saving them to disk...
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Align reads to reference genome</hands-on-title>
>
> The alignment process consists of choosing an appropriate reference genome to map our reads against and then deciding on an aligner. We will use the STAR algorithm as said.
>
> An example of what a `STAR` command looks like is below. This command will not run, as we do not have the files: `input_file_R1` or `input_file_R2`.
>
>
> ```
> $ STAR --genomeDir /index_folder --runThreadN 16 --readFilesIn /trimmedData/input_file_R1 /trimmedData/input_file_R2
> ```
>
>
> Have a look at the [STAR manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf), while we are running STAR with the default
>
> parameters here, your use case might require a change of parameters. *NOTE: Always read the manual page for any tool before using
> and make sure the options you use are appropriate for your data.*
>
> We are going to start by aligning the reads from just one of the
> samples in our dataset (GSM461177). After that, we will be
> iterating the following process on the second pair of our sample files (GSM461180).
>
> > <code-in-title>Aligning with STAR</code-in-title>
> > ```bash
> > $ STAR --genomeDir ~/index --runThreadN 16 -- readFilesIn /trimmedData/GSM461177_R1 /trimmedData/GSM461177_R2 --outFileNamePrefix GSM461177
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > Date Time ..... started STAR run
> > Date Time ..... loading genome
> > Date Time ..... started mapping
> > ```
> {: .code-out}
>
{: .hands_on}

# SAM/BAM format

The [SAM file](https://genome.sph.umich.edu/wiki/SAM),
is a tab-delimited text file that contains information for each individual read and its alignment to the genome. While we do not
have time to go into detail about the features of the SAM format, the paper by
[Heng Li et al.](http://bioinformatics.oxfordjournals.org/content/25/16/2078.full) provides a lot more detail on the specification.

**The compressed binary version of SAM is called a BAM file.** We use this version to reduce size and to allow for *indexing*, which enables efficient random access of the data contained within the file.

The file begins with a **header**, which is optional for SAM files. The header is used to describe the source of data, reference sequence, method of
alignment, etc., this will change depending on the aligner being used. Following the header is the **alignment section**. Each line
that follows corresponds to alignment information for a single read. Each alignment line has **11 mandatory fields** for essential
mapping information and a variable number of other fields for aligner specific information. An example entry from a SAM file is
displayed below with the different fields highlighted.

![sam file example with fields description, showing qname, flag, rname, pos, maqp, and cigar as the first fields](../../images/bash-star-rnaseq/sam_bam.png)

Image from [Data Wrangling and Processing for Genomics](https://datacarpentry.org/wrangling-genomics/)

![sam file example with fields description, showing seq, qual, mrnm, mpos, and size as the second set of fields.](../../images/bash-star-rnaseq/sam_bam3.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

> <hands-on-title>SAM to BAM</hands-on-title>
> > <code-in-title>bash</code-in-title>
> > We will convert the SAM file to BAM format using the `samtools` program with the `view` command and tell this command that the input is in SAM format (`-S`) and to output BAM format (`-b`). We will start with the first sample (GSM461177) and repeat the process with the second (GSM461180):
> >
> > ```bash
> > $ samtools view -S -b GSM461177Aligned.out.sam > GSM461177Aligned.out.bam
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > [samopen] SAM header is present: 1 sequences.
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Sort BAM file by coordinates</hands-on-title>
>
> Next we sort the BAM file using the `sort` command from `samtools`. `-o` tells the command where to write the output.
> > <code-in-title>`sort` command</code-in-title>
> > ```bash
> > $ samtools sort -o GSM461177Aligned.out.sorted.bam GSM461177Aligned.out.bam
> > ```
> {: .code-in}
>
> Our files are pretty small, so we will not see this output. If you run the workflow with larger files, you will see something like this:
> > <code-out-title></code-out-title>
> >
> > ```
> > [bam_sort_core] merging from 2 files...
> > ```
> {: .code-out}
>
> SAM/BAM files can be sorted in multiple ways, e.g. by location of alignment on the chromosome, by read name, etc. It is important to be aware that different alignment tools will output differently sorted SAM/BAM, and different downstream tools require differently sorted alignment files as input.
>
> You can use samtools to learn more about this bam file as well.
>
> 
{: .hands_on}

# Counting reads per genes

Time to tie things up! To compare the expression of single genes between different conditions (e.g. with or without PS depletion), an essential first step is to quantify the number of reads per gene, or more specifically the number of reads mapping to the exons of each gene.

![Counting the number of reads per annotated gene](../../images/transcriptomics_images/gene_counting.png "Counting the number of reads per annotated gene")

For the final step of this tutorial we will use featureCounts to count the number of reads per annotated gene.

> <hands-on-title>FeatureCounts</hands-on-title>
>
> The main output of featureCounts is a table with the counts, i.e. the number of reads (or fragments in the case of paired-end reads) mapped to each gene (in rows, with their ID in the first column) in the provided annotation. FeatureCount generates also the feature length output datasets. Keep in mind that the "/import/7/" corresponds to *D. melanogaster's* gene annotation file we imported in RStudio way back. 
>
> > <code-in-title>featureCounts</code-in-title>
> > ```bash
> > $ featureCounts -a /import/7 -T 8 -o featurecounts.txt -p GSM461177Aligned.out.sorted.bam GSM461180Aligned.out.sorted.bam
> > ```
> {: .code-in}
>
> The **featurecounts.txt** produced contains all the information needed for further downstream analysis of the sequences we aligned (e.g. Differential Expression).
>
{: .hands_on}


topic: transcriptomics
tutorial: rna-seq-viz-with-heatmap2
title: Visualization of RNA-Seq results with heatmap2
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 47ae286ffae7d045b857961b34f5de87
last_modified: 2026-02-26T21:12:18.574677
zenodo_link: https://zenodo.org/record/2529926
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-viz-with-heatmap2/tutorial.html

content:
Heatmaps are commonly used to visualize RNA-Seq results. They are useful for visualizing the expression of genes across the samples. In this tutorial we show how the **heatmap2** tool in Galaxy can be used to generate heatmaps. The heatmap2 tool uses the heatmap.2 function from the R gplots package. Here we will demonstrate how to make a heatmap of the top differentially expressed (DE) genes in an RNA-Seq experiment, similar to what is shown for the fruitfly dataset in the [RNA-seq ref-based tutorial]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}). We will also show how a heatmap for a custom set of genes an be created.

To generate a heatmap of RNA-seq results, we need a file of normalized counts. This file is provided for you here. The expression values have been normalized for differences in sequencing depth and composition bias between the samples. To generate this file yourself, see the [RNA-seq counts to genes]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %}) tutorial, and run limma-voom selecting *"Output Normalised Counts Table?"*: `Yes`. You could also use a file of normalized counts from other RNA-seq differential expression tools, such as edgeR or DESeq2. We also need some genes to plot in the heatmap.

The data for this tutorial comes from a Nature Cell Biology paper, [EGF-mediated induction of Mcl-1 at the switch to lactation is essential for alveolar cell survival](https://www.ncbi.nlm.nih.gov/pubmed/25730472)), Fu et al. 2015. This study examined the expression profiles of basal and luminal cells in the mammary gland of virgin, pregnant and lactating mice. Six groups are present, with one for each combination of cell type and mouse status.

![Tutorial Dataset](../../images/rna-seq-reads-to-counts/mouse_exp.png "Tutorial Dataset")


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing the inputs

We will use three files for this analysis:

 * **Normalized counts file** (genes in rows, samples in columns)
 * **Differentially expressed results file** (genes in rows, columns for P values and log fold change)
 * **Genes of interest** (custom list of genes to be visualized in heatmap)

## Import data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-seq exercise e.g. `RNA-seq heatmap`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the normalized counts table.
>
>    To import the file, there are two options:
>    - Option 1: From a shared data library if available (ask your instructor)
>    - Option 2: From [Zenodo](https://zenodo.org/record/2529926)
>
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
>    - You can paste the links below into the **Paste/Fetch** box:
>
>      ```
>      https://zenodo.org/record/2529926/files/limma-voom_normalised_counts
>      https://zenodo.org/record/2529926/files/limma-voom_luminalpregnant-luminallactate
>      https://zenodo.org/record/2529926/files/heatmap_genes
>      ```
>
>    - Select *"Genome"*: `mm10`
>
> 2. Rename the counts dataset as `normalized counts`, the limma-voom_luminalpregnant-luminallactate file as `DE results` and the list of genes as `heatmap genes` using the {% icon galaxy-pencil %} (pencil) icon.
> 3. Check that the datatype is `tabular`.
>    If the datatype is not `tabular`, please change the file type to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

Click on the {% icon galaxy-eye %} (eye) icon of the the `DE results` file and take a look. It should look like below with 8 columns (just the first few rows are shown).

![DE results](../../images/rna-seq-viz-with-heatmap2/DE_results.png "DE results")

# Create heatmap of top DE genes

First we'll demonstrate how to create a heatmap of the top differentially expressed genes in an RNA-seq dataset. To do this we need to extract the differentially expressed genes from the `DE results` file. This file contains the results from comparing gene expression in the luminal cells in the pregnant versus lactating mice. It includes genes that are not significantly differentially expressed. As in the [RNA-seq counts to genes tutorial]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %})), we'll call genes significantly differentially expressed in this dataset if they pass the thresholds of adjusted P value < 0.01 and fold change of > 1.5 (log2FC of 0.58).

## Extract all significant genes

> <hands-on-title>Extract the significant genes</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters to extract genes with adj P < 0.01:
>    - {% icon param-file %} *"Filter"*: `DE results`
>    - {% icon param-text %} *"With following condition"*: `c8<0.01`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
> 2. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters to extract genes with absolute fold change > 1.5 (log2fc of 0.58):
>    - {% icon param-file %} *"Filter"*: output of Filter {% icon tool %}
>    - {% icon param-text %} *"With following condition"*: `abs(c4)>0.58`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
> 3. Rename file to `Significant genes`
{: .hands_on}

## Extract the top significant genes

This gives us a file with all the significant genes, the genes that pass our thresholds for statistical and biological significance. As we can see there are many genes (~1,610), too many to plot in one heatmap, so we'll select the top 20 by P value.

> <hands-on-title>Extract the top significant genes by P value</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} data in ascending or descending order, with the following settings to sort by P value:
>    - {% icon param-file %} *"Sort Query"*: `Significant genes` (output of 2nd Filter {% icon tool %})
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>    - {% icon param-select %} *"on Column"*: `Column: 7`
>    - {% icon param-select %} *"in"*: `Ascending order`
>    - {% icon param-select %} *"Flavor"*: `General numeric sort ( scientific notation -g)`
> 2. {% tool [Select first](Show beginning1) %}  lines from a dataset with the following parameters to select top genes:
>    - {% icon param-text %} *"Select first"*: `21` (20 genes plus header row)
>    - {% icon param-file %} *"from"*: output of Sort {% icon tool %}
> 3. Rename file as `top 20 by Pvalue`
{: .hands_on}

## Extract the normalized counts for top genes

Now we have a file that contains only the top 20 genes from the DE results. Next we need to get the normalized counts for these genes, from the file containing the normalized counts for all genes in the experiment, and then extract just the columns we need for the heatmap (the normalized counts and gene labels).

First click on the {% icon galaxy-eye %} (eye) icon and take a look at the `normalized counts` file that we imported. It should look like below (just the first few rows and columns are shown). Note that the normalized count values are log2. We will join our `top 20 by Pvalue` file to the `normalized counts` file, matching on the ENTREZID columns.

![Normalized counts](../../images/rna-seq-viz-with-heatmap2/normalized_counts.png "Normalized counts")

> <hands-on-title>Extract the normalized counts for the top genes</hands-on-title>
> 1. {% tool [Join two Datasets side by side on a specified field](join1) %} with the following parameters to join on the ENTREZID column:
>    - {% icon param-file %} *"Join"*: the `top 20 by Pvalue` file
>    - {% icon param-select %} *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `normalized counts` file
>    - {% icon param-select %} *"and column"*: `Column: 1`
>    - {% icon param-select %} *"Keep the header lines"*: `Yes`
>
>    The generated file has more columns than we need for the heatmap. In addition to the columns with normalized counts (in log2), there is the $$log_{2} FC$$ and other information. We need to remove the extra columns.
>
> 2. {% tool [Cut](Cut1) %} columns from a table to extract the columns with the gene symbols and normalized counts
>    - {% icon param-text %} *"Cut columns"*: `c2,c12-c23`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the joined dataset (output of **Join two Datasets** {% icon tool %})
{: .hands_on}

The file should look like below with the 20 genes in the rows and the 12 samples in the columns (just the first few columns are shown).

![Heatmap input](../../images/rna-seq-viz-with-heatmap2/heatmap_input.png "Heatmap input")

Now that we have our file with just the normalized counts for the genes we want, we can create a heatmap.

## Create heatmap of top genes

> <hands-on-title>Plot the heatmap of top genes</hands-on-title>
>
> 1. {% tool [heatmap2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1) %} to plot the heatmap:
>    - {% icon param-file %} *"Input should have column headers"*: output of **Cut** {% icon tool %}
>    - {% icon param-select %} *"Data transformation"*: `Plot the data as it is`
>    - {% icon param-select %} *"Compute z-scores prior to clustering"*: `Compute on rows` (scale genes)
>    - {% icon param-select %} *"Enable data clustering"*: `No`
>    - {% icon param-select %} *"Labeling columns and rows"*: `Label my columns and rows`
>    - {% icon param-select %} *"Type of colormap to use"*: `Gradient with 3 colors`
{: .hands_on}

You should see a heatmap like below. Note that here we are plotting the top genes differentially expressed in the luminal cells from the pregnant mice (`MCL1.LC` and `MCL1.LD`) versus the luminal cells from the lactating mice (`MCL1.LE` and `MCL1.LF`). This heatmap enables us to see the expression of these genes in all the samples from the different groups in the experiment (basal virgin, basal pregnant, basal lactating, luminal virgin, luminal pregnant, luminal lactating).

![Heatmap of top DE genes](../../images/rna-seq-viz-with-heatmap2/heatmap2_top20.png "Heatmap of top DE genes"){:width="150%"}


> <question-title></question-title>
> 1. Why do we not use clustering here?
> 2. Why do we compute z-scores on rows (genes)? Try rerunning heatmap2 changing the *"Compute z-scores prior to clustering"* parameter to `Do not compute z-scores`.
> 3. The genes are ordered by P value. Can you make the heatmap with the genes ordered by fold change? Hint: Sort by the logFC column in ascending order to have the genes downregulated in the luminal pregnant vs lactating (negative fold change) at the top and the upregulated genes (positive fold change) at the bottom.
> 4. How could we make a heatmap of the top 10 most upregulated and top 10 most downregulated significant genes?
>
>    > <solution-title></solution-title>
>    > 1. We don't use clustering here as we want to keep the genes in the order we input (ordered by P value).
>    > 2. We scale the genes as otherwise large expression values from highly expressed genes would dominate the plot, see below.
>    >     ![Heatmap with no scaling](../../images/rna-seq-viz-with-heatmap2/heatmap2_top20_noscale.png "Heatmap without scaling genes")
>    >
>    > 3. To make the heatmap with these genes ordered by logFC, we could {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} the output of the **Join** {% icon tool %} step on the logFC column (Column 4), then {% tool [Cut](Cut1) %} the columns as before and remake the heatmap. It should look like below.
>    >     ![Heatmap sorted by logFC](../../images/rna-seq-viz-with-heatmap2/heatmap2_top20_sorted_by_lfc.png "Heatmap sorted by logFC")
>    > 4. To make a heatmap of the most upregulated and downregulated significant genes, we could {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} the `Significant genes` on the logFC column (instead of P value above), then use the {% tool [Select first](Show beginning1) %} and {% tool [Select last](Show tail1) %} tools to select the genes with the largest and smallest fold changes (10 genes for each) and {% tool [Concatenate datasets](cat1) %} to combine the outputs of the first and last Selects into one file. We would then repeat the {% tool [Join two Datasets side by side on a specified field](join1) %} and {% tool [Cut](Cut1) %} steps as before to get the normalized counts and make the heatmap as below (Note that one gene name in the middle is missing as it is a gene that has NA for gene symbol)
>    >     ![Heatmap for top logFC](../../images/rna-seq-viz-with-heatmap2/heatmap2_top_up_and_down.png "Heatmap of top 10 up and down")
>    {: .solution}
{: .question}


# Create heatmap of custom genes

You can also create a heatmap for a custom set of genes. To demonstrate this, we will create a heatmap for the 31 genes in Figure 6b from the original paper using this dataset (see below). These 31 genes include the authors' main gene of interest in the paper, Mcl1, and a set of cytokines/growth factors identified as differentially expressed. We will recreate this heatmap here. To see how to visualize these genes in a volcano plot see [this tutorial]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %}).

![Fu heatmap](../../images/rna-seq-viz-with-heatmap2/fu_heatmap.png "Fu et al, Nat Cell Biol 2015")

These 31 genes are in the file we imported called `heatmap genes`, shown below.

![Heatmap genes](../../images/rna-seq-viz-with-heatmap2/heatmap_genes.png "Heatmap genes")

As in the previous example, we need to extract the normalized counts for just these 31 genes. To do that we will join the `heatmap genes` file with the `normalized counts` file, on the Gene Symbol columns this time (instead of ENTREZID), and then extract the columns we need.

## Extract normalized counts for custom genes

> <hands-on-title>Extract the normalized counts for the genes of interest</hands-on-title>
>
> 1. {% tool [Join two Datasets side by side on a specified field](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: the `heatmap genes` file
>    - {% icon param-select %} *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `normalized counts` file
>    - {% icon param-select %} *"and column"*: `Column: 2`
>    - {% icon param-select %} *"Keep the header lines"*: `Yes`
> 2. {% tool [Cut](Cut1) %} columns from a table to extract the columns with the gene ids and normalized counts:
>    - {% icon param-text %} *"Cut columns"*: `c1,c5-c16`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the joined dataset (output of **Join two Datasets** {% icon tool %})
>
>    The genes are in rows and the samples in columns, we could leave the genes in rows but we will transpose to have genes in columns and samples in rows as in the Figure in the paper.
>
> 3. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0) %} to have samples in rows and genes in columns:
>    - *"Input tabular dataset"*:
>        - {% icon param-file %} *"From"*: the `Cut` dataset (output of **Cut** {% icon tool %})
{: .hands_on}

We now have a table with the 31 genes in columns and the normalized counts for the 12 samples in rows, similar to below (just the first few columns are shown).

![Transposed input](../../images/rna-seq-viz-with-heatmap2/transposed_input.png "Transposed input")

## Create heatmap of custom genes

> <hands-on-title>Plot the heatmap of custom genes</hands-on-title>
>
> 1. {% tool [heatmap2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input should have column headers"*: the generated table (output of **Transpose** {% icon tool %})
>    - {% icon param-select %} *"Data transformation"*: `Plot the data as it is`
>    - {% icon param-select %} *"Compute z-scores prior to clustering"*: `Compute on columns` (scale genes)
>    - {% icon param-select %} *"Enable data clustering"*: `No`
>    - {% icon param-select %} *"Labeling columns and rows"*: `Label my columns and rows`
>    - {% icon param-select %} *"Type of colormap to use"*: `Gradient with 3 colors`
{: .hands_on}

You should see a heatmap like below.

![Fu heatmap regenerated](../../images/rna-seq-viz-with-heatmap2/fu_heatmap_regenerated.png "Fu heatmap regenerated"){: width="30%"}

> <question-title></question-title>
>
> How does the heatmap compare to the one from the Fu paper Fig 6 (above)?
>
>    > <solution-title></solution-title>
>    >
>    > The heatmap looks similar to the heatmap in the paper.
>    >
>    {: .solution}
{: .question}


# Conclusion


In this tutorial we have seen how heatmaps can be used to visualize RNA-seq results using the **heatmap2** tool in Galaxy. We use the same dataset from the tutorials, [RNA-seq reads to counts]({% link topics/transcriptomics/tutorials/rna-seq-reads-to-counts/tutorial.md %}), [RNA-seq counts to genes]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %}), [RNA-seq genes to pathways]({% link topics/transcriptomics/tutorials/rna-seq-genes-to-pathways/tutorial.md %}) and [Visualization of RNA-Seq results with Volcano Plot]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %}).


