# tutorials

topic: introduction
tutorial: galaxy-reproduce
title: How to reproduce published Galaxy analyses
difficulty: introductory
time_estimation: 1H
tools_json: ["Import", "upload1"]
requirements_json: []
tags_json: []
content_hash: 5af8ea90b32ca1168c9019c9dcfd427b
last_modified: 2026-05-15T22:28:13.729966
zenodo_link: https://zenodo.org/record/1319069
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-reproduce/tutorial.html

content:
This training will demonstrate how to reproduce analyses performed in the Galaxy framework. Before we start with the hands-on part, we would like to give you some information about Galaxy.

Galaxy is a scientific workflow, data integration and data analysis and publishing platform. Galaxy is an open-source platform for accessible, reproducible, and transparent computational research. While Galaxy was started to allow non-bioinformaticians to analyze DNA sequencing data, it nowadays enables analysis tasks of many different domains including machine learning, ecology, climate science and  omics-type of analyses. Galaxy is easy to use because it is accessible via a web-browser and provides a graphical user interface which enables access to pre-installed tools and large computational resources. In Galaxy, all analyses are stored in so-called histories. The history keeps track of all the tools, tool versions and parameters that were used in the analysis. From such a history, a workflow can be extracted; this workflow can be used to easily repeat the analysis on different data. Both, histories and workflows, can either be shared privately with colleagues or publicly, for example as part of a published manuscript.

For more background information about Galaxy, have a look into the Galaxy publication ({% cite Afgan_2018 %}). In depth technical details about technologies that enable reproducible analyses within Galaxy are described in {% cite Gr_ning_2018 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# What does Galaxy look like?

{% assign servers = nil | list_usegalaxy_servers_shuffle %}
Many different Galaxy servers exist. Some are public, some are private, some focus on a specific topic and others like the usegalaxy.\* servers cover a broad range of tools. To reproduce published results it is highly recommended to use the same Galaxy server that was used in the original study. In the case that this was a private server that is not accessible to you, you might want to use one of the main Galaxy servers: {% for server in servers %}[{{ server.name }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %}. To learn more about the different Galaxy servers visit the [slides: options for using Galaxy]({% link /topics/introduction/tutorials/options-for-using-galaxy/slides.html %}). The particular Galaxy server that you are using may look slightly different than the one shown in this training. Galaxy instance administrators can choose the exact version of Galaxy they would like to offer and can customize its look to some extent. The basic functionality will be rather similar across instances, so don’t worry! In this training we will use the European Galaxy server on which the original analysis was performed and shared.


> <hands-on-title>Log in or register</hands-on-title>
> 1. Open your favorite browser (Chrome/Chromium, Safari or Firefox, but not Internet Explorer/Edge!)
> 2. Browse to the [Galaxy Europe instance](https://usegalaxy.eu/) or to a Galaxy instance of your choosing
> 3. Choose *Login or Register* from the navigation bar at the top of the page
> 4. If you have previously registered an account with this particular instance of Galaxy (user accounts are *not* shared between public servers!), proceed by logging in with your registered *public name*, or email address, and your password.
>
>    If you need to create a new account, click on *Register here* instead.
>
{: .hands_on}


The Galaxy interface consists of three main parts:

1. The available tools are listed on the left
2. Your analysis history is recorded on the right
3. The central panel will let you run analyses and view outputs

![Galaxy interface with tools panel on left, main panel in center, and history panel on the right.]({% link shared/images/galaxy_interface.png %})


# Create a history and load data into it

Each analysis in Galaxy starts by creating a new analysis history and loading data into it. Galaxy supports a huge variety of data types and data sources. Different ways of bringing data into Galaxy are explained [in the interface slides]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}). To reproduce published results, the data needs to be loaded from the public repository where the authors have deposited the data. This is most often done by importing data via a web link.


> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. For instance, you can choose **Reproduction of published Galaxy results** as the name of your new history.
>
>    {% snippet faqs/galaxy/histories_rename.md name="Reproduction of published Galaxy results" %}
>
{: .hands_on}


> <comment-title>Background about the dataset</comment-title>
> The Iris flower data set, also known as Fisher’s or Anderson's Iris data set, is a multivariate dataset introduced by the British statistician and biologist Ronald Fisher in his 1936 paper ({% cite Fisher1936 %}).
> Each row of the table represents an iris flower sample, describing its species and the dimensions in centimeters of its botanical parts, the sepals and petals.
> You can find more detailed information about this dataset on its dedicated [Wikipedia page](https://en.wikipedia.org/wiki/Iris_flower_data_set).
{: .comment}


> <hands-on-title>Data upload</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `iris.csv` from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1319069/files/iris.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Check the **datatype**
>    - Click on the history item to expand it to get more information.
>    - The datatype of the iris dataset should be `csv`.
>    - **Change** {% icon galaxy-pencil %} the datatype *if* it is different than `csv`.
>      - Option 1: Datatypes can be **autodetected**
>      - Option 2: Datatypes can be **manually set**
>
>    {% snippet faqs/galaxy/datasets_detect_datatype.md datatype="datatypes" %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
>
> 4. Add an `#iris` tag {% icon galaxy-tags %} to the dataset
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    Make sure the tag starts with a hash symbol (`#`), which will make the tag stick not only to this dataset, but also to any results derived from it.
>    This will help you make sense of your history.
>
{: .hands_on}

> <details-title>Working with different types of datasets</details-title>
> Some input datasets might need more specialized treatment than explained here.
> A few data types contain more than one subfile. These are uploaded via the composite data function, which is a new tab on the right of regular upload. Then at the bottom set "composite type" to your file format. For each subfile a select box will appear with a description next to it, about which subfile has to be selected where.
> Some workflows require input files as dataset collections, in such cases "Input dataset collection" are shown as input when editing or viewing the workflow in the workflow menu. Collections contain several single dataset of the same type tied together. In case a workflow input requires a collection, you’ll need to build a collection out of your files after uploading them.
> [A specialized training explains how to use collections]({{site.baseurl}}/topics/galaxy-interface/tutorials/collections/tutorial.html).
>
> {% snippet faqs/galaxy/collections_build_list.md %}
>
> In case you want to run a published Galaxy workflow on your own data, you can find explanations about the options to upload your own data [in the interface slides]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}).
{: .details}


# Import and run a Galaxy workflow

Galaxy workflows may be published either directly via the Galaxy server or on public workflow repositories such as [WorkflowHub](https://workflowhub.eu/). Thus the workflow may be present in one of the three ways:

1. As a .ga file or url link, which needs to be imported into Galaxy
2. As a link from a personal Galaxy server account that needs to be added to the own Galaxy account
3. as a link that directly starts running the workflow in a specific Galaxy server, which is possible via the WorkflowHub website.

This tutorial follows option 1, but options 2 and 3 are no more difficult

> <tip-title>How to run workflows via options 2 and 3</tip-title>
> This is not part of the training, but information in case you received a workflow of interest via way 2) or 3).
>
> **Link from a personal Galaxy**:
> In case you received a link from a personal Galaxy user account
> 1. you need to log into exactly the same Galaxy server from where the workflow link is shared, which should be clear from the start of the link, e.g. "https://usegalaxy.eu/..."
> 2. Click on the link and on the upper right on to the plus symbol (import workflow)
> 3. Continue with Step 2 of the following hands-on box
>
> **Link from WorkflowHub**:
> In WorkflowHub there is an option to directly run a workflow.
> 1. Make sure that you prepared your input data on the same server as specified in the run button.
> 2. Click on "Run on usegalaxy.eu"
> 3. select inputs
> 4. "Run workflow"
{: .tip}

> <hands-on-title>Import and run workflow available as .ga file or link</hands-on-title>
>
> 1. **Import** the workflow either via url directly from [Zenodo](https://doi.org/10.5281/zenodo.5090049) or by uploading the .ga file
>
>    ```
>    https://zenodo.org/record/5090049/files/main_workflow.ga
>    ```
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 2. **Start** the workflow by clicking on the run symbol on the last column in the workflow overview list
>
> 3. Select the `iris` dataset as the input dataset.
>
> 4. **Run** the workflow by clicking on `run workflow`
>
>    > <question-title></question-title>
>    >
>    > How many history items do you have after running the workflow?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 12, out of which 4 are shown and 8 hidden (at top of history right ander the history name)
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <comment-title>Hidden datasets in history</comment-title>
> Some workflow outputs might be considered as not very important intermediate results and are thus marked as getting hidden in the analysis history after they turned green. This makes the history easier to navigate through the main results which are visible in the history. Hidden datasets can be made visible individually by clicking on the eye with a slash on top of the history and then clicking "Unhide" for the individual datasets. To unhide many dataset at once, click {% icon galaxy-selector %} "Select Items" at the top left of the history; then select all hidden datasets that you would like to unhide, then click "For n of N selected" and then "Unhide".
{: .comment}

By starting the workflow all jobs are sent to the Galaxy cluster for analysis. Sometimes it can take a bit until the datasets show up in your history. The jobs are processed one after the other or in parallel if the same input is used for several steps. Grey means waiting to run, yellow means running and green means finished. Red means there was an error.

{% snippet faqs/galaxy/analysis_troubleshooting.md %}

> <comment-title>The tool stays grey</comment-title>
> This scenario will likely not happen with this training analysis, but might happen with a real workflow. The tool runs are scheduled on the computing infrastructures according to their consumption of cores and memory. Thus, tools that are assigned lots of cores and/or memory need to wait until an appropriate computing spot is available. Several Galaxy server display the current computational load which gives you an idea how busy it currently is.
{: .comment}

Each dataset that turns green can already be inspected even though later datasets are still running. The second part of the training will focus on how to inspect datasets in a history.


# Inspecting the analysis history

Each history item represents one file or dataset, except when collections are used. History items are numbered, duplicates are not possible because any type of dataset manipulation will automatically generate a separate dataset in the history. Some tools produce several output files and therefore the number of history items can be larger than the number of steps in a workflow. Dataset names in the analysis history are not relevant for the tool run, therefore they can be adjusted in order to make the history easier to understand. The default name of a history item is composed of the tool name that was run and the history item number(s) of the input file(s), e.g. 'Unique on data 5'

> <hands-on-title>Inspect history</hands-on-title>
>
> 1. **Vizualize** the two scatter plots by clicking on their eye icons (view data)
>
>    {% snippet faqs/galaxy/features_scratchbook.md %}
>
> 2. **Show** all datasets by clicking on `hidden` on top of your history, right below the history name
>
> 3. **Compare** the `Convert csv to tabular` file with the `Datamash` file side by side using the scratchbook
>
> 4. **Track** how the `Datamash` results where obtained by clicking on the `Datamash` item in the history and then on its `i` icon (view details). The performed operations can be found in the section `Tool parameters`
>
>    > <question-title></question-title>
>    >
>    > 1. What are the different Iris species?
>    > 2. How many lines has the `Convert csv to tabular` file?
>    > 3. Which column was grouped in during the Datamash operation?
>    > 4. Which column of the `Remove beginning` file contains sepal length and which petal length?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1.  The 3 different Iris species are:
>    > >     - setosa
>    > >     - versicolor
>    > >     - virginica
>    > >
>    > > 2. 151 lines (by clicking on the file one can see the line count under its name)
>    > >
>    > > 3. Column 5 (details of Datamash tool: Group by fields - 5)
>    > >
>    > > 4. Column 1 and 3 (the dataset was generated by removing the header line from data 2, thus the content of the columns is the same as in data file 2)
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

If you got the same answers as written in the above solution box, then congratulations! You have imported and fully reproduced the results of a previous analysis.

However, sometimes you may wish to do more, like...

# Manipulating the analysis

Maybe you are interested in changing some of the original tool parameters and see how this changes the result. The parameter changes can be either done inside the workflow editor, which makes it easy to change many parameters at once (training on [Using Workflow Parameters]({% link topics/galaxy-interface/tutorials/workflow-parameters/tutorial.md %})) or directly in the history. To repeat a single analysis step with new parameters this can be done directly in the Galaxy history with the `re-run` option.

> <hands-on-title>Manipulate the analysis steps</hands-on-title>
>
> 1. **Rerun** the Scatterplot to plot Sepal length vs. Petal length
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
>   - *"Column to plot on x-axis"*: `1`
>   - *"Column to plot on y-axis"*: `3`
>
{: .hands_on}


# Additional Exercise: Import a published analysis history and explore it

Often not only workflows and raw data are published, but also the full Galaxy histories. These histories can either be inspected via their provided link, or imported in order to enable manipulating them in your own Galaxy account.

> <hands-on-title>Importing a history</hands-on-title>
>
>
> 1. {% tool [Import](upload1) %} a published history shared via an EU Server account
>
>    ```
>    https://usegalaxy.eu/u/annefou/h/galaxy101-for-everyone-diamond-dataset
>    ```
>
>    {% snippet faqs/galaxy/histories_import.md %}
>
{: .hands_on}

The `diamonds` dataset comes from the well-known [ggplot2](https://ggplot2.tidyverse.org/reference/diamonds.html) package developed by Hadley Wickham and was initially collected from the Diamond Search Engine in 2008.
The original dataset consists of 53940 specimen of diamonds, for which it lists the prices and various properties.
For this training, we have created a simpler dataset from the original, in which only the five columns relating to the price and the so-called 4 Cs (carat, cut, color and clarity) of diamond characteristics have been retained. The same workflow as before was used to re-analysis this second dataset and create the analysis history, which highlights the re-usability of workflows across inputs.

> <comment-title>The 4 Cs of diamond grading</comment-title>
> According to the [GIA's (Gemological Institute of America) diamond grading system](https://4cs.gia.edu/wp-content/uploads/2013/03/All-Scales.jpg)
> - **Carat** refers to the weight of the diamond when measured on a scale
> - **Cut** refers to the quality of the cut and can take the grades *Fair*, *Good*, *Very Good*, *Premium* and *Ideal*
> - **Color** describes the overall tint, or lack thereof, of the diamond from colorless/white to yellow and is given on a letter scale ranging from D to Z (D being the best, known as colorless).
> - **Clarity** describes the amount and location of naturally occuring "inclusions" found in nearly all diamonds on a scale of eleven grades ranging from *Flawless* (the ideal situation) to *I3* (Included level 3, the worst quality).
{:.comment}

> <question-title></question-title>
>
> 1. What are the different Cut categories?
> 2. How many lines has the `diamonds.csv` file?
> 3. Is the color an important factor for the Diamond price?
>
> > <solution-title></solution-title>
> >
> > 1.  The 5 different Cut categories are:
> >     - Fair
> >     - Good
> >     - Ideal
> >     - Premium
> >     - Very Good
> >
> > 2. 53940 lines (by clicking on the file one can see the line count under its name)
> >
> > 3. We can create a new scatter plot and use color as a factor (Advanced, column differentiating the different groups: 3). Then, holding carat weight constant, we see on
> > the scatter plot that color is linked to the price of the diamond. So color also explains a lot of the variance found in price!
> > ![price-diamond-color-dependence](../../images/reproduce-diamond-colour.png)
> >
> {: .solution}
{: .question}

# Conclusion


{% icon trophy %} Well done! You have just reproduced your first analysis in Galaxy.


topic: introduction
tutorial: galaxy-intro-101-everyone
title: Galaxy Basics for everyone
difficulty: introductory
time_estimation: 1H30M
requirements_json: []
tags_json: []
content_hash: 4dfccae356f990b5f07c9728c4eb82a9
last_modified: 2026-05-27T09:01:49.212850
zenodo_link: https://zenodo.org/record/1319069/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.html

content:
This practical aims at familiarizing you with the Galaxy user interface.
It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows and sharing your work.
Not everyone has the same background and that's ok!

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title>Background</comment-title>
> The Iris flower data set, also known as Fisher’s or Anderson's Iris data set, is a multivariate dataset introduced by the British statistician and biologist Ronald Fisher in his 1936 paper ({% cite Fisher1936 %}).
> Each row of the table represents an iris flower sample, describing its species and the dimensions in centimeters of its botanical parts, the sepals and petals.
> You can find more detailed information about this dataset on its dedicated [Wikipedia page](https://en.wikipedia.org/wiki/Iris_flower_data_set).
{: .comment}

## What does Galaxy look like?

> <hands-on-title>Log in or register</hands-on-title>
> 1. Open your favorite browser (Chrome/Chromium, Safari, or Firefox, but not Internet Explorer/Edge!)
> 2. Browse to a [Galaxy instance](https://galaxyproject.org/use/) of your choice
> 3. Choose *Login or Register* from the navigation bar at the top of the page
> 4. If you have previously registered an account with this particular instance of Galaxy (user accounts are *not* shared between public servers!), proceed by logging in with your registered *public name*, or email address, and your password.
>
>    If you need to create a new account, click on *Register here* instead.
>
>   > <comment-title>Different Galaxy servers</comment-title>
>   > The particular Galaxy server that you are using may look slightly different than the one shown in this training.
>   > Galaxy instance administrators can choose the exact version of Galaxy they would like to offer and can customize its look and feel to some extent.
>   > The basic functionality will be rather similar across instances, so don't worry!
>   {: .comment}
>
{: .hands_on}


The Galaxy interface is divided into four sections (panels):
* The Activity Bar on the left: _This is where you will navigate to the resources in Galaxy (Tools, Workflows, Histories etc.)_
* Currently active "Activity Panel" on the left: _By default, the {% icon tool %} **Tools** activity will be active and its panel will be expanded_
* Viewing panel in the middle: _The main area for context for your analysis_
* History of analysis and files on the right: _Shows your "current" history; i.e.: Where any new files for your analysis will be stored_

![Screenshot of the Galaxy interface with aforementioned structure](../../images/galaxy_interface.png)

The first time you use Galaxy, there will be no files in your history panel.




# Create a history

Galaxy allows you to create analysis histories. A **history** can be thought of as an electronic experimental lab book; it keeps track of all the tools and parameters you used in your analysis. From such a history, a **workflow** can be extracted; this workflow can be used to easily repeat the analysis on different data.

Think of a workflow as a cooking recipe with a list of ingredients (datasets) and a set of instructions
(pipeline of operations) that describes how to prepare or make something (such as a plot, or a new dataset).
The order of operations is important as very often the next operation takes as input the result of the previous operations. For instance, when baking
a cake, you would first sift the flour and then mix it with eggs as it would be impossible to sift the flour afterward.
That is what we call a pipeline. To make a full meal, we may need to combine multiple recipes (pipelines) together.

The finalized pipelines can be generalized as a workflow. If we use cooking as an analogy, a workflow could represent an entire menu with all the recipes for each meal.
In other words, using a workflow makes it possible to apply the same procedure to a different dataset, just by changing the input.

> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. For instance, you can choose **Galaxy 101 for everyone** as the name of your new history.
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

## Upload the Iris dataset

> <hands-on-title>Data upload</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `iris.csv` from [Zenodo]({{ page.zenodo_link }}) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/1319069/files/iris.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Check the **datatype**
>    - Click on the history item to expand it to get more information.
>    - The datatype of the iris dataset should be `csv`.
>    - **Change** {% icon galaxy-pencil %} the datatype *if* it is different than `csv`.
>      - Option 1: Datatypes can be **autodetected**
>      - Option 2: Datatypes can be **manually set**
>
>    {% snippet faqs/galaxy/datasets_detect_datatype.md datatype="datatypes" %}
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
>
> 4. Add an `#iris` tag {% icon galaxy-tags %} to the dataset
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    Make sure the tag starts with a hash symbol (`#`), which will make the tag stick not only to this dataset, but also to any results derived from it.
>    This will help you make sense of your history.
>
{: .hands_on}

# Pre-processing

Often, one or more data pre-processing step(s) may be required to proceed with the analysis.
In our case, the tools we will use require tab-separated input data and assume there is no header line. Since our data is comma-separated and has a header line, we will have to perform the following pre-processing steps to prepare it for the actual analysis:

- Format conversion
- Header removal


## Convert format

First, we will convert the file from comma-separated to tab-separated format. Galaxy has built-in format converters we can use for this.


> <hands-on-title>Converting dataset format</hands-on-title>
>
> 1. **Convert** {% icon galaxy-pencil %} the CSV file (comma-separated values) to tabular format (tsv; tab-separated values)
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="tabular (using csv-to-tabular)" %}
>
> 2. **Rename** {% icon galaxy-pencil %} the resulting dataset to `iris tabular`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. **View** the generated file by clicking on the {% icon galaxy-eye %} (eye) icon
>
>    > <question-title></question-title>
>    >
>    > How many header lines does our file have?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The file has one header line, it contains the column names.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Remove header

Now it is time to run your first tool! We saw in the previous step that our file has 1 header line. This line does not contain any data, but the names of each column. We will now remove that line from our file before moving on to our analysis.

> <comment-title>Tip: Finding your tool</comment-title>
>
> Different Galaxy servers may have tools available under different sections, therefore it is often useful to use the **search bar** at the top of the tool panel to find your tool.
>
> Additionally different servers may have multiple, similarly named tools which accomplish similar functions. When following tutorials, you should use precisely the tools that they describe. For real analyses, however, you will need to search among the various options to find the one that works for you.
>
{: .comment}

> <hands-on-title>Removing header</hands-on-title>
>
> 1. {% tool [Remove Beginning](Remove beginning1) %} with the following parameters:
>    - *Remove first*: `1` (to remove the first line only)
>    - {% icon param-file %} *"from"*: select the **iris tabular** file from your history
>    - Click **Run Tool**
>
>    > <comment-title>Tip: search for the tool</comment-title>
>    >
>    > Use the **tools search box** at the top of the tool panel to find **Remove beginning** {% icon tool %}.
>    {: .comment}
>
>    ![Settings for the `Remove beginning` tool](../../images/101_foreveryone_remove_beginning.png)
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris clean`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Click on the new history item to expand it
>
>    > <question-title></question-title>
>    >
>    > 1. Which tags are present on this resulting dataset? (You may have to refresh the history panel to see the tags)
>    > 2. How many samples (lines) does our dataset contain?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The output of **Remove beginning** {% icon tool %} is also tagged with the label `iris`. Tags beginning with a hashtag (`#`) will propagate; they will appear on any datasets derived from your original tagged file.
>    > >
>    > > 2. There are 150 lines in our file (we can see this under the file name when we have expanded the history item). This means we have 150 samples.
>    > {: .solution}
>    {: .question}
>
> 4. **View** {% icon galaxy-eye%} the contents of the resulting file.
>    - You should see that the header line is now no longer present.
>
{: .hands_on}



# Data Analysis: What does the dataset contain?

Now we are going to inspect the dataset using simple tools in order to get used to the Galaxy interface and answer basic questions.

## How many different species are in the dataset?

In order to answer this question, we will have to look at column 5 of our file, and count how many different values (species) appear there. There are several ways we could do this in Galaxy. One approach might be to first extract this column from the file, and then count how many unique lines the file contains. Let's do it!

> <hands-on-title>Extract species</hands-on-title>
>
> 1. {% tool [Cut](Cut1) %} columns from a table with the following parameters:
>      - *"Cut columns"*: `c5`
>      - *"Delimited by"*: `Tab`
>      - {% icon param-file %} *"From"*: `iris clean` dataset
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris species column`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. **View** {% icon galaxy-eye %} the resulting file
>
> 4. {% tool [Unique](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0) %} occurrences of each record  with the following parameters:
>      - {% icon param-file %} *"File to scan for unique values"*: `iris species column` (the output from **Cut** {% icon tool %})
>
> 5. **Rename** {% icon galaxy-pencil %} the dataset to `iris species`
>
> 6. **View** {% icon galaxy-eye %} the resulting file
>
>    > <question-title></question-title>
>    >
>    > 1. How many different species are in the dataset?
>    > 2. What are the different Iris species?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 3 species.
>    > > 2. The 3 different Iris species are:
>    > >     - setosa
>    > >     - versicolor
>    > >     - virginica
>    > {: .solution}
>    {: .question}
{: .hands_on}

Now we have our answer! There are 3 different Iris species in our file.

Like we mentioned before, there are often multiple ways to reach your answer in Galaxy. For example, we could have done this with just a single tool, **Group** {% icon tool %} as well.
(This tool also can be searched for by the term "grouping" )


> <hands-on-title>Exercise: Grouping dataset</hands-on-title>
>
> 1. Try answering this question (how many Iris species are in the file?) again, using a different approach:
>    - Tool: **Group** data by a column and perform aggregate operation on other columns {% icon tool %}
>    - Input dataset: `iris clean` dataset to answer the same question.
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris species group`
>
> 3. Did you get the same answer as before?
>
> > <solution-title></solution-title>
> > 1. {% tool [Group](Grouping1) %}   with the following parameters:
> >   - *"Select data"* select `iris clean` dataset
> >   - *"Group by column"*: `Column: 5`
> >
> > 2. This approach should give the same answer. There are often multiple ways to do a task in Galaxy, which way you choose is up to you!
> {: .solution}
>
{: .hands_on}


## How many samples by species are in the dataset?

Now that we know that there are 3 different species in our dataset, our next objective is determining how many samples of each species we have. To answer this, we need to look at column 5 again, but instead of just determining how many unique values there are, we need to count how many times each of them occurs.

You may have noticed there were more parameters in the **Group** {% icon tool %} tool that we did not use. Let's have a closer look and see if any of them might help us answer this question.

> <comment-title>Tool Help</comment-title>
> To find out more about how a tool works, look at the help text (below the **Execute** button).
>
> Look at the tool help for the **Group** {% icon tool %}. Do you see any parameters that could help answer this question?
{: .comment}

Looking at the tool help for **Group** {% icon tool %}, we see that we can also perform aggregate operations such as mean, median, sum, max, min, count (and more). Counting sounds just like what we need, let's try it!

> <hands-on-title>Grouping dataset and adding information</hands-on-title>
>
> 1. **Re-run** {% icon galaxy-refresh %} the **Group** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Select data"*: `iris clean`
>    - {% icon param-select %} *"Group by column"*: `Column: 5`
>    - {% icon param-repeat %} *"Insert operation"*
>      - *"Type"*: `Count`
>      - *"On column"*: `Column: 1`
>
>    {% snippet faqs/galaxy/tools_rerun.md %}
>
> 2. **Rename** {% icon galaxy-pencil %} the dataset to `iris samples per species group`
>
> 3. **View** {% icon galaxy-eye %} the resulting file.
>
>    > <question-title></question-title>
>    > How many samples per species are in the dataset?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > We have 50 samples per species:
>    > >
>    > > | 1         | 2   |
>    > > |---------- | --- |
>    > > |setosa     | 50  |
>    > > |versicolor | 50  |
>    > > |virginica  | 50  |
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Analysis: How to differentiate the different Iris species?

Our objective is to find what distinguishes the different Iris species (Figure 1). We know that we have **3** species of iris flowers, with
**50** samples for each:
- setosa
- versicolor
- virginica

These species look very much alike as shown on the figure below.

![Three species of Iris flowers](../../images/iris_flowers.png "Three species of Iris flowers (Image attributions: versicolor by Danielle Langlois licensed under CC BY-SA 3.0, retrieved from <a href="https://commons.wikimedia.org/wiki/File:Iris_versicolor_3.jpg">WikiMedia</a>; virginica by Christer Johansson licensed under CC BY-SA 3.0, retrieved from <a href="https://commons.wikimedia.org/wiki/File:IMG_7911-Iris_virginica.jpg">WikiMedia</a>; setosa by and used with permission of Sonja Keohane, retrieved from <a href="http://www.twofrog.com/irissetosa.html">www.twofrog.com</a>)")

And our objective is to find out whether the features we have been given for each species can help us to highlight the differences between the 3 species.

In our dataset, we have the following features measured for each sample:
- Petal length
- Petal width
- Sepal length
- Sepal width

> <comment-title>petal and sepal</comment-title>
> The image below shows you what the terms sepal and petal mean.
> ![Iris flower image showing Petal and Sepal](../../images/iris_sepal_petal.png "Sepal and petal of Iris flowers")
{: .comment}

## Generate summary and descriptive statistics with

> <hands-on-title>Get the mean and sample standard deviation of Iris flower features</hands-on-title>
>
> 1. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `iris tabular`
>    - *"Group by fields"*: `5`
>    - *"Sort input"*: `Yes`
>    - *"Input file has a header line"*: `Yes`
>    - *"Print header line"*: `Yes`
>    - "Print all fields from input file": `No`
>    - *"Ignore case when grouping"*: `Yes`
>    - In *"Operation to perform on each group"*:
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c1`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c1`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c2`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c2`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c3`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c3`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Mean`
>            - *"On column"*: `c4`
>        - {% icon param-repeat %} *"Insert Operation to perform on each group"*
>            - *"Type"*: `Sample Standard deviation`
>            - *"On column"*: `c4`
>
> 2. Rename the dataset to `iris summary and statistics`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. **View** {% icon galaxy-eye %} the generated file
>
>
>   > <question-title></question-title>
>   >
>   > 1. Can we differentiate the different Iris flower species?
>   >
>   > > <solution-title></solution-title>
>   > >
>   > > 1. From the results, we can see that the average *Iris setosa* petal length is lower than 1.5 with a relatively small standard deviation (<0.2).
>   > > The same can be observed for *Iris setosa* petal widths. These numbers are much smaller (width and length) than *Iris versicolor* and *Iris virginica* petals.
>   > > We can then use these characteristics to differentiate *Iris setosa* from the two other species (*I. versicolor* and *I. virginica*). On the other hand,
>   > > we cannot easily differentiate *Iris Versicolor* from *Iris Virginica*. Further analysis is necessary.
>   > >
>   > {: .solution}
>   >
>   {: .question}
>
{: .hands_on}

## Visualize Iris dataset features with two-dimensional scatterplots

Let's visualize the Iris dataset to see how the features depend on each other, and
check whether we can spot any immediate patterns.

> <hands-on-title>Plot iris feature pairs in two dimensions</hands-on-title>
>
> 1. {% tool [Scatterplot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.3.5+galaxy0) %}   with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: **iris clean**
>    - *"Column to plot on x-axis"*: `1`
>    - *"Column to plot on y-axis"*: `2`
>    - *"Plot title"*: `Sepal length as a function of sepal width`
>    - *"Label for x axis"*: `Sepal length`
>    - *"Label for y axis"*: `Sepal width`
>    - In *"Advanced Options"*:
>        - *"Data point options"*: `User defined point options`
>            - *"relative size of points"*: `2.0`
>        - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>            - *"column differentiating the different groups"*: `5`
>            - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>    - In *"Output Options"*:
>        - *Additional output format*: `PDF`
>
> 2. **View** {% icon galaxy-eye%} the resulting plot:
>
>    ![Contents of the `Group` output dataset](../../images/101_foreveryone_scatter.png)
>
> 3. Rename the dataset to `iris sepal scatterplot`
>
> > <question-title></question-title>
>  >
>  > 1. What does this scatter plot tell us about Iris species?
>  > 2. Make a new scatter plot, this time with *Petal* length versus *Petal* width.
>  > 3. Can we differentiate between the three Iris species?
>  >
>  >    > <tip-title>Re-running the ggplot tool</tip-title>
>  >    > Instead of clicking on **Scatterplot w ggplot2** {% icon tool %} again, it is possible to recall the previous scatterplot parameters by clicking on re-run button and updating the parameters we wish to modify.
>  >    >    > <hands-on-title>Re-run the tool</hands-on-title>
>  >    >    > 1. Click on the {% icon galaxy-refresh %} icon (**Run this job again**) for the output dataset of **Scatterplot w ggplot2** {% icon tool %}
>  >    >    > ![rerun](../../images/rerun_scatterplot.png)
>  >    >    >
>  >    >    > This brings up the tool interface in the central panel with the parameters set to the values used previously to generate this dataset.
>  >    >    {: .hands_on}
>  >    {: .tip}
>  >
>  > > <solution-title></solution-title>
>  > >
>  > > 1. We get similar results than with Summary and statistics: *Iris setosa* can clearly be distinguished from *Iris versicolor* and
>  > > *Iris virginica*. We can also see that sepal width and length are not sufficient features to differentiate *Iris versicolor* from *Iris
>    > > virginica*.
>  > > 2. {% tool [Scatterplot w ggplot2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.3.5+galaxy0) %} with the following parameters:
>  > >     - {% icon param-file %} *"Input tabular dataset"*: `iris clean`
>  > >     - *"Column to plot on x-axis"*: `3`
>  > >     - *"Column to plot on y-axis"*: `4`
>  > >     - *"Plot title"*: `Petal length as a function of petal width`
>  > >     - *"Label for x axis"*: `Petal length`
>  > >     - *"Label for y axis"*: `Petal width`
>  > >     - In *"Advanced Options"*:
>  > >         - *"Data point options"*: `User defined point options`
>  > >             - *"relative size of points"*: `2.0`
>  > >         - *"Plotting multiple groups"*: `Plot multiple groups of data on one plot`
>  > >             - *"column differentiating the different groups"*: `5`
>  > >             - *"Color schemes to differentiate your groups"*: `Set 2 - predefined color pallete`
>  > >
>  > > 3. Your new output dataset will look something like this:
>  > >
>  > >   ![Contents of the `Group` output dataset](../../images/101_foreveryone_scatter_petal.png)
>  > >
>  > > We can better differentiate between the 3 Iris species but for some samples the petal length versus width is still insufficient
>  > > to differentiate *Iris versicolor* from *Iris virginica*. And as before, *Iris setosa* can easily be distinguished from the two other species.
>  > {: .solution}
>  {: .question}
{: .hands_on}

# Galaxy management

## Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step. But when you receive new data, or a new report is requested, it would be tedious to do each step over again. Wouldn't it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the `Extract workflow` option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking on the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click on {% icon galaxy-history-options %} (**History options**) at the top of your history panel and select **Extract workflow**.
>
>    ![`Extract Workflow` entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
>    ![Selection of steps for `Extract Workflow` from history](../../images/101_foreveryone_workflow_create.png)
>
> 3. Replace the **Workflow name** to something more descriptive, for example: `Exploring Iris dataset with statistics and scatterplots`.
>
> 4. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes.
>
> 5. Click on the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created. But where did it go?
>
>    ![`Where workflows go` list](../../images/101_foreveryone_workflow.png)
>
> 6. Click on **Workflow** in the activity bar on the left side.
>    - Here you have a list of all your workflows.
>    - Your newly created workflow should be listed at the top:
>
>    ![`Your workflows` list](../../images/101_foreveryone_workflow_list.png)
{: .hands_on}

## The workflow editor


> <comment-title>Tip: Problems creating your workflow?</comment-title>
> If you had problems extracting your workflow in the previous step, we provide [a working copy for you]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/workflows/main_workflow.ga %}),
> which you can import to Galaxy and use for the next sections (see below how to import a workflow to Galaxy).
>
> {% snippet faqs/galaxy/workflows_run_trs.md path="topics/introduction/tutorials/galaxy-intro-101-everyone/workflows/main_workflow.ga" title="Main Workflow" %}
>
{: .comment}


We can examine the workflow in Galaxy's workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.

> <hands-on-title>Editing our workflow</hands-on-title>
>
> 1. Open the **workflow editor**
>    - Click on **Workflows** {% icon galaxy-workflows-activity %} at the interactive bar and locate your freshly created or imported workflow.
>
> 2. Select {% icon galaxy-wf-edit %} **Edit** to launch the workflow editor.
>    - You should see something like this:
>
>    ![Workflow editor](../../images/101_foreveryone_workflow_editor.png)
>
>    - When you click on a workflow step, you will get a view of all the parameter settings for that tool on the right-hand side of your screen (the **Details** section)
>    - You can also change the parameter settings of your workflow here, and also do more advanced configuration.
>
> 3. **Hiding intermediate outputs**
>    - We can tell Galaxy which outputs of a workflow are important and should be shown in our history when we run it, and which can be hidden.
>    - By default, all outputs will be shown
>    - Click the **checkbox** {% icon galaxy-selector %} next to the outputs to mark them as important:
>        - `outfile` in  **Unique** {% icon tool %}
>        - `out_file1` in **Group** {% icon tool%} step
>            - This should be the Group tool where we performed the counting, you can check which one that is by clicking on it and looking at the parameter settings in the **Details** box on the right.
>        - `png` in both **Scatterplot w ggplot2** {% icon tool %} steps
>    - To hide all other datasets in your history you will need to unselect the **eye** icon {% icon galaxy-eye %} in each tool box where you did not select the **checkbox** {% icon galaxy-selector %}
>    - Now, when we run the workflow, we will only see these final outputs
>        - i.e. the two dataset with species, the dataset with number of samples by species and the two scatterplots.
>
>    {% snippet faqs/galaxy/workflows_hide_intermediate_steps.md %}
>
> 4. **Renaming output datasets**
>    - When we performed the analysis manually, we often renamed output datasets to something more meaningful
>    - We can do the same in a workflow (see the tip box below)
>    - Let's rename the outputs we marked as important with the checkbox (and thus do not hide) to more meaningful names:
>        - **Unique** {% icon tool%}, output `outfile`: rename to `categories` tool
>        - **Group** {% icon tool %}, output `out_file1`: rename to `samples per category`
>        - Rename the scatterplot outputs as well, remember to choose a generic name, since we can now also run this on data other than iris plants.
>
>    {% snippet faqs/galaxy/workflows_rename_output.md %}
>
> 5. **Save your workflow** (important!) by clicking on the {% icon galaxy-save %} icon at the top right of the screen.
>
>
> 6. **Return** to the analysis view by clicking on **Galaxy** (or the Home icon {% icon galaxy-home %} or **Analyze Data** on older Galaxy versions) at the top menu bar.
>
{: .hands_on}

> <comment-title></comment-title>
> We could **validate** our newly built workflow by running it on the same input datasets that we used at the start of this tutorial, in order to make sure we do obtain the same results.
{: .comment}


## Run workflow on different data

Now that we have built our workflow, let's use it on some different data. For example, let us explore the `diamonds` R dataset with it.

> <hands-on-title>Create a new history and upload a new data</hands-on-title>
>
> 1. Create a **new history** and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. {% tool [Import](upload1) %} the file `diamonds.csv` from [Zenodo](https://zenodo.org/record/3540705/files/diamonds.csv) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/3540705/files/diamonds.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. **Rename** {% icon galaxy-pencil %} the dataset to `diamonds`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 4. Add a propagating tag {% icon galaxy-tags %} (e.g. `#diamonds`)
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
{: .hands_on}

The `diamonds` dataset comes from the well-known [ggplot2](https://ggplot2.tidyverse.org/reference/diamonds.html) package developed by Hadley Wickham and was initially collected from the Diamond Search Engine in 2008.
The original dataset consists of 53940 specimen of diamonds, for which it lists the prices and various properties.
For this training, we have created a simpler dataset from the original, in which only the five columns relating to the price and the so-called 4 Cs (carat, cut, color and clarity) of diamond characteristics have been retained.

> <comment-title>The 4 Cs of diamond grading</comment-title>
> According to the [GIA's (Gemological Institute of America) diamond grading system](https://4cs.gia.edu/wp-content/uploads/2013/03/All-Scales.jpg)
> - **Carat** refers to the weight of the diamond when measured on a scale
> - **Cut** refers to the quality of the cut and can take the grades *Fair*, *Good*, *Very Good*, *Premium* and *Ideal*
> - **Color** describes the overall tint, or lack thereof, of the diamond from colorless/white to yellow and is given on a letter scale ranging from D to Z (D being the best, known as colorless).
> - **Clarity** describes the amount and location of naturally occuring "inclusions" found in nearly all diamonds on a scale of eleven grades ranging from *Flawless* (the ideal situation) to *I3* (Included level 3, the worst quality).
{:.comment}

As a further simplification, our training dataset has the qualities in the *color* and *clarity* columns re-encoded as integer values (1-23 for color qualities D-Z, and 1-11 for the clarity levels from *Flawless* to *I3*).
With this adjustment, we can reuse our workflow on the data, and analyze and visualize it following the same steps as we took for the Iris dataset.

> <hands-on-title>Run workflow</hands-on-title>
>
> To analyze the diamonds price/4 Cs dataset by reusing our workflow:
>
> 1. Open the **workflow menu** (from activity bar, left side).
>    - Find the workflow you made in the previous section,
>    - Select the option `Run` {% icon workflow-run %}.
>    - The central panel will change to allow you to configure and launch the workflow.
>    - Click on **Expanded workflow form**
>
> 2. Select the `diamonds` dataset as the input dataset.
>
> 3. Customize the first scatter plot:
>
>    This step is preconfigured to plot column 1 along the x and column 2 along the y axis, while grouping by column 5.
>    This is fine and will result in *price* getting plotted against *carat* with grouping by *cut*, but you would want to adjust the plot title and axis labels accordingly. To change any parameter click on the pencil icon {% icon galaxy-wf-edit %}:
>
>    - Change *"Plot title"* to `Diamond price as a function of carat with cut as a factor`
>    - Change *"Label for x axis"* to `Weight of the diamond (carat)`
>    - Change *"Label for y axis"* to `Price (US dollars)`
>
>    ![Customize scatter plot](../../images/customize_diamond_plot.png)
>
> 4. Customize the second scatter plot.
>
>    This one is preconfigured to plot column 3 along the x and column 4 along the y axis, which, for our new data, would plot *color* as a function of *clarity*. However, we would rather want to stick to plotting *price* against weight in *carat* as in the first plot, but group by *clarity* instead of by *cut* this time, so:
>
>    - Change *"Column to plot on x-axis"* to `1`
>    - Change *"Column to plot on y-axis"* to `2`
>    - Change *"Plot title"* to `Diamond price as a function of carat with clarity as a factor`
>    - Change *"Label for x axis"* to `Weight of the diamond (carat)`
>    - Change *"Label for y axis"* to `Price (US dollars)`
>    - And finally in *"Advanced Options"* change *"column differentiating the different groups"* to `4` (clarity).
>
> 6. Click `Run workflow`.
>
> 7. Once the workflow has started, you will initially be able to see all its steps, but the unimportant intermediates will disappear after they complete successfully:
>
>    > <question-title></question-title>
>    > 1. How many cut category are there in the Diamond dataset ?
>    > 2. How many samples are there in each cut category ?
>    > 3. What do you notice about the relationship between price and carat ?
>    > 4. Based on the plot showing Price vs. Carat with Clarity as a factor, do you think clarity accounts for some of the variance in price? Why ?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. There are 5 different cut categories:
>    > > - Fair
>    > > - Good
>    > > - Ideal
>    > > - Premium
>    > > - Very Good
>    > >
>    > > 2. We have the following number of samples in each cut category:
>    > >
>    > >    | 1         | 2      |
>    > >    |---------- | ------ |
>    > >    | Fair      |	1610  |
>    > >    | Good	     |  4906  |
>    > >    | Ideal     |	21551 |
>    > >    | Premium   |	13791 |
>    > >    | Very Good |	12082 |
>    > >
>    > > 3. Using any of the scatter plots we made, we can see an obvious positive (non-linear) relationship between both variables: as carat size increases, price also increases. There is also very clear discrete values that carat size takes on, which are those vertical strips on the graph.
>    > >
>    > >    ![Price by Carat and Clarity](../../images/price_by_carat_and_clarity.png)
>    > >
>    > > 4. Holding carat weight constant, we see on the scatter plot shown above that diamonds with lower clarity are almost always cheaper than diamonds with better clarity: diamonds that are “Internally Flawless” are the most expensive whereas “I1” are the least expensive clarity types. So clarity explains a lot of the variance found in price!
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}


## Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

{% snippet faqs/galaxy/histories_sharing.md %}

> <hands-on-title>Share history</hands-on-title>
> 1. Share your history with your neighbour.
> 2. Find the history shared by your neighbour. Histories shared with specific users can be accessed by opening the history menu for the activity bar and selecting in the History menu the **Shared with Me** tab.
{: .hands_on}

# Conclusion


{% icon trophy %} Well done! You have just performed your first analysis in Galaxy. Additionally you can share your results and methods with others.

tools_json:
["Import", "upload1", "Remove Beginning", "Remove beginning1", "Cut", "Cut1", "Unique", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0", "Group", "Grouping1", "Datamash", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0", "Scatterplot w ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/3.3.5+galaxy0"]


topic: introduction
tutorial: galaxy-intro-strands
title: Introduction to Genomics and Galaxy
difficulty: introductory
time_estimation: 2H30M
requirements_json: []
tags_json: []
content_hash: 1ffa9c073d5ccc36d38f3942d14d1e9c
last_modified: 2026-05-27T09:01:49.216079
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-strands/tutorial.html

content:
This practical aims to familiarize you with the Galaxy user interface. It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows, and sharing your work.

> ###  {% icon comment %} Audience
> This tutorial is for those who are new to Galaxy, genomics, and bioinformatics.  If you aren't new to bioinformatics you can just do the items listed in the Hands-On boxes ({% icon hands_on %}), or you can try one of the [other introductory tutorials]({% link topics/introduction/index.md %}).
>
> This tutorial teaches the same basic content as [Galaxy 101]({% link topics/introduction/tutorials/galaxy-intro-101/tutorial.md %}), but requires less knowledge of biology to understand the questions this tutorial addresses.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction

> <comment-title>Requirements</comment-title>
>
> To run this practical you will need
>
> 1. An internet-connected computer.  Galaxy can run on your laptop without an internet connection, but this practical requires access to resources on the web.
> 1. A web browser. [Firefox](https://www.mozilla.org/firefox) and [Google Chrome](https://www.google.com/chrome/) work well, as does [Safari](https://www.apple.com/safari/).  Internet Explorer is known to have issues with Galaxy so avoid using that.
> 1. Access to a Galaxy instance.  Galaxy is available in many ways. If you are doing this practical as part of a workshop, the instructor will tell you which instance to use. If you are doing this on your own, you can use [usegalaxy.org](https://usegalaxy.org).
{: .comment}


> <question-title>Our Motivating Question</question-title>
> *I wonder if genes on opposite strands ever overlap with each other, and if so, how common is that?*
{: .question}

To explore this question we need a basic understanding of *genomes, chromosomes, strands,* and  *genes.*

> <comment-title>Definitions 1</comment-title>
>
> - **Genome**
>
>   The genome is the collection of all DNA native to an organism. For humans, the genome is all of a person's chromosomes.
>
> - **Chromosome**
>
>   The largest unit of DNA organization in an organism.  Humans have two copies of 23 chromosomes.  Chromosomes are *linear* in humans, and all animals and plants.  (Bacteria have *circular* chromosomes.)
>
> - **Strand**
>
>   Chromosomes are *double-stranded*.  One is the forward strand, is typically drawn on top, and moves from left to right. The other, reverse strand, is typically drawn on the bottom and moves from right to left.  Genes can occur on either strand.  A single gene will have parts on only one stand.
>
> - **Gene**
>
>   "What is a gene?" is actually a hotly debated question.  For our purposes, a gene is a section of DNA on chromosome strand that creates a molecule used by an organism.
>
> Graphically, the human genome can be shown as the chromosomes that are in it:
>
> ![Human genome consists of 23 (pairs of) chromosomes](../../images/human_genome_chroms.png)
>
> And here's a representation of a chromosome, strands, and genes on the chromosome.
>
> ![Chromosomes are double stranded and have forward and reverse strands; strands have genes on them](../../images/chromosome_strands_genes.png)
{: .comment}


The question we are asking looks like:

![Does a situation like this exist?](../../images/do_genes_on_opposite_strands_ever_overlap.png)

*Non-overlapping genes are common.  How common are overlapping genes?*

# Get human data

To answer this question we need to know where genes start and stop on human chromosomes.  That seems like a simple question, but if you are new to bioinformatics it's actually a hard question to answer.  Web searches will land you at any number of useful places on the web, but without a lot of background knowledge it's hard to know what you want:  *What's the difference between sequence and annotation?  What are FASTA, BED, GTF, GFF3, and VCF?  What are GRCh37, GRCh38, hg19, and hg38 (and what happened to hg20 through hg37 - are they okay)?*

It turns out that for this particular question (and for many others), most **Galaxy** instances can help us find this information.

> <hands-on-title>Log in to Galaxy</hands-on-title>
> In your web browser, *go to* **your Galaxy instance** and *log in or register*.
{: .hands_on}

The Galaxy interface consists of three main parts. The available tools are listed on the left, your analysis history is recorded on the right, and the central panel will show the home page, tool forms, and dataset content.

![Galaxy interface showing history panel on the right, tools panel on the left and main panel at the center](../../images/galaxy_interface.png)

> <hands-on-title>Start with an empty history</hands-on-title>
>
> - At the start of the tutorial you should ensure that you are working in a new history, i.e. one without any datasets
>
>   {% snippet faqs/galaxy/histories_create_new.md %}
>
{: .hands_on}

## Get data into Galaxy

There are [many ways to get data into a Galaxy instance]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}). We are going to use the **Get Data** toolbox in the **Tools** panel on the left.

> <hands-on-title>Open Get Data toolbox</hands-on-title>
>
> 1. *Click* on the **Get Data** toolbox to expand it.
>
>    ![The Get Data toolbox](../../images/getdata.PNG)
>
{: .hands_on}

The **Get Data** toolbox contains a list of data sources that this Galaxy instance can get data directly from.  **Upload file** is quite useful for getting data from your computer or from the web (see the [Getting data into Galaxy slides]({% link topics/galaxy-interface/tutorials/get-data/slides.html %})).  Today we are going to use the **UCSC Main table browser**.

### Get genes

> <hands-on-title>Go to UCSC</hands-on-title>
>
> 1. *Click* on the tool {% tool UCSC Main table browser %} to go to UCSC.
>
>    ![Click on UCSC Main table browser](../../images/101_01.png)
>
{: .hands_on}

This will take you to the UCSC Table Browser:

 ![UCSC table browser tool, first screen for genes](../../images/ucsc_table_browser_w_default_settings.png)

The [UCSC Table Browser](https://genome.ucsc.edu/cgi-bin/hgTables) provides access to all the data that is shown in the [UCSC Genome Browser](https://genome.ucsc.edu/cgi-bin/hgGateway) (see box below). If you are working on a species that UCSC supports (like human) then the Table Browser is a great place to get genomic data.

The Table Browser has a daunting number of options. Fortunately, they are all set to commonly used defaults, greatly simplifying things, and most of the options are already set to what we want. In particular, the human genome at version hg38 is already preselected. If the human genome was not what we are interested in, or if we were interested in a different version of it, we could click on `Select dataset` near the top of the screen to change the default, but for our purpose we only have to verify that what is selected is:

* **Assembly:** `Dec. 2013 (GRCh38/hg38)`
* **Group:** `Genes and Gene Predictions`
* **Track:** `GENCODE V49`
* **Table:** `knownGene`

Here, **Assembly** asks which version/definition of the human genome we want.  (Any will do for our question, but UCSC is suggesting `hg38`, which is also an up to date one.)  **Group** is set to `Genes and Gene Predictions` which sounds like what we want. So far so good.

**Track** has a bewildering list of options. UCSC suggests `GENCODE V49`.  A web search leads us to the [GENCODE web site](https://www.gencodegenes.org/) which prominently states:

>The goal of the GENCODE project is to identify and classify all gene features in the human and mouse genomes with high accuracy based on biological evidence...
{: .quote}

> <warning-title>ALL GENCODE is different from GENCODE</warning-title>
> **Warning** Any recent GENCODE version, V49 or higher, will do for this tutorial though exact numbers calculated in some of the steps will depend on the exact version of the track. Do not, however, select any ALL GENCODE version or you will end up with entirely wrong input data!
{: .warning}


Time for a few more definitions.

> <comment-title>Definitions 2</comment-title>
>
> - **Reference genome**
>
>    A reference genome is the *genome of a single individual* that has been thoroughly studied, to the point that we know exactly what most of that individual's DNA is.  In practice a reference genome is used as shared map by researchers working on that organism. Reference genomes are updated periodically as techniques improve.
>
> - **Sequence**
>
>   A genome's sequence describes the DNA in that genome, down to the A, C, T, and G (single nucleotide) level including the exact location where each is.  Given a reference genome, you can ask questions like, "What's the DNA on chromosome 2 between positions 1,678,901 and 1.688,322?"
>
> - **Genome/Gene annotation**
>
>   The sequence tells us what DNA is where, but it doesn't tell us anything about the function of that DNA.  *Annotation* is additional information about particular regions of the genome like where genes, repeats, promotors, and centromeres are, or how active a particular gene is.
{: .comment}

The **track** option asks us which set of annotations do we want to get?  There are so many choices because annotation is the result of analysis and interpretation, and there are many ways to do this. (And in this case, many of the options aren't even genes or gene predictions.)

GENCODE is "high-quality" and  "gene annotation." That sounds like a good thing to use.  Lets stay with the default: `GENCODE V49` or whatever version is the latest GENCODE when you are doing this tutorial.

So far we haven't changed *anything* from the defaults.  Lets change something.  The default  **region** is the whole genome, which can be done, but it's a lot of information. For this exercise lets use just one (small) chromosome.

> <hands-on-title>Limit the region and get the data.</hands-on-title>
>
> 1. Say that we just want chromosome 22
>    - For **region** select `position`.
>    - In the text box next to `position` enter `chr22` (case matters).
>
>    ![Change the region](../../images/ucsc_tb_set_region_chr22.png)
>
> 2. *Click* the **get output** button.
>
>    And, that doesn't actually get us the output. It sends us to a second UCSC page that asks us exactly what we want.
>
>    ![UCSC Table Browser 2nd page](../../images/ucsc_tb_2nd_page_whole_gene.png)
>
> 3. Under **Create one BED record per** make sure that **Whole Gene** is selected.
>
> 4. *Click* the **Send query to Galaxy** button at the bottom of the form.
{: .hands_on}

This returns us to Galaxy, first displaying a big green box (that's good!) and then returning us to the view we started with (if that doesn't happen, hit the *Back* button of your browser several/probably three times until you're back at the regular view of Galaxy). An item should now appear in your history, the dataset from UCSC.


### History item status

Watch your new history item.  It will go through three statuses before it's done.

| Color | Icon | Meaning | |
| ---- | ---- | ---- |
| **Grey** | Clock | Item is waiting to start (waiting for data transfer to start) | ![Status: Queued](../../images/status_queued.png) |
| **Yellow** | Spinner | Item is running (data is actively being transferred). | ![Status: Running](../../images/status_running.png) |
| **Green** | None | Item has finished successfully (data transfer complete). | ![Status: Successfully finished](../../images/status_finished_success.png) |
| **Red**  | Cross | The job has failed. There can be [many reasons](https://galaxyproject.org/support/tool-error/). | ![Status: Failed](../../images/status_failed.png) |
| ---- | ---- | ---- |

You can find more information in the [Undestanding Galaxy history system]({% link topics/galaxy-interface/tutorials/history/tutorial.md %}) training.

## Examine the data

> <hands-on-title>Look at the data.</hands-on-title>
> Once the dataset is green, *click* on the dataset name (something like **UCSC Main on Human...**)
{: .hands_on}

This expands the dataset and shows you information about it, and a preview of its contents.

![Preview of the data from UCSC Table Browser](../../images/genes_human_chr22_preview.png)

From this preview we can obtain a huge amount of useful information:
1. The dataset has over 4000 regions, meaning that there are over 4000 genes on chromosome 22.
1. The dataset is in **BED** format, one of several standards for representing genome annotation.  We got BED format because BED was preselected as the output format in the UCSC table browser.
1. The dataset's "database" is **hg38**.  This says which revision of the reference genome this data maps too.  hg38 is the latest human reference genome.  hg38 was also selected by default in UCSC.
1. Finally, it shows us the first 5 rows in the dataset.

The dataset preview is informative, but you can't see much of the actual dataset.  Lets use one of the dataset icons to see the whole dataset:

> <hands-on-title>Look at all the data.</hands-on-title>
> 1. Click on the {% icon galaxy-eye %} (eye) icon to view the contents of the dataset. This displays all of the data.
>
>    ![Full dataset in central panel](../../images/genes_human_chr22_dataset_view.png)
>
> 2. Use the side scroll bars to move through the file.
>
> > <question-title></question-title>
> >
> > How many columns include the BED file?
> >
> > > <solution-title></solution-title>
> > >
> > > Our BED file includes 12 columns, on which the information about the genes distribution along the genome in the chromosome 12 is summarized.
> > {: .solution}
> >
> {: .question}
>
{: .hands_on}

> <details-title>Additional information about BED files</details-title>
>
>[BED](https://genome.ucsc.edu/FAQ/FAQformat.html#format1) is one of several well-established tabular formats for genomic data.  Other formats include GFF3 and GTF.  For the type of analysis we are doing today, BED format is easiest to work with.  BED was created to power the UCSC Genome Browser.  BED files contain between 3 and 15 columns.  Our example BED file describes genes and contains 12 columns.
>
>We care about columns 1, 2, 3, and 6:
>
>| # | Column Name | Meaning |
>| ---- | ---- | ---- |
>| 1 | Chromosome | The name of the chromosome this gene is on. |
>| 2 | Start | Where on the chromosome the gene starts. |
>| 3 | End | Where on the chromosome the gene ends. |
>| 6 | Strand | Which strand the gene is on.  `+` means forward (top, left to right), `-` means reverse (bottom, right to left) |
>
>See the [BED format description at UCSC](https://genome.ucsc.edu/FAQ/FAQformat.html#format1) for a full description of all the columns.
>
>
{: .details}


## Naming

Galaxy allows you to name your analyses (your histories) and your datasets.  We only have one history ("Unnamed history") and one dataset ("UCSC Main on Human:...") so far, but it's a good idea to

1. Always name your histories
2. Name your input and final output datasets, and any significant intermediate datasets.

You don't have to do this.  Galaxy is quite happy for you to have an infinite number of "Unnamed history" histories, and to have all your datasets be obscurely named.  However, once you've run your first 5 unnamed analyses, all with obscurely named datasets, you'll might wish you would have named everything.

> <hands-on-title>Name your stuff</hands-on-title>
>
> 1. **Name your history** to be meaningful and easy to find.
>    - *Click* on the title of the history and enter something like **Intro - Strands** as the name.  Hit the `enter` key on your keyboard to save it.
>   ![Rename the history]({% link shared/images/rename_history.png %})
> 1. **Rename your dataset**
>    - *Click* on the **pencil icon** to edit the dataset attributes.
>    - In the next screen change the name of the dataset to something like `Genes` or `Genes chr22`.
>    - *Click* the **Save** button at the bottom of the screen.
>
>    Your history should now look something like this:
>
>    ![Rename dataset](../../images/genes_human_chr22_rename.png)
{: .hands_on}

# We've got the data - what's our plan for answering the question?

You have to know what's possible, before you can build a plan.  If you don't have experience with data analysis then you might not have any idea how you would answer our question.  Before we dive in using a particular solution, think about how you might solve this.  If you don't have any experience with tools, then think about how you might solve it manually, using pencil and paper (it may help to assume you have an infinite supply of helpers to do the pencil and paper work).

Here's how we'll answer this question:

1. Split the genes dataset in two: one for genes on the forward strand, and one for genes on the reverse strand.
1. Compare the two datasets to see which ones, if any, overlap.
1. Check how many (or what percentage) of our genes overlap with another gene.

It turns out that all of these steps are easy in Galaxy!

## Split the genes into forward and reverse datasets

How might we do this?  Column 6 contains the strand information.  Can we split genes into two datasets based on the value of Column 6.  How?  Lets take a look at our available tools.  And *whoa! There are over 40 toolboxes, and several hundred tools.* How are we going to find a tool that can do the split?


> <hands-on-title>Searching for and launching a tool</hands-on-title>
>
> 1. Try the **tools search box**.  Think of terms that might describe what we want to do and type them in the search box.  Do you see anything promising?  Explore a little.
> 2. If you haven't already searched with it, *enter* `filter` in the **tool search box**.  Among the results you should see
>    - {% tool [Filter](Filter1) %} data on any column using simple expressions.
>    - That might work.
> 3. *Click* on {% tool [Filter](Filter1) %} to open the Filter tool in the central panel.
> 4. Take a look at the **Syntax** and **Example** sections to understand what the tool does.
>
> ![The Filter Tool form](../../images/filter_tool_defaults.png)
{: .hands_on}

It doesn't say anything about **Filter** being able to split a file into multiple files.  It does look like we can use Filter to get only genes on the forward strand, or only genes on the reverse strand.  We would have to run Filter twice, once for forward strand genes, and once for reverse strand genes. Let's do that.

> <hands-on-title>Run the Filter tool to get genes on the forward strand.</hands-on-title>
>
> * The filter tool has 3 fields:
>
>   1. **Dataset**: This pulldown will list any dataset from your history that this tool can work on.  In your case that's probably only one dataset. {% icon param-files %} *"filter"*: `Genes chr22` .
>   2. **Condition**: this free text field is where we specify which records we want in the output dataset. Put in *"Condition"*:  `c6 == "+" ` .
>       This specifies that column 6 (the strand) must be equal to (`==` is Python for *is equal to*) a plus sign.
>   3.  *"Header lines to skip"*:  `0 `
>
>       Our dataset does not have any header lines.
>
> * Finally, *click* the **Execute** button.
>
>   ![Run the Filter tool to get only the forward strand genes](../../images/filter_tool_forward_strand_genes_only.png)
>
{: .hands_on}

This adds another dataset to your history. This one should contain only genes on the forward strand.  Once the dataset is green, click the {% icon galaxy-eye %} (eye) icon to confirm this. We also recommend that you rename this dataset to something like `Genes, forward strand` (remember how?).

Now we want to get the genes on the reverse strand.  There are actually many ways to get this.  Here are two of them.

> <hands-on-title>Get genes on the reverse strand</hands-on-title>
>
> **Method 1**
>
> 1. Open the dataset preview by *clicking* on the name of the `Genes, forward strand` dataset.  This shows an icon than the uploaded `Genes chr22` dataset did not: a looping arrow.
> 1. *Click* the **looping arrow** ("Run this job again") icon.  This won't actually run the job again.  What it will do is bring up the Filter tool form with *the exact same settings that were used to produce this dataset.*
> 1. Rerun {% tool [Filter](Filter1) %} but with
>    - *"Condition"*:  `c6 == "-" `
> 1. *Click* the **Run Tool** button.
>
> **Method 2**
> 1. *Click* on {% tool [Filter](Filter1) %} in the tool panel to open the Filter tool in the central panel.
> 1. *Fill* the form as before, *except*:
>    * Make sure the **Dataset** pulldown is set to the `Genes chr22` dataset.
>    * *Set* **Condition** to `c6 == "-"`.
> 1. *Click* the **Run Tool** button.
>
> **Both Methods**
> 1. *Rename* your new dataset to something like `Genes, reverse strand`
{: .hands_on}

The rerun button can be a huge help as you run more complex tools.

> <comment-title>Empty result?</comment-title>
>
> If you used Method 2 and didn't explicitly set the dataset, then you ran Filter on the `Genes, forward strand` dataset. None of the genes in the forward strand dataset have "-" in column 6 so all of them were filtered out from the result.
>
> Try again and set the dataset to your `Genes chr22` dataset.
{: .comment}


Your history should now have (at least) 3 datasets in it, with names like:

* `Genes chr22`
* `Genes, forward strand`
* `Genes, reverse strand`

The number of genes in the `forward` plus `reverse` datasets should be the same as in the `Genes chr22` dataset.  If they aren't can you figure out why?

## Check for overlaps

Genes are an example of a *genomic interval*.

> <comment-title>Definitions 3</comment-title>
>
> - **Genomic interval**
>   In Galaxy, a *genomic interval* is a something that spans part of a chromosome (or some other linear frame of reference like a contig).  Genes are a common example of a genomic interval.  Even a chromosome is a genomic interval, albeit a very long one.
{: .comment}

Galaxy excels at answering questions about genomic intervals and different sets of genomic intervals relate to each other.  Lets take a look.

> <hands-on-title>Genomic Interval Tools</hands-on-title>
>
> * The next step is finding overlapping intervals, so type `interval` in the tool search box. There are many results from this search, so you might want to click the *Show sections* button to see which sections the tools are organised into.
> * *Explore* the tools in this toolbox, looking for something that we can use to see which genes on opposite strands overlap.
{: .hands_on}

Although there are the bedtools (section **BED**) and a section **Operate on Genomic Intervals** offering promising tools, the simple **Intersect** tool under **Text Manipulation** appears sufficient.  Let's try **Intersect**.

> <hands-on-title>Genomic Interval Tools</hands-on-title>
>
> 1. {% tool [Intersect](toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0) %} with the following parameters:
>     - *"Return"*:  `Overlapping Intervals`.
>       This looks like it might return whole genes, while `Overlapping pieces` may return only the parts that overlap.  We suspect that whole genes might be more useful.
>     - {% icon param-files %}*"of"*:  `Genes, forward strand` (the first dataset)
>     - {% icon param-files %} *"that intersect"* : `Genes, reverse strand` (the second dataset)
>     - *"for at least"*: `1`
>
>       This will return genes with even just one position overlapping.
>     - *Click* **Run Tool**.
>
>     ![Run Intersect](../../images/genes_human_intersect_strands.png)
>
> 3. Now repeat the intersect, but make the first dataset be the reverse genes, and the second be the forward genes.
>
> 4. Finally give both of the new datasets meaningful names, like `Overlapping forward genes` and `Overlapping reverse genes`
{: .hands_on}


# Results and final steps.

At this point we *could* say that we have answered our question. Using dataset previews in the history panel, we can compare the number of genes in the `Overlapping forward` and `Overlapping reverse` datasets with the number of genes in the full `Genes chr22` dataset, and *conclude that overlapping genes on opposite strands are actually pretty common.*

However, before we rush off to publish our conclusions, let's

1. Get both the forward and reverse overlapping genes into a single dataset (one link will look better in our publication), and
2. *Visualize* our new dataset, just to double-check our results.

## Combine forward and reverse overlapping genes into one dataset.

What tool can we use to combine the two datasets into one?  Try *searching* for `combine` or `join` or `stack` in the tool search box.  You'll find lots of tools, but none of them do what we want to do. *Some times you just have to manually look through toolboxes to find what you need.*  Where should we look?  It's probably not **Get Data** or **Send Data**, but it could easily be in any of the next 4 toolboxes: **Lift-Over, Collection Operations, Text Manipulation, or Datamash**.

It turns out that **Lift-Over** and **Collection Operations** are not what we want. (But, take a look at them: if you are going to work with genomic data it won't take long before you'll need both.)

> <hands-on-title>Concatenate two datasets</hands-on-title>
>
> 1. *Open* the **Text Manipulation** toolbox.
> 2. Near the top of the toolbox is **Concatenate multiple datasets or collections**. *Click* on it. Lets try that tool.
> 3. {% tool [Concatenate multiple datasets or collections](cat1) %} with the following parameters:
>   - {% icon param-files %} *"Concatenate Dataset"*: `Overlapping reverse genes`.
>   - *"Dataset*"
>      - Click on {% icon param-repeat %} *"Insert Dataset"*
>
>         This adds a second dataset pull-down menu to the form.
>
>      - In *"1: Dataset"*:
>         - {% icon param-files %} *"Select"*: `Overlapping forward genes` as the second dataset.
> 4. *Click* **Run Tool**
> 5. *Rename* the resulting dataset something informative like `Overlapping genes`
{: .hands_on}

Once the concatenate operation is finished, preview the dataset in your history panel.  Does it have the expected number of genes in it?  If not, see if you can figure out what happened.

As the name of the tool suggests, it has simply taken the second input and glued it to the end of the first input to form the new dataset. Now all reverse-strand genes are listed before the forward-strand genes, independent of their genomic location. While that's not necessarily a problem, it may be nicer to sort the genes back into the position-based order we originally obtained them in.

> <hands-on-title>Sort the overlapping genes by genomic position</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy3) %} data in ascending or descending order with the following parameters:
>   - {% icon param-file %} *"Sort Query"*: `Overlapping genes`.
>   - {% icon param-repeat %} *"Column selections"*:
>      - In *"1: Column selections"*:
>         - *"Sort on column"*: `Column 1`
>         - *"in"*: `Ascending order`
>         - *"using sort flavor"*: `Natural/Version sort (-V)`
>
>      - Click on {% icon param-repeat %} *"Insert Column selections"*
>      - In *"2: Column selections"*:
>         - *"Sort on column"*: `Column 2`
>         - *"in"*: `Ascending order`
>         - *"using sort flavor"*: `Fast numeric sort (-n)`
>
>      - Click on {% icon param-repeat %} *"Insert Column selections"*
>      - In *"3: Column selections"*:
>         - *"Sort on column"*: `Column 3`
>         - *"in"*: `Ascending order`
>         - *"using sort flavor"*: `Fast numeric sort (-n)`
>
> 2. *Click* **Run Tool**
> 3. *Rename* the resulting dataset to, for example, `Overlapping genes sorted`
{: .hands_on}

We now have a new dataset with the genes sorted first by chromosome (which is not strictly necessary in our case because we know all genes are from the same chromosome chr22, but is important, of course, in the general case), then by start position on the chromosome and, finally, by end position (in case that's needed to break ties).

## Visualize the overlapping genes

Galaxy knows about several visualization options for lots of different dataset types, including BED.  Whenever you preview a dataset in the history panel, Galaxy provides links to these visualizations.  For BED files (which is the format we have), options include **IGB, IGV,** and **UCSC main.**  IGB and IGV are widely used desktop applications and eventually you may want to install one or both of them.  For now, let's visualize the data at UCSC, using the UCSC *Genome* Browser.

> <hands-on-title>Display data in Genome Browser</hands-on-title>
>
> 1. Click on your `Overlapping genes sorted` dataset in your history panel. This will show the dataset preview in the history panel.
> 2. Click to expand the dataset, if it isn't already, so that you can see the dataset metadata and additional actions like Visualize.
> 3. Click on the {% icon galaxy-barchart %} (**Visualize**) icon
> 4. Click on the **display at UCSC (main)** link that appears in the blue box at the top of the screen.
>    ![visualisation options are shown in Galaxy's middle panel]({% link topics/introduction/images/101_displayucsc.png %})
>
> Possibly after some delay, this will launch a new window, showing UCSC's Genome Browser with our dataset shown right at the top.  UCSC figures out that our first overlapping gene is ~11 million bases into chromosome 22, and it has landed us there.
>
>   ![UCSC Genome Browser](../../images/ucsc_gb_genes_strands_overlapping.png)
{: .hands_on}


> <comment-title>Background: UCSC Genome Browser</comment-title>
>
> * *Genome browsers* are software for viewing genomic information graphically.  The [UCSC Genome Browser](https://genome.ucsc.edu/cgi-bin/hgGateway) (and most genome browsers) typically display different types of *annotation* about a region of a genome.  This is displayed as a stack of *tracks* and each track contains a different type of information.
>
> * Genome browsers are useful for seeing information in context and for seeing (and discovering) correlations between different types of information.  (They are also useful for visually checking results, which is what we are doing now.)
>
> * The UCSC Genome Browser has information on over 100 animals, and their [Archaeal Genome Browser](http://archaea.ucsc.edu/cgi-bin/hgGateway?db=pyrFur2) has genomic information on well over 100 microbial species.
{: .comment}

Now, take a look at one of our results.  (Any pair of overlapping genes will do.)  Our data is in the second to top track (**User Track / User Supplied Track**). That track shows a line of small black boxes, sometimes connected with a line.

> <hands-on-title>Zoom in on an area of the chromosome that shows a set of linked black boxes</hands-on-title>
> To zoom in,
> * *Click* on the **Scale** track (the top track) just to the left of the start of the black boxes.
> * Now *drag* the mouse across the Scale track to just to the right of the  black boxes and let go.
>   ![Zoom by clicking and dragging on the Scale track](../../images/ucsc_gb_zoom_by_drag.png)
> * A window pops up describing several ways to interact with the browser.  Just *click* the **Zoom In** button at the bottom.
> * This redraws the window, this time zoomed in to what you highlighted.
> * Continue to zoom in until you have the set of linked black boxes you picked centered on the screen.
> * Once you are as zoomed as you want to be, click on one of the linked boxes.  This will expand the track:
>   ![One gene, fully zoomed and expanded](../../images/ucsc_gb_zoom_to_1_gene.png)
{: .hands_on}

The black boxes connected by lines represent genes, and each set of connected boxes is a single gene (actually, a single transcript of a gene).  Take a close look at the top several tracks.

* It looks like we preserved the gene definitions just fine.
* It looks like, sure enough, there are overlapping genes here, and they are on opposite sides.  The arrows on the connecting lines tell us which strand the gene is on.

Um, *what's up with the boxes and the lines connecting them?*

> <comment-title>Definitions 4</comment-title>
>
> - **Exon**
>   In humans (and in all plants and animals) the molecules that are built from genes are often only built from a part of the DNA in the gene. The sections of DNA that can produce the molecules are called *exons*.
{: .comment}

As you may have guessed (or already knew): The black boxes are exons.  *Genes* are defined as *covering the entire area from the first black box to the last connected black box.*

## Do we have a problem?

*Maybe.*

Our *ad hoc* review of identified overlapping genes in the UCSC Genome Browser has *(or should have!)* confirmed that every gene we said has an overlapping gene on the opposite strand does in fact have that.  So, our conclusion appears solid: A significant percentage of genes *do* overlap with other genes on the opposite strand.

But, our conclusion may not be as *significant* as we had hoped. If only parts of genes, the exons, make stuff in our bodies, then should we have run this analysis on just the exons rather then the entire genes?  Probably.

Let's refine our question slightly

> <question-title>Our Revised Motivating Question</question-title>
> *I wonder if **exons** on opposite strands ever overlap with each other, and if so, how common is that?*
{: .question}


### So, I have to do this all over again, but with exons?

*Well, yes and no.*  We will have to run the analysis again, this time on exons instead of whole genes. But we won't have to manually recreate every step of our analysis. Galaxy enables us to create a reusable *workflow* from the analysis we just did, and then we can rerun the analysis, as one step, anytime we want to, and on any dataset.

## Walk through the workflow tutorial

Run the [Create a reusable workflow from a history]({% link topics/galaxy-interface/tutorials/history-to-workflow/tutorial.md %}) tutorial for how to do this, *and then come back here to run your newly created workflow with the exon data.*

# Rerun analysis with exon data

We want to run the same analysis, but this time only look for overlaps that happen in *exons*, the parts of genes that produce stuff our body uses.  Before we start looking at exons, let's start a new history, one that contains only the genes file we got from UCSC. We could go back to UCSC and refetch the file, but there is an easier way.

> <hands-on-title>Create a new history that contains some data from current history</hands-on-title>
>
> 1. *Click* on {% icon galaxy-history-options %} **History options** at the top of the history panel.
> 2. *Select* the **Copy Datasets** option from the pull down menu.  This launches the copy datasets form.
> 3. Under **Source History:**, *select* the dataset(s) you want to copy to the new history.
>    - In our case, we only want the `Genes chr22` dataset.
> 4. Under **Destination History** enter an informative history name in the **New history named:** box.
>    - For example, `Exon overlaps on opposite strands`
>      ![Copy only the Genes chr22  dataset to the new history](../../images/copy_datasets_to_new_history.png)
> 5. *Click* the **Copy History Items** button to create your new history.
>    - This creates a new history (with the copied dataset) and throws up a green box saying:
>      "1 dataset copied to 1 history: `name you gave your new history`."
> 6. The history name is a link. *Click* on it.
{: .hands_on}

## Get the exon data

And your new history appears in the history panel with the copied *genes* dataset. What we need is *exons*. How can we get the exon information? There are two relatively easy ways to get this information, one of which will be very familiar.

The first method involves going back to the UCSC Table Browser.  Everything on the first form would stay the same: We still want human, hg38, GENCODE v41, and just `chr22`. The second form is what changes. Instead of getting the **Whole Gene**, we need to retrieve the **Coding Exons** only.

The second method is to use the **Gene BED To Exon/Intron/Codon BED expander** tool in the **Operate on Genomic Intervals** toolbox to extract the exon information from the genes BED file we already have. (*Of course!* you say.  Umm, there is no way that you should have known that you already had this information in the genes file, or that this tool existed.)

> <hands-on-title>Get the exon data</hands-on-title>
>
> 1. Get the exon information, either by revisiting UCSC, or by using the {% tool [Gene BED To Exon/Intron/Codon BED expander](gene2exon1) %} tool.  If you use the expander tool select **Coding Exons only** from the **Extract** pull-down.
> 2. Give the resulting dataset a meaningful name.
{: .hands_on}


If you got the data from UCSC it will look something like this:
![Contents of the exons dataset from UCSC](../../images/101_exons.png)


Your history should now have two datasets: one describing entire genes, and one describing just the exons.

## Rerun the analysis, this time on exons.

When you did the *History to Workflow* tutorial you created a new workflow that was then added to your list of defined workflows.

> <hands-on-title>Run the workflow on the exon data.</hands-on-title>
>
> 1. *Click* the **Workflow** tab in the menu at the top of the Galaxy page.
>
>    This lists all of your defined workflows, including the one you just created.
> 2. *Click* on the {% icon workflow-run %} (Run workflow) button next to your workflow.
>
>    This launches the workflow run form (which you also saw when testing this workflow)
> 3. *Select* **No** under **Send results to a new history**
>
>    If you do not see this option displayed, *click* on **Expand to full workflow form** to reveal it.
> 4. *Set* the first (and only) input dataset to the `Exons` dataset, using the pull-down menu.
> 5. *Click* the **Run workflow** button.
{: .hands_on}

And Galaxy launches the workflow and says (in a nice big green box) something like:

> You can check the status of queued jobs and view the resulting data by refreshing the History pane.
{: .quote}

Which in this case *is the truth*.  You can refresh the history panel by either reloading the whole page, of by clicking the looping arrow icon at the top of the history panel.  What you'll see is a stack of history steps that will go from queued to running to done as you watch them.

All steps in the history will be green when the workflow is done. Once that happens compare your output dataset with your input dataset?  What percentage of exons overlap with other exons on the opposite strand, and is it common or rare?  Is it a smaller percentage than we saw for whole genes?


## Are we done?

Probably.  Note that we can no longer say what percentage of *genes* overlap.  We *can* say what percentage of exons overlap, and that is probably close enough for our goals.  If it isn't and we actually need to say what percentage of genes overlap, then we will have to do some extra work.  This can be done is several ways, but the Galaxy 101 tutorial may give you some ideas on how to follow this question all the way to genes.

# Conclusion

{% icon trophy %} Well done! You have just performed your first analysis in Galaxy!

{% snippet faqs/galaxy/why_not_excel.md %}

tools_json:
["Filter", "Filter1", "Intersect", "toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0", "Concatenate multiple datasets or collections", "cat1", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy3", "Gene BED To Exon/Intron/Codon BED expander", "gene2exon1"]


topic: transcriptomics
tutorial: rna-seq-analysis-clustering-viz
title: RNA-Seq data analysis, clustering and visualisation tutorial
difficulty: introductory
time_estimation: 6h
requirements_json: []
tags_json: []
content_hash: 788374e1e4f8b1b73cc7fdc2b760819e
last_modified: 2026-05-27T09:01:49.981594
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-analysis-clustering-viz/tutorial.html

content:
In this computer assignment, we will analyze the results of an RNA-seq dataset. This is one type of high-dimensional -omics data that biomedical scientists frequently use. The raw RNA-seq data are sequence reads, but we will use the processed data from an RNA-seq experiment, often referred to as a count matrix. A count matrix contains for every sample (in the columns of the matrix) and every gene or transcript (in the rows of the matrix) the number of sequencing reads representing that gene or transcript (an integer, i.e., 0, 1, 2, 3, ...). The expression level of that gene is derived from the read count by correcting (normalization) for the total number of sequencing reads in a particular sample.

RNA-seq is often used to explore the function of a gene or the effect of a genetic variant. Since RNA-seq can survey the expression levels of all genes, it is well suited to obtain a global idea of the effect on the physiology of a cell and the biological processes affected. In this computer assignment, we will explore a dataset from mouse B-cells from TP53-/- knockout and wild-type mice. It is a two-factor experiment, because cells of both genotypes have been analyzed with or without being exposed to ionizing radiation (IR); thus, genotype and exposure to IR represent different factors in the experiment. The research paper describing this dataset is {% cite Tonelli2015 %} ([direct link](http://dx.doi.org/10.18632/oncotarget.5232)). The TP53 gene coding for the p53 protein is a well-known tumor suppressor gene. p53 is involved in the induction of apoptosis, senescence, cell cycle arrest and metabolic reprogramming. p53 is activated by double strand DNA breaks and routes cells to apoptosis when the DNA damage cannot be repaired anymore. TP53 mutation is one the most common driver of mutations in cancer, because TP53-deficient cells continue to proliferate, even after significant DNA damage. IR induces double strand DNA breaks and cell death (apoptosis) and is a well-known activator of p53. The authors used TP53-/- cells and wild-type cells to explore which part of the response of B-cells to IR is dependent on p53.

> <comment-title>Full data</comment-title>
>
> The original data are available at NCBI Gene Expression Omnibus (GEO) under accession number [GSE71176](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE71176). The raw RNA-Seq reads have been extracted from the Sequence Read Archive (SRA) files and converted into FASTQ files and counts.
>
{: .comment}

# Data upload

In the first part of this tutorial we will upload the count matrices to your Galaxy environment.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-Seq exercise
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the count files from our [github repository](https://github.com/pacthoen/BMW2_RNA_clust_vis) using the Data Upload menu (top left) and the button _Paste/Fetch Data_.
>
>    ```
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_2.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_3.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_mock_4.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_2.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_3.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/p53_IR_4.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_mock_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_mock_2.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_IR_1.csv
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/null_IR_2.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    See also [this screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20115748.png)
>
>    *Note:* The files with null in the filename are for the p53 knockout (p53-/-) cells and the files with p53 in the filename are for the wild-type cells
>
>    **Alternative:** If this results in an error, you may download the zip file with all the input data from the [github site](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/data/input_data_Bcell_dataset.zip) to your local computer, unzip and import all the files using the _Choose local file_ option from the Upload menu.
>
> 3. **Inspect** {% icon galaxy-eye %} some of the files by hitting they eye button
> 4. Upload also the gene annotation file:
>
>    ```text
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/Mus_musculus.NCBIM37.65.gtf.gz
>    ```
>    **Alternative** Like before, if this results in an error, you can download the gtf.gz file from the [github site](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/data/Mus_musculus.NCBIM37.65.gtf.gz) to your local computer and import the gtf.gz file (no need for unzipping) using _Choose local file_ option from the Upload menu
>
>    _Note:_ The numbers in the count files are the number of sequence reads to a specific gene. Only mRNAs that were detected in all samples were selected during the preprocessing of the data.
>
> 5. **Set the datatype** {% icon galaxy-pencil %} of the annotation file to `gtf.gz`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="gtf.gz" %}
>
>    > <question-title></question-title>
>    >
>    > 1. How many mRNAs were detected in all samples?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1.By clicking on the dataset you can see the attributes of the dataset. There are 6,882 lines and a comment line. Thus, there are 6,882 detectable mRNAs in the dataset
>    > >
>    > {: .solution}
>    {: .question}
{: .hands_on}

# Analysis of the differential gene expression

## Identification of the differentially expressed features

To be able to identify differential gene expression between wild-type and p53-null cells and between IR- and mock-treated cells, we need to normalise our data first. In the lecture you have seen TPM and RPKM normalization. Here we use a bit more advance way of normalizing data using the Variance Stabilization and Transformation (VST) method from the DESeq2 package and tool. (see [DESeq2 documentation](http://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html#collapsing-technical-replicates)). DESeq2 also runs the Differential Gene Expression (DGE) analysis, which has two basic tasks:

- Estimate the biological variance using the replicates for each condition
- Estimate the significance of expression differences between any two conditions

This expression analysis is estimated from read counts and attempts are made to correct for variability in measurements using replicates, that are absolutely essential for accurate results.

> <details-title>Technical vs biological replicates</details-title>
>
> A technical replicate is an experiment which is performed once but measured several times (e.g. multiple sequencing of the same library). A biological replicate is an experiment performed (and also measured) several times.
>
> In our data, we have 2-4 biological replicates (here called samples) per condition.
>    > <question-title></question-title>
>    >
>    > 1. How many conditions are there in this experiment and what is the number of replicates in each of the conditions?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1.There are 4 conditions
>    > > - Wild-type cells not exposed to IR - 4 replicates
>    > > - Wild-type cells exposed to IR - 4 replicates
>    > > - TP53-/- cells not exposed to IR - 2 replicates
>    > > - TP53-/- cells exposed to IR - 2 replicates
>    > >
>    > {: .solution}
>    {: .question}
{: .details}

Multiple factors with several levels can then be incorporated in the analysis describing known sources of variation (e.g. treatment, tissue type, gender, batches), with two or more levels representing the conditions for each factor. After normalization we can compare the response of the expression of any gene to the presence of different levels of a factor in a statistically reliable way.

In our example, we have samples with two varying factors that can contribute to differences in gene expression:

- Genotype (either wild-type or knock-out)
- Treatment (either IR or mock)

The effect of Ionizing Radiation (IR) is our primary interest. Later on, we will explore the differences in IR response between wild-type and knock-out cells and which genes are dependent on p53 for their response to IR.
We will now assign each sample to the right group

We can now run **DESeq2**:

> <hands-on-title>Determine differentially expressed features</hands-on-title>
>
> 1. {% tool [DESeq2](toolshed.g2.bx.psu.edu/repos/iuc/deseq2/deseq2/2.11.40.8+galaxy0) %} with the following parameters:
>    - *"how"*: `Select datasets per level`
>        - In *"Factor"*:
>           - *"Specify the factor name"*: `Treatment`
>           - In *"1: Factor level"*:
>               - *"Specify the factor level"*: `IR`
>               - In *"Count file(s)"*: `Select all the IR treated count files`. _Note1:_ Use the _Switch to column select_ option to select the files. _Note2:_ The first factor level is compared to the second factor level. In this case IR vs. mock. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20123708.png)
>           - In *"2: Factor level"*:
>               - *"Specify the factor level"*: `Mock`
>               - In *"Count file(s)"*: `Select all the mock treated count files`
>       - {% icon param-repeat %} *"Insert Factor"*
>           - *"Specify the factor name"*: `Genotype`
>               - In *"Factor level"*:
>                    - {% icon param-repeat %} *"Insert Factor level"*
>                        - *"Specify the factor level"*: `KO`
>                        - In *"Count file(s)"*: `Select all the KO count files.`  i.e. those with null in the file name
>                    - {% icon param-repeat %} *"Insert Factor level"*
>                        - *"Specify the factor level"*: `WT`
>                        - In *"Count file(s)"*: `Select all the WT count files` i.e. those with p53 in the file name
>    - *"Files have header?"*: `Yes`
>    - *"Choice of Input data"*: `Count data (e.g. from HTSeq-count, featureCounts or StringTie)`
>    - In *"Advanced options"*:
>        - *"Use beta priors"*: `Yes`
>    - In *"Output options"*:
>        - *"Output selector"*: `Generate plots for visualizing the analysis results`, `Output normalised counts`, `Output VST normalized table`
>
{: .hands_on}


**DESeq2** {% icon tool %} generated 4 outputs:

- A table with the normalized counts for each gene (rows) in the samples (columns)
- A graphical summary of the results, useful to evaluate the quality of the experiment:

    1. A plot of the first 2 dimensions from a principal component analysis

    2. Heatmap of the sample-to-sample distance matrix (with clustering) based on the normalized counts.

        The heatmap gives an overview of similarities and dissimilarities between samples: the color represents the distance between the samples. Dark blue means shorter distance, i.e. closer samples given the normalized counts.


    3. Dispersion estimates: gene-wise estimates (black), the fitted values (red), and the final maximum a posteriori estimates used in testing (blue)

        This dispersion plot is typical, with the final estimates shrunk from the gene-wise estimates towards the fitted estimates. Some gene-wise estimates are flagged as outliers and not shrunk towards the fitted value. The amount of shrinkage can be more or less than seen here, depending on the sample size, the number of coefficients, the row mean and the variability of the gene-wise estimates.

    4. Histogram of *p*-values for the genes in the comparison between the 2 levels of the 1st factor

    5. An [MA plot](https://en.wikipedia.org/wiki/MA_plot):

        This displays the global view of the relationship between the expression change of conditions (log ratios, M), the average expression strength of the genes (average mean, A), and the ability of the algorithm to detect differential gene expression. The genes that passed the significance threshold (adjusted p-value < 0.1) are colored in blue.

- A summary file with the following values for each gene:

    1. Gene identifiers
    2. Mean normalized counts, averaged over all samples from both conditions
    3. Fold change in log2 (logarithm base 2)

       The log2 fold changes are based on the primary factor level 1 vs factor level 2, hence the input order of factor levels is important. Here, DESeq2 computes fold changes of 'treated' samples against 'untreated' from the first factor 'Treatment', *i.e.* the values correspond to up- or downregulation of genes in treated samples.

    4. Standard error estimate for the log2 fold change estimate
    5. [Wald](https://en.wikipedia.org/wiki/Wald_test) statistic
    6. *p*-value for the statistical significance of this change
    7. *p*-value adjusted for multiple testing with the Benjamini-Hochberg procedure, which controls false discovery rate ([FDR](https://en.wikipedia.org/wiki/False_discovery_rate))

    > <tip-title>What are p-values and what are they used for?</tip-title>
    >
    > The p-value is a measure often used to determine whether or not a particular observation possesses statistical significance. Strictly speaking, the p-value is the probability that the data could have arisen randomly, assuming that the null hypothesis is correct. In the concrete case of RNA-Seq, the null hypothesis is that there is no differential gene expression. So a p-value of 0.13 for a particular gene indicates that, for that gene, assuming it is not differentially expressed, there is a 13% chance that any apparent differential expression could simply be produced by random variation in the experimental data.
    >
    > 13% is still quite high, so we cannot really be confident differential gene expression is taking place. The most common way that scientists use p-values is to set a threshold (commonly 0.05, sometimes other values such as 0.01) and reject the null hypothesis only for p-values below this value. Thus, for genes with p-values less than 0.05, we can feel safe stating that differential gene expression plays a role. It should be noted that any such threshold is arbitrary and there is no meaningful difference between a p-value of 0.049 and 0.051, even if we only reject the null hypothesis in the first case.
    >
    > Unfortunately, p-values are often heavily misused in scientific research, enough so that Wikipedia provides a [dedicated article](https://en.wikipedia.org/wiki/Misuse_of_p-values) on the subject. See also [this article](https://fivethirtyeight.com/features/not-even-scientists-can-easily-explain-p-values/) (aimed at a general, non-scientific audience).
    {: .tip}

For more information about **DESeq2** and its outputs, you can have a look at the [**DESeq2** documentation](https://www.bioconductor.org/packages/release/bioc/manuals/DESeq2/man/DESeq2.pdf).

- An output file with Normalized counts
- An output file with VST-Normalized counts (VST stands for Variance Stabilization and Transformation; this is the preferred normalization method in DESeq2 and we will use these counts later on for PCA and clustering

Use the **filter** {% icon tool %} tool on the DESeq2 result file to identify the genes with absolute fold-change greater than 2. Use the condition: abs(c3)>2. Number of header lines to skip: 1. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20130228.png). And use the **sort** tool to identify the genes with the largest fold-change (most upregulated). See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20132050.png)

> <question-title></question-title>
>
> 1. What does it mean when an mRNA has an absolute 2logFC greater than 2?
> 2. How many mRNAs have an absolute 2logFC greater than 2?
> 3. What is the mRNA that is most upregulated in IR- vs mock-treated cells and what is its false discovery rate (the chance that this is a false positive)?
>
> > <solution-title></solution-title>
> >
> > 1. That the genes is 2^2 = 4 times higher or lower expressed in the cells treated with IR than in mock-treated cells
> > 2. 666 (!)
> > 3. ENSMUSG00000000308; FDR is in p-adj column. It is 1.3*10^-26
> >
> {: .solution}
{: .question}

## Annotation of the DESeq2 results

The ID for each gene is something like ENSMUSG00000026581, which is an ID from the Ensembl gene database. These IDs are unique but sometimes we prefer to have the gene names, even if they may not reference an unique gene (e.g. duplicated after re-annotation). But gene names may hint already to a function or they help you to search for desired candidates. We would also like to display the location of these genes within the genome. We can extract such information from the annotation file which we uploaded before.

> <hands-on-title>Annotation of the DESeq2 results</hands-on-title>
>
>
> 1. {% tool [Annotate DESeq2/DEXSeq output tables](toolshed.g2.bx.psu.edu/repos/iuc/deg_annotate/deg_annotate/1.1.0) %} with:
>    - {% icon param-file %} *"Tabular output of DESeq2/edgeR/limma/DEXSeq"*: the `DESeq2 result file` (output of **DESeq2** {% icon tool %})
>    - *"Input file type"*: `DESeq2/edgeR/limma`
>    - {% icon param-file %} *"Reference annotation in GFF/GTF format"*: imported gtf `Annotation file`
>    - See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20132922.png)
{: .hands_on}

The generated output is an extension of the previous file:

1. Gene identifiers
2. Mean normalized counts over all samples
3. Log2 fold change
4. Standard error estimate for the log2 fold change estimate
5. Wald statistic
6. *p*-value for the Wald statistic
7. *p*-value adjusted for multiple testing with the Benjamini-Hochberg procedure for the Wald statistic
8. Chromosome
9. Start
10. End
11. Strand
12. Feature
13. Gene name

> <question-title></question-title>
>
> 1. What is the Gene Symbol of the most overexpressed gene
> 2. On which chromosome is the most over-expressed gene located?
>
> > <solution-title></solution-title>
> >
> > 1. ENSMUSG00000000308 (the top-ranked gene with the highest positive log2FC value)
> > 2. It is located on the reverse strand of chromosome 2, between 121183449 bp and 121189473 bp.
> {: .solution}
{: .question}

The annotated table contains no column names, which makes it difficult to read. We would like to add them before going further.

> <hands-on-title>Add column names</hands-on-title>
>
> 1. In the Upload menu, create a new file using the _Paste/Fetch data_ option form the Upload menu, by directly pasting in the line below in the white area, and call it `header` instead of `New File`
>
>    ```text
>    GeneID	Base mean	log2(FC)	StdErr	Wald-Stats	P-value	P-adj	Chromosome	Start	End	Strand	Feature	Gene name
>    ```
>
>    {% snippet faqs/galaxy/datasets_create_new_file.md name="header" format="tabular" %}
>    See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20134800.png)
>
> 3. {% tool [Concatenate multiple datasets or collections](cat1) %} to add this header line to the **Annotate** output:
>    - {% icon param-file %} *"Concatenate Dataset"*: the `header` dataset
>    - *"Dataset"*
>       - Click on {% icon param-repeat %} *"Insert Dataset"*
>         - {% icon param-file %} *"select"*: output of **Annotate** {% icon tool %}
>
> 4. Rename the output to `Annotated DESeq2 results`
{: .hands_on}

## Create Volcano plot to visualise differentially expressed genes

> <hands-on-title></hands-on-title>
> 1. Select the tool **Volcano Plot**
>    In a Volcano plot, the 2logFC (x-axis) is plotted against the -log10(P-value).
>    - Choose the right column numbers. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20162602.png). Change the significant threshold to 0.05 and LogFC 2 as as thresholds to colour.
>    - Put in correct titles under _Plot options_.
>
> 2. Repeat the DESeq2 analysis but now with `Genotype` as first factor (contrasting KO and WT cells). Repeat the Volcano plot.
>
> > <question-title></question-title>
> >
> > 1. Is IR causing more up- or more downregulation of gene expression?
> > 2. Does the Treatment or the Genotype have a bigger effect on gene expression
> >
> > > <solution-title></solution-title>
> > >
> > > 1. There are more mRNA upregulated than downregulated. The upregulated genes can be found on the right side of the plot.
> > > 2. There are more mRNA with high FC and strong significance in the comparison between IR and mock-treated cells than between KO and WT cells.
> > {: .solution}
> {: .question}
{: .hands_on}



# Functional enrichment analysis of the DE genes

We have extracted genes that are differentially expressed in IR- vs. mock-treated samples. Now, we would like to know if the differentially expressed genes are enriched transcripts of genes which belong to more common or specific categories in order to identify biological functions that might be impacted.

## Gene Ontology analysis

[Gene Ontology (GO)](http://www.geneontology.org/) analysis is widely used to reduce complexity and highlight biological processes in genome-wide expression studies. However, standard methods give biased results on RNA-Seq data due to over-detection of differential expression for long and highly-expressed transcripts.

[**goseq**](https://bioconductor.org/packages/release/bioc/vignettes/goseq/inst/doc/goseq.pdf) ({% cite young2010gene %}) provides methods for performing GO analysis of RNA-Seq data while taking length bias into account. **goseq** could also be applied to other category-based tests of RNA-Seq data, such as KEGG pathway analysis, as discussed in a further section.

**goseq** needs 2 files as inputs:

- A tabular file with the differentially expressed genes from all genes assayed in the RNA-Seq experiment with 2 columns:
  - the Gene IDs (unique within the file), in uppercase letters
  - a boolean indicating whether the gene is differentially expressed or not (`True` if differentially expressed or `False` if not). We will use abs(2LogFC) > 2 as a criterion
- A file with information about the length of a gene to correct for potential length bias in differentially expressed genes

> <hands-on-title> Prepare input for goseq </hands-on-title>
>
> 1. Use the **Gene length and GC content** tool on the _Annotation file_ (gtf format). See this [screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20210701.png)
>    - *"Analysis to perform"*: `gene lengths only`
> 2. Merge the gene length and the annotated DESeq2 ouput file **DESeq2 result file** with primary factor Treatment using the **Join two datasets side by side** tool. In *"Keep the header lines"*: `No`. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20214135.png)
> 3. Use the **Compute on rows** tool to create a column with TRUE and FALSE using the following expression: `bool(float(c3)>2)`. In the *"Error handling"* choose in *"Autodetect column types"* `No` and *"Fail on references to non-existent columns"* `No` and *"If an expression cannot be computed for a row"* choose `Fill in a replacement value` and Replacement value `False`. See this [screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-22%20222441.png)
> 4. {% tool [Cut](Cut1) %} columns from a table with the following parameters:
>    - *"Cut columns"*: `c1,c9`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the output generated in step 3
> 5. Rename the output to `Gene IDs and differential expression - IR`
> 6. {% tool [Cut](Cut1) %} columns from a table with the following parameters:
>    - *"Cut columns"*: `c1,c8`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the output generated in step 3
> 7. Rename the output to `Gene length differential expression - IR`
{: .hands_on}

We have now the two required input files for goseq.

> <hands-on-title>Perform GO analysis</hands-on-title>
>
> 1. {% tool [goseq](toolshed.g2.bx.psu.edu/repos/iuc/goseq/goseq/1.50.0+galaxy0) %} with
>    - *"Differentially expressed genes file"*: `Gene IDs and differential expression`
>    - *"Gene lengths file"*: `Gene IDs and length`
>    - *"Gene categories"*: `Get categories`
>       - *"Select a genome to use"*: `mouse (mm10)`
>       - *"Select Gene ID format"*: `Ensembl Gene ID`
>       - *"Select one or more categories"*: `GO: Cellular Component`, `GO: Biological Process`, `GO: Molecular Function`
>    - In *"Output Options"*
>      - *"Output Top GO terms plot?"*: `Yes`
>      - *"Extract the DE genes for the categories (GO/KEGG terms)?"*: `Yes`
>
{: .hands_on}

**goseq** generates with these parameters 3 outputs:

1. A table (`Ranked category list - Wallenius method`) with the following columns for each GO term:

    1. `category`: GO category
    2. `over_rep_pval`: *p*-value for over-representation of the term in the differentially expressed genes
    3. `under_rep_pval`: *p*-value for under-representation of the term in the differentially expressed genes
    4. `numDEInCat`: number of differentially expressed genes in this category
    5. `numInCat`: number of genes in this category
    6. `term`: detail of the term
    7. `ontology`: MF (Molecular Function - molecular activities of gene products), CC (Cellular Component - where gene products are active), BP (Biological Process - pathways and larger processes made up of the activities of multiple gene products)
    8. `p.adjust.over_represented`: *p*-value for over-representation of the term in the differentially expressed genes, adjusted for multiple testing with the Benjamini-Hochberg procedure
    9. `p.adjust.under_represented`: *p*-value for under-representation of the term in the differentially expressed genes, adjusted for multiple testing with the Benjamini-Hochberg procedure

    To identify categories significantly enriched/unenriched below some p-value cutoff, it is necessary to use the adjusted *p*-value.

    > <question-title></question-title>
    >
    > 1. How many GO terms are over-represented with an adjusted P-value < 0.05? How many are under-represented?
    > 2. Do you find evidence that p53 plays a role in the response to IR?
    >
    > > <solution-title></solution-title>
    > >
    > > 1. 5 GO terms (4 from the Biological Processes category, 1 from the Molecular Function category) are over-represented.
    > > 2. Signal transduction by p53 class mediator is second in rank and contains 19 differentially expressed genes.
    > >    {% tool [Filter data on any column using simple expressions](Filter1) %} on c8 (adjusted p-value for over-represented GO terms) and c9 (adjusted p-value for under-represented GO terms)
    > >
    > > 2. For over-represented, 50 BP, 5 CC and 5 MF and for under-represented, 5 BP, 2 CC and 0 MF
    > >
    > >    {% tool [Group data](Grouping1) %} on column 7 (category) and count on column 1 (IDs)
    > >
    > {: .solution}
    {: .question}

2. A graph with the top 10 over-represented GO terms

    > <question-title></question-title>
    >
    > What is the x-axis? How is it computed?
    >
    > > <solution-title></solution-title>
    > >
    > > The x-axis is the percentage of genes in the category that have been identified as differentially expressed: $$100 \times \frac{numDEInCat}{numInCat}$$
    > >
    > {: .solution}
    {: .question}

3. A table with the differentially expressed genes (from the list we provided) associated to the GO terms (`DE genes for categories (GO/KEGG terms)`)

> <comment-title>Advanced tutorial on enrichment analysis</comment-title>
>
> In this tutorial, we covered GO enrichment analysis with **goseq**. You can also use the same method to identify metabolic and signal transduction pathways from the KEGG database (do not select plotting the Top GO pathways in that case). To learn other methods and tools for gene set enrichment analysis, please have a look at the ["RNA-Seq genes to pathways"]({% link topics/transcriptomics/tutorials/rna-seq-genes-to-pathways/tutorial.md %}) tutorial.
{: .comment}


# Visualisation of high-dimensional datasets
In the second part of this computer assignment we will use commonly used techniques to identify the main sources of variation and to evaluate which samples and sample groups are most similar. We will continue using the same dataset but first filter for the most variable genes. This is for computational efficiency and to reduce the effect of noise.


## Filter for most variable genes
For filtering the most variable genes, we will compute the coefficient of variation (CV). This is defined as the standard deviation divided by the mean.
> <hands-on-title> Filter for most variable genes </hands-on-title>
>
> 1. Use the the **Table compute** tool to calculate the standard deviation (std)
>    - *"Table"*: `VST normalized data`
>    - *"Input data has"*: `Column names on the first row` and `Row names on the first column`
>    - *"Type of table operation "*: `Compute expression across rows or columns`
>       - *"Calculate"*: `Custom`
>       - *"Custom function on vec"*: `vec.std()`
>       - *"For each"*: `row`
>       - *"Output formatting options"*: keep everything checked. Output should have column and row headers
>    - See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20085500.png)
> 2. Use the the **Table compute** tool to calculate the mean (mean)
>    - *"Table"*: `VST normalized data`
>    - *"Input data has"*: `Column names on the first row` and `Row names on the first column`
>    - *"Type of table operation "*: `Compute expression across rows or columns`
>       - *"Calculate"*: `Custom`
>       - *"Custom function on vec"*: `vec.mean()`
>       - *"For each"*: `row`
>       - *"Output formatting options"*: keep everything checked. Output should have column and row headers
> 3. Use the **Join two files** tool to merge the normalised count data with the standard deviation and mean datasets (two steps)
>    -  *"First line is a header line"*: `Yes`
> 4. Use the **Compute on row** tool to calculate the CV. Expression `c14/c15`. Label the new row as `CV`
> 5. Use the **Sort** tool to sort the genes based on CV in descending order. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20104444.png)
>    - *"Number of header lines"*: `1`
>    - *"Column selections"*
>       - *"on column"*: `16`
>       - *"in"*: `decsending order`
>       - *"Flavour"*: `Fast numeric sort (-n)`
> 6. Select the first 2000 rows using the **Select first** tool. See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20105700.png)
> 7. Select only the columns with data (and the first column with the gene identifiers; columns 1-13) using the **Cut** tool
> 8. Relabel the file as `Input data for PCA and clustering`
>
{: .hands_on}


## Principal Component Analysis (PCA)
PCA is a dimension reduction technique to identify the main sources of variation.
For this part of the assignment we will use the _multivariate_ tool{% icon tool %}. There is a separate [tutorial](https://training.galaxyproject.org/topics/metabolomics/tutorials/lcms-dataprocessing/tutorial.html#bibliography) for the **multivariate** tool applied to metabolomics data.

The tool needs a sampleMetadata file where the different groups / factors are defined. You will be able to colour the PCA score plots according to those groups.
The tool needs a featureMetadata file with the names (and optional characteristics) of the genes.

### Upload metadata

> <hands-on-title> Upload metadata</hands-on-title>
>
> 1. Create the featureMetadata file using the **cut** tool on the `Input data for PCA and clustering` object
>    - *"Cut columns"*: c1
> 2. Import the metadata files from our [github repository](https://github.com/pacthoen/BMW2_RNA_clust_vis) using the Data Upload menu (top left) and the button _Paste/Fetch Data_.
>
>    ```text
>    https://raw.githubusercontent.com/pacthoen/BMW2_RNA_clust_vis/refs/heads/main/data/GSE71176_sample_metadata.txt
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    *Note*: Make sure that the order of the sample names in the data matrix and the sampleMetadata file are the same!
>
> 3. **Set the datatype** {% icon galaxy-pencil %} to `Tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="Tabular" %}
>
> 4. Run {% tool [Multivariate](toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10) %} with the following parameters, according to this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20124140.png):
>    - {% icon param-file %} *"Data matrix file"*: `Input data for PCA and clustering`
>    - {% icon param-file %} *"Sample metadata file"*: Your uploaded metadata
>    - {% icon param-file %} *"Variable metadata file"*: `Filtered_variableMetadata` (output of the last _cut_ job)
>    - *"Number of predictive components"*: `2`
>    - *"Advanced graphical parameters"*: `Full parameter list`
>        - *"Sample colors"*: `Treatment`
>        - *"Amount by which plotting text should be magnified relative to the default"*: `0.4`
>    - *"Advanced computational parameters"*: `Use default`
>
{: .hands_on}

> <tip-title>What if I get an error?</tip-title>
>
> If the multivariate analysis tool gives an error, it may have to do with the fact that the column headers do not exactly match with the metadata or that there is another problem with the input files. You may solve this by first running the _*Check_format*_ tool and run the analysis again with the output files from that tool.
{: .tip}

Among the four output files generated by the tool, the one we are interested in is the *Multivariate_figure.pdf* file.
The PCA scores plot representing the projection of samples on the two first components is located at the bottom left corner. The PCA loadings plot is located at the bottom right corner.

> <question-title></question-title>
>
> 1. What is the cumulative percentage explained by the first two components.
> 2. Which one of the two factors in the experiment has the largest effect on the gene expression levels: the genotype (TP53-/- or wild-type) or the exposure to IR? Play with the plot colors and shapes and create a figure that nicely visualizes this.
>
>
> > <solution-title></solution-title>
> >
> > 1.  The score plots contains the percentage of variance explained in the axis titles. Principal component t1  explains 51% of total variability and t2 18% of total variability. The cumulative variance explained by the first two components is 51+18 = 69%.
> > 2. The t1 nicely separates the IR_treated cells from the mock-treated cells. t1 explains most of the variation thus IR treatment explains most of the variation. The t2 mainly separates the KO cells treated with IR, suggesting that KO cells respond differently to IR than WT cells, which makes sense because p53 is important for the response to IR.
> {: .solution}
{: .question}

> <hands-on-title> Repeat the analysis</hands-on-title>
>
> 1. Repeat the analysis but now with four principal components and plot principal component 3 at the x-axis (abscissa) and component 4 at the y-axis (ordinate)
> 2. Optional: one can also OPLS (Orthogonal Partial Least Squares) which is a supervised form of multivariate analysis
{: .hands_on}

## Clustering

We will use the **heatmap2** tool to perform the clustering of samples based on their mRNA expression patterns and evaluate the influence of different clustering algorithms (average linkage vs single linkage) and distance measures (Euclidean distance or Pearson correlation). See this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20152150.png)

> <hands-on-title>Clustering with Heatmap2 </hands-on-title>
>
> 1. {% tool [heatmap2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1) %} with the following parameters:
>    - *"Input"*: `Input data for PCA and clustering`
>    - *"Plot title"*: indicate the clustering algorithm and the distance measure used
>    - *"Data transformation"*: `Plot the data as is`. Because the data has already been VST normalized
>    - *"Compute Z-scores prior to clustering"*: `Compute on rows`. This brings all the genes on the same scale (mean zero and standard deviation. If you do not this (you can check!) the clustering will group high expressed genes together and low expressed together, whereas it is more interesting to cluster genes based on the differences seen between samples.
>    - *"Distance method"*: `Euclidean` OR `Pearson`
>    - *"Clustering method"*: `Average (=UPGMA)` OR `Single`
>    - *"Type of colormap to use"*: `Gradient with 3 colors`
>
>
> > <question-title></question-title>
> >
> > 1. How does the clustering of samples relate to the PCA results?
> > 2. Which differences do you observe between different clustering algorithms and different distance measures?
> > 3. Indicate the cluster of genes for which their IR response is p53-dependent. Are these genes mostly up- or downregulated after IR?
> >
> > > <solution-title></solution-title>
> > >
> > > 1. Again, the IR-exposed cells cluster separately from the mock-exposed cells.
> > >   There is not much difference between the TP53-/- and wild-type mock-exposed cells, whilst the IR-exposed TP53-/- and wild-type cells cluster separately.
> > > 2. The clustering is not very dependent on the Distance measure used. However, with single linkage, it is much more difficult to identify discrete clusters, because the single, closest feature is added each time to an already existing cluster. Average and complete linkage methods usually start with finding the closest features that make up distinct clusters.
> > > 3. There is a cluster of genes that is only upregulated in the cells that are wild-type for p53 and not in IR-treated knockout cells. These mRNAs are dependent on p53 expression for their upregulation after exposure to IR.
> > {: .solution}
> {: .question}
>
{: .hands_on}

# Conclusion

In this computer assignment, we have analyzed real RNA sequencing data to extract useful information, such as which genes are up or downregulated by IR. Your complete workflow can be extracted using the _Worklows_ button. Do this according to this [Screenshot](https://github.com/pacthoen/BMW2_RNA_clust_vis/blob/main/screenshots/Screenshot%202025-05-23%20155239.png).

{% snippet faqs/galaxy/workflows_extract_from_history.md %}

You can always rerun and adapt your entire workflow. A workflow as you could have obtained it can be found [on this tutorial's workflow page](./workflows/)
The output of this computer assignment can also be found in [this folder](https://github.com/pacthoen/BMW2_RNA_clust_vis/upload/main/results) on the github site.

tools_json:
["DESeq2", "toolshed.g2.bx.psu.edu/repos/iuc/deseq2/deseq2/2.11.40.8+galaxy0", "Annotate DESeq2/DEXSeq output tables", "toolshed.g2.bx.psu.edu/repos/iuc/deg_annotate/deg_annotate/1.1.0", "Concatenate multiple datasets or collections", "cat1", "Cut", "Cut1", "goseq", "toolshed.g2.bx.psu.edu/repos/iuc/goseq/goseq/1.50.0+galaxy0", "Filter data on any column using simple expressions", "Filter1", "Group data", "Grouping1", "Multivariate", "toolshed.g2.bx.psu.edu/repos/ethevenot/multivariate/Multivariate/2.3.10", "heatmap2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1"]


topic: transcriptomics
tutorial: rna-seq-counts-to-viz-in-r
title: RNA Seq Counts to Viz in R
difficulty: intermediate
time_estimation: 1H
tools_json: ["RStudio", "interactive_tool_rstudio"]
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 84557990c01a68c2e44f34706a00ce4e
last_modified: 2026-05-15T22:28:17.648940
zenodo_link: https://zenodo.org/record/3477564/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-counts-to-viz-in-r/tutorial.html

content:
This tutorial will show you how to visualise RNA Sequencing Counts with R

{% include topics/data-science/tutorials/r-basics/tutorial_origin.md %}

With RNA-Seq data analysis, we generated tables containing list of DE genes, their expression, some statistics, etc. We can manipulate these tables using Galaxy, as we saw in some tutorials, e.g. ["Reference-based RNA-Seq data analysis"]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), and create some visualisations.

Sometimes we want to have some customizations on visualization, some complex table manipulations or some statistical analysis. If we can not find a Galaxy tools for that or the right parameters, we may need to use programming languages as R or Python.

> <tip-title>Prerequisites</tip-title>
> It is expected that you are already somewhat familiar with the R basics (how to create variables, create and access lists, etc.) If you are not yet comfortable with those topics, we recommend that you complete the requirements listed at the start of this tutorial first, before proceeding.
>
> If you are starting with this tutorial, you will need to import a dataset:
>
> > <hands-on-title>Using datasets from Galaxy</hands-on-title>
> >
> > 1. Upload the following URL to Galaxy:
> >    ```
> >    https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular
> >    ```
> >
> > 2. Note the history ID of the dataset in your Galaxy history
> >
> > 3. {% tool [RStudio](interactive_tool_rstudio) %} in Galaxy provides some special functions to import and export from your history:
> >
> >    ```R
> >    annotatedDEgenes <- read.csv(gx_get(2), sep="\t") # will import dataset number 2 from your history, use the correct ID for your dataset.
> >    ```
> >
> >
> {: .hands_on}
>
> > <hands-on-title>Using datasets without Galaxy</hands-on-title>
> > 1. Read the tabular file into an object called `annotatedDEgenes`:
> >
> >    ```R
> >    ## read in a CSV file and save it as 'annotatedDEgenes'
> >    annotatedDEgenes <- read.csv("https://zenodo.org/record/3477564/files/annotatedDEgenes.tabular", sep="\t")
> >    ```
> >
> >  {: .tip}
> {: .hands_on}
{: .tip}

In this tutorial, we will take the list of DE genes extracted from DESEq2's output that we generated in the ["Reference-based RNA-Seq data analysis" tutorial]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}), manipulate it and create some visualizations.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

# Visualization

In RNA-seq data analysis and other omics experiments, visualization are an important step to check the data, their quality and represent the results found for publication.

`ggplot2` is a plotting package that makes it simple to create complex plots from data in a data frame. It provides a more programmatic interface for specifying what variables to plot, how they are displayed, and general visual properties. Therefore, we only need minimal changes if the underlying data change or if we decide to change from a bar plot to a scatter plot. This helps in creating publication quality plots with minimal amounts of adjustments and tweaking.

`ggplot2` graphics are built step by step by adding new elements, i.e. layers:

```R
ggplot(data = <DATA>, mapping = aes(<MAPPINGS>)) +
  <GEOM_FUNCTION>() +
  <GEOM_FUNCTION>() +
  <GEOM_FUNCTION>()
```

Adding layers in this fashion allows for extensive flexibility and customization of plots.

## Volcano plot

> <hands-on-title>First plot, step by step</hands-on-title>
>
> 1. Load `ggplot2`
>
>     ```R
>     library(ggplot2)
>     ```
>
>    > <tip-title>Didn't work?</tip-title>
>    > You might need to install ggplot2 first
>    >
>    > ```R
>    > > install.packages("ggplot2")
>    > ```
>    {: .tip}
>
> 3. Bind a plot to a specific data frame
>
>    ```R
>    ggplot(data = annotatedDEgenes)
>    ```
>
>    A new window will open in the bottom-right panel
>
> 4. Define a mapping (using the `aes` aesthetic function), by selecting the variables to be plotted and specifying how to present them in the graph, e.g. as x/y positions or characteristics such as size, shape, color, etc.
>
>     ```R
>     ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = P.value))
>     ```
>
>     X-axis is now the log2 FC and and Y-axis the p-value
>
>     > <comment-title>Data format</comment-title>
>     >
>     > Some `ggplot2` functions need data in the 'long' format:
>     > - a column for every dimension
>     > - a row for every observation
>     >
>     > Well-structured data will save you lots of time when making figures with `ggplot2`.
>     {: .comment}
>
> 5. Add information about the graphical representations of the data in the plot (points, lines, bars) using `geoms` function added via the `+` operator
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = P.value)) +
>          geom_point()
>    ```
>
>    > <comment-title>Graphical representations of the data in the plot</comment-title>
>    >
>    > `ggplot2` offers many different `geoms`; we will use some common ones today, including:
>    > - `geom_point()` for scatter plots, dot plots, etc.
>    > - `geom_boxplot()` for, well, boxplots!
>    > - `geom_line()` for trend lines, time series, etc.
>    {: .comment}
{: .hands_on}

We should obtain our first ggplot2 plot:

![Our first ggplot2 plot](../../images/rna-seq-counts-to-viz-in-r/firstPlot.png)

This plot is called a volcano plot, a type of scatterplot that shows statistical significance (P value) versus magnitude of change (fold change). The most upregulated genes are towards the right, the most downregulated genes are towards the left, and the most statistically significant genes are towards the top. With this plot, we can then quickly identify genes with large fold changes that are also statistically significant, i.e. probably the most biologically significant genes.

The current version of the plot is not really informative, mostly due to the high number of p-values close to zero. In order to resolve this, we will apply `-log10()` to the y-axis values.

> <hands-on-title>Volcano plot with log values on y-axis</hands-on-title>
>
> 1. Create volcano plot with log values on the y-axis
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value))) +
>         geom_point()
>    ```
{: .hands_on}

![Volcano Plot version 0](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v0.png)

> <question-title>Categorising expression levels</question-title>
>
> 1. Why are there no points with log2 FC between -1 and 1?
>
> > <solution-title></solution-title>
> >
> > The input file we used are the 130 genes with a significant adjusted p-values (below < 0.05) and an absolute fold change higher than 2 (log2 FC < -1 or log 2 FC > 1).
> >
> {: .solution}
{: .question}

> <comment-title>The `+` sign</comment-title>
>
> The `+` in the `ggplot2` package is particularly useful because it allows you to modify existing `ggplot` objects. This means you can easily set up plot templates and conveniently explore different types of plots, so the above plot can also be generated with code like this:
>
> ```R
> # Assign plot to a variable
> de_genes_plot <- ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value)))
>
> # Draw the plot
> de_genes_plot +
>     geom_point()
> ```
>
> The `+` sign used to add new layers must be placed at the end of the line containing the previous layer. If, instead, the `+` sign is added at the beginning of the line containing the new layer, `ggplot2` will not add the new layer and will return an error message.
>
> ```R
> # This is the correct syntax for adding layers
> > de_genes_plot +
>       geom_point()
>
> # This will not add the new layer and will return an error message
> > de_genes_plot
>     + geom_point()
> ```
{: .comment}

> <comment-title>Some extra comments</comment-title>
>
> Anything you put in the `ggplot()` function can be seen by any `geom` layers that you add (i.e., these are universal plot settings). This includes the `x-` and `y-axis` mapping you set up in `aes()`.
>
> You can also specify mappings for a given `geom` independently of the mappings defined globally in the `ggplot()` function.
{: .comment}

Building plots with `ggplot2` is typically an iterative process. We start by defining the dataset we'll use, lay out the axes, and choose a `geom`. We can now modify this plot to extract more information from it.

> <hands-on-title>Format plot</hands-on-title>
>
> 1. Add transparency (`alpha`) to avoid overplotting
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value))) +
>         geom_point(alpha = 0.5)
>    ```
>
>    ![Volcano Plot version 1](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v1.png)
>
> 2. Add colors for all the points
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value))) +
>        geom_point(alpha = 0.5, color = "blue")
>    ```
>
>    ![Volcano Plot version 2](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v2.png)
>
> 3. Color point based on their strand
>
>     ```R
>     ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>         geom_point(alpha = 0.5)
>     ```
>
>     ![Volcano Plot version 3](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v3.png)
>
>     We use a vector as an input to the argument color and `ggplot2` provides a different color corresponding to different values in the vector.
>
> 4. Add axis labels
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>        geom_point(alpha = 0.5) +
>          labs(x = "log2(Fold Change)",
>               y = "-log10(P-Value)")
>    ```
{: .hands_on}

We have now a nice Volcano plot:

![Volcano Plot version 5](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v5.png)

> <question-title>Standard error over the `-log10` of the adjusted `P-value`</question-title>
>
> Create a scatter plot of the standard error over the `-log10` of the adjusted `P-value` with the chromosomes showing in different colors. Make sure to give your plot relevant axis labels.
>
> > <solution-title></solution-title>
> >
> > ```
> > ggplot(data = annotatedDEgenes, aes(x = -log10(P.adj), y = StdErr, color = Chromosome)) +
> >   geom_point() +
> >      labs(x = "-log10(P.adj)",
> >           y = "StdErr")
> > ```
> >
> > ![Exercise Plot 1](../../images/rna-seq-counts-to-viz-in-r/ggplotExercise01.png)
> {: .solution}
{: .question}

## Volcano plot by chromosomes

`ggplot2` has a special technique called faceting that allows us to split one plot into multiple plots based on a factor included in the dataset. We will use it to split our volcano plot into five panels, one for each chromosome.

> <hands-on-title>Volcano plot by chromosomes</hands-on-title>
>
> 1. Split volcano plot by chromosome using `facet_grid`
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(. ~ Chromosome)
>    ```
>
>    ![Volcano Plot version 6](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v6.png)
>
> 2. Split volcano plot by chromosome using `facet_grid` with plots stacked vertically
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(Chromosome ~ .)
>    ```
>
>    ![Volcano Plot version 7](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v7.png)
>
>    The `facet_grid` geometry allows to explicitly specify how we want your plots to be arranged via formula notation (`rows ~ columns`). The `.` can be used as a placeholder that indicates only one row or column).
>
> 3. Add white background using `theme_bw()` to make the plot more readable when printed
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(Chromosome ~ .) +
>      theme_bw()
>    ```
>
>    > <details-title>`ggplot2` themes</details-title>
>    >
>    > In addition to `theme_bw()`, which changes the plot background to white, `ggplot2` comes with several other themes which can be useful to quickly change the look of your visualization.
>    >
>    > - `theme_minimal()` and `theme_light() `are popular
>    > - `theme_void()` can be useful as a starting point to create a new hand-crafted theme
>    >
>    > The complete list of themes is available on the [ggplot2 website](https://ggplot2.tidyverse.org/reference/ggtheme.html). The `ggthemes` package provides a wide variety of options (including an Excel 2003 theme). The [`ggplot2` extensions website](https://www.ggplot2-exts.org/) provides a list of packages that extend the capabilities of `ggplot2`, including additional themes.
>    {: .details}
>
> 4. Remove the grid
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = log2.FC., y = -log10(P.value), color = Strand)) +
>      geom_point() +
>      labs(x = "log2(Fold Change)",
>           y = "-log10(P-Value)") +
>      facet_grid(Chromosome ~ .) +
>      theme_bw() +
>      theme(panel.grid = element_blank())
>    ```
>
{: .hands_on}

![Volcano Plot version 8](../../images/rna-seq-counts-to-viz-in-r/volcanoPlot_v8.png)

> <question-title>Standard error over the `-log10` of the adjusted `P-value`</question-title>
>
> Create a scatter plot of the standard error over the `-log10` of the adjusted `P-value` with the chromosomes showing in different colors and one facet per strand. Make sure to give your plot relevant axis labels.
>
> > <solution-title></solution-title>
> >
> > ```
> > ggplot(data = annotatedDEgenes, aes(x = -log10(P.adj), y = StdErr, color = Chromosome)) +
> >  geom_point() +
> >  labs(x = "-log10(P.adj)",
> >       y = "StdErr") +
> >  facet_grid(Strand ~ .)
> > ```
> >
> > ![Exercise Plot 2](../../images/rna-seq-counts-to-viz-in-r/ggplotExercise02.png)
> >
> {: .solution}
{: .question}

## Barplot of the number of DE genes

We would like now to make a barplot showing the number of differentially expressed genes.

> <hands-on-title>Barplot of the number of DE genes</hands-on-title>
>
> 1. Create a barplot with `geom_bar` function of the number of DE genes for each feature with one plot per chromosome
>
>    ```R
>    ggplot(data = annotatedDEgenes, aes(x = Feature, fill = Chromosome)) +
>    geom_bar() +
>    facet_grid(Chromosome ~ .)
>    ```
{: .hands_on}

![Bar plot 1](../../images/rna-seq-counts-to-viz-in-r/barPlot_v1.png)

> <question-title>Remove the legend</question-title>
>
> The chromosome is labeled on the individual plot facets so we don't need the legend. Use the help file for `geom_bar` and any other online resources to remove the legend from the plot.
>
> > <solution-title></solution-title>
> >
> > ```
> > ggplot(data = annotatedDEgenes, aes(x = Feature, fill = Chromosome)) +
> >  geom_bar(show.legend = F) +
> >  facet_grid(Chromosome ~ .)
> > ```
> > ![Bar plot 2](../../images/rna-seq-counts-to-viz-in-r/barPlot_v2.png)
> >
> {: .solution}
{: .question}


> <question-title>Create a R script with plots</question-title>
>
> Take another few minutes to either improve one of the plots generated in this exercise or create a beautiful graph of your own (using the [RStudio `ggplot2` cheat sheet](https://raw.githubusercontent.com/rstudio/cheatsheets/main/data-visualization-2.1.pdf) for inspiration).
>
> Here are some ideas:
> 1. Change the size or shape of the plotting symbol
> 2. Change the name of the legend
> 3. Change the label of the legend
> 4. Use a different color palette (see the [cookbook here](http://www.cookbook-r.com/Graphs/Colors_(ggplot2)/)).
{: .question}

# Conclusion


Data manipulation and visualization are important parts of any RNA-Seq analysis. Galaxy provides several tools for that as explained in several tutorials:

- [Reference-based RNA-Seq]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %})
- [Visualization of RNA-seq results with heatmap2]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-heatmap2/tutorial.md %})
- [Visualization of RNA-seq results with Volcano Plot]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %})

But sometimes we need more customization and then need to use programming languages as R or Python.

Working with a programming language (especially if it's your first time) often feels intimidating, but the rewards outweigh any frustrations. An important secret of coding is that even experienced programmers find it difficult and frustrating at times – so if even the best feel that way, why let intimidation stop you? Given time and practice you will soon find it easier and easier to accomplish what you want.

Finally, we won't lie; R is not the easiest-to-learn programming language ever created. So, don't get discouraged! The truth is that even with the modest amount of R covered today, you can start using some sophisticated R software packages, and have a general sense of how to interpret an R script. Get through these lessons, and you are on your way to being an accomplished R user!


topic: transcriptomics
tutorial: goenrichment
title: GO Enrichment Analysis
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: eba748051f52eacd124e5fc8efa4f6f5
last_modified: 2026-05-15T22:28:17.600588
zenodo_link: https://zenodo.org/record/2565417
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/goenrichment/tutorial.html

content:
When we have a large list of genes of interest, such as a list of differentially expressed genes obtained from an RNA-Seq experiment, how do we extract biological meaning from it?

One way to do so is to perform functional enrichment analysis. This method consists of applying statistical tests to verify if genes of interest are more often associated to certain biological functions than what would be expected in a random set of genes. In this tutorial you will learn about enrichment analysis and how to perform it.

## What is the Gene Ontology?
The [Gene Ontology](http://www.geneontology.org/) (GO) is a structured, controlled vocabulary for the classification of gene function at the molecular and cellular level. It is divided in three separate sub-ontologies or GO types: biological process (e.g., signal transduction), molecular function (e.g., ATPase activity) and cellular component (e.g., ribosome). These sub-ontologies are structured as directed acyclic graphs (a hierarchy with multi-parenting) of GO terms.

![GO Example](../../images/transcriptomics_images/goenrichment_GOexample.png "QuickGO - http://www.ebi.ac.uk/QuickGO")

The GO Ontology, like other ontologies, are usually coded in the [OBO](http://www.geneontology.org/faq/what-obo-file-format) or the [OWL](http://www.geneontology.org/faq/what-owl-file) formats. It can be downloaded from the [Gene Ontology website](http://geneontology.org/page/download-ontology) or from the [OBO foundry](http://www.obofoundry.org/). You can also find in Galaxy tools that allow you to manipulate and extract information from OBO files, but this is outside the scope of this tutorial.

> <comment-title></comment-title>
> Take note of when and where you obtained your ontology file, as these are constantly being updated.
>
{: .comment}


## What are GO annotations?
Genes are associated to GO terms via GO annotations. Each gene can have multiple annotations, even of the same GO type. An important notion to take into account when using GO is that, according to the **true path rule**, a gene annotated to a term is also implicitly annotated to each ancestor of that term in the GO graph. GO annotations have evidence codes that encode the type of evidence supporting them: only a small minority of genes have experimentally verified  annotations; the large majority have annotations inferred electronically based on sequence homology or known patterns.

GO annotations can be obtained from the [Gene Ontology website](http://geneontology.org/page/download-go-annotations), or from species-specific databases. One useful resource to obtain GO annotations is [Ensembl biomart](http://www.ensembl.org/biomart/martview). Again, take note to when and from where you obtained your annotations. For example, if you obtained your data from Ensembl, record the release you used.

> ### Overview
>
> In this tutorial, we will deal with:
>
> 1. Functional Enrichment Analysis
> 2. Methods to simplify the results
> 3. Interpretation of the results
> {:toc}
>
{: .agenda}


# Functional Enrichment Analysis

To perform functional enrichment analysis, we need to have:
- A set of genes of interest (e.g., differentially expressed genes): **study set**
- A set with all the genes to consider in the analysis: **population set** (which must contain the study set)
- **GO annotations**, associating the genes in the population set to GO terms
- The **GO ontology**, with the description of GO terms and their relationships

For each GO term, we need to count the frequency (**k**) of genes in the study set (**n**) that are associated to the term, and the frequency (**K**) of genes in the population set (**N**) that are associated to the same term. Then we test how likely would it be to obtain at least **k** genes associated to the term if **n** genes would be randomly sampled from the population, given the frequency **K** and size **N** of the population.

The appropriate statistical test is the one-tailed variant of Fisher’s exact test, also known as the hypergeometric test for over-representation. When the one-tailed version is applied, this test will compute the probability of observing at least the sample frequency, given the population frequency.  The [hypergeometric distribution](https://en.wikipedia.org/wiki/Hypergeometric_distribution) measures precisely the probability of **k** successes in **n** draws, without replacement, from a finite population of size **N** that contains exactly **K** successful objects:

![Go Enrichment Formula](../../images/transcriptomics_images/goenrichment_formula.png)

For this first exercise we will use data from [Trapnell et al. 2014](https://www.ncbi.nlm.nih.gov/pubmed/22383036 "Trapnell et al. data"). In this work, the authors created an artificial dataset of gene expression in *Drosophila melanogaster*, where 300 random genes were set (insilico) to be differentially expressed between two conditions.

> <hands-on-title></hands-on-title>
>
> The data for this tutorial is available at [Zenodo](https://zenodo.org/record/2565417) to download. For convenience and reproducibility of results, we already added the GO ontology and annotations in the Zenodo repository.
>
> 1. **Create a new history**
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Upload to the Galaxy** the following files:
>    - go.obo
>    - drosophila_gene_association.fb
>    - trapnellPopulation.tab
>
>    > <tip-title>Upload data to Galaxy</tip-title>
>    > * **Click** on the upload button in the upper left of the interface.
>    >
>    > ![Galaxy Upload](../../images/transcriptomics_images/goenrichment_galaxy_upload.png)
>    >
>    > * Press **Choose local file** and search for your file.
>    > * Press **Start** and wait for the upload to finish.
>    {: .tip}
>
> 3. **Rename** the *go.obo* file to `GO` and *drosophila_gene_association.fb* file to `GO annotations Drosophila melanogaster`.
>
>    After you upload the files, and if you press the {% icon galaxy-eye %} (eye) icon of `trapnellPopulation.tab` it should look something like this:
>    ![Trapnell File](../../images/transcriptomics_images/goenrichment_trapnellFile.png "Trapnell file")
>
>
>    > <comment-title></comment-title>
>    > The study set represents the differentially expressed genes. These were chosen as having an adjusted p-value for the differential expression test (last column) smaller than a given threshold. In this case, we want to select the genes with an adjusted p-value < 0.05.
>    {: .comment}
>
> 4. **Filter data on any column using simple expressions** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `trapnellPopulation.tab`
>    - {% icon param-text %} *"With following condition"*: `c7 < 0.05`
> ![Galaxy Filter](../../images/transcriptomics_images/goenrichment_galaxyFilter.png)
>
> 5. This generates one file. **Rename** it to `trapnellStudy`.
>
>    > <comment-title></comment-title>
>    > Both files have the same type of information, the difference between them being the number of genes, as the genes in the study sample are a subset of the population.
>    {: .comment}
>
> 6. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `Go Annotations Drosophila melanogaster`
>    - {% icon param-file %} *"Study set File"*: `trapnellStudy`
>    - {% icon param-file %} *"Population set File"*: `trapnellPopulation.tab`
>    - Use the default options for the rest.
> ![GO Enrichment Trapnell](../../images/transcriptomics_images/goenrichment_galaxyTrapnell.png)
>
>
>    > <question-title></question-title>
>    >
>    > What were the results from running GOEnrichment?
>    >
>    > > <solution-title></solution-title>
>    > > This will generate 6 files with the respective default names: `goenrichment on trapnellStudy MF Table', `goenrichment on trapnellStudy BP Table', `goenrichment on trapnellStudy MF Table', `goenrichment on trapnellStudy MF Graph', `goenrichment on trapnellStudy BP Graph' and `goenrichment on trapnellStudy CC Graph'. The three Table files list the results of the statistical test for each GO Term, ordered by p-value, and the three Graph files are image files displaying a graph view of the enriched GO terms.
>    > {: .solution}
>    {: .question}
>
>    > <comment-title></comment-title>
>    > For each GO term we obtain a p-value corresponding to a single, independent test. Since we are making multiple similar tests, the probability of at least one of them being a false positive increases. Therefore we need to make a correction for multiple testing.
>    {: .comment}
>
>    > <question-title></question-title>
>    >
>    > How many significant terms do we get?
>    >
>    > > <solution-title></solution-title>
>    > > When we ask how many significant terms, we want to see GO terms that have a p-value < 0.05. According to the results, for Molecular Function we have 5 GO terms, Biological Process we have 43 GO terms and Component Cellular we have 10 GO terms.
>    > {: .solution}
>    {: .question}
>
> If you press the {% icon galaxy-eye %} (eye) icon of the *Molecular Function* file (`MF Trapnell`) you should see something like this:
>
> ![GO Enrichment Trapnell MF](../../images/transcriptomics_images/goenrichment_mfTrapnell.png)
>
>    > <question-title></question-title>
>    >
>    > Did you expect to see significant terms?
>    >
>    > > <solution-title></solution-title>
>    > > The ~300 genes should be random, so we wouldn't expect to see any enriched term. Nonetheless we still have significant terms.
>    > {: .solution}
>    {: .question}
{: .hands_on}

> <comment-title></comment-title>
>
> Let's go back a little bit, and reopen the `trapnellPopulation.tab` file. If you go through the file, you'll see genes with 'NA' as an adjusted p-value. This means that there are genes in our background population for which the differential expression test was not even performed (usually genes that were not expressed in any sample). These genes are irrelevant for this functional enrichment analysis. We also need to note that the study set does not include those genes!
{: .comment}

Let's remove the irrelevant genes from the background population (`trapnellPopulation.tab`), to see the differences in results.

> <hands-on-title></hands-on-title>
>
> 1. **Filter data on any column using simple expressions** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `trapnellPopulation.tab`
>    - {% icon param-text %} *"With following condition"*: `c7 != 'NA'`
> ![Galaxy Filter NA](../../images/transcriptomics_images/goenrichment_galaxyFilterNA.png)
>
> 2. This generates one file. **Rename** to `trapnellFilteredPopulation`.
> 3. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `Go Annotations Drosophila melanogaster`
>    - {% icon param-file %} *"Study set File"*: `trapnellStudy`
>    - {% icon param-file %} *"Population set File"*: `trapnellFilteredPopulation`
>    - Use the default options for the rest.
> ![GO Enrichment Trapnell new Population](../../images/transcriptomics_images/goenrichment_galaxyTrapnellNewPop.png)
>
> 4. **Rename** all 6 output files by appending `FilteredPop` to the name, to distinguish them from the previous outputs.
> Let's check the new graph **goenrichment on trapnellStudy MF Graph FilteredPop**.
>
> ![GO Enrichment Trapnell new Population MF](../../images/transcriptomics_images/goenrichment_mfTrapnellNew.png)
>
>
>    > <question-title></question-title>
>    >
>    > 1. How many significant terms do we get now?
>    > 2. Why do you see these differences?
>    >
>    > > <solution-title></solution-title>
>    > > - According to the results, in *Molecular Function* and *Biological Process* we have 0 GO terms and *Component Cellular* just 1 GO term.
>    > > - The background population genes that we removed are not random, they are usually genes that are expressed in specific conditions, tissues or time points. If they are included in the test, we will obtain false enrichments, as we saw.
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Simplification of graphs

Graphical views are essential, but sometimes the graph view can become overwhelming due to the size of the results. To exemplify this issue, we will next perform functional enrichment analysis using a more realistic dataset from a study using the mouse model organism. The original dataset can be found [on NCBI](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE30352). In this [study](https://www.nature.com/articles/nature10532), the authors compared the gene expression of several tissues. Here, we will use results from the comparison between heart and brain.

> <hands-on-title></hands-on-title>
>
> For the first exercise we will use as a study set the differential genes (padjusted<0.05).
> 1. **Upload to Galaxy** the mouse_brain_vs_heart.txt, Mus_musculus_annotations_biomart_e92.tab and mouse_brain_vs_heart.difgenes.txt files.
> 2. **Rename** the *mouse_brain_vs_heart.txt* file to `Mouse population`, *Mus_musculus_annotations_biomart_e92.tab* file to `GO annotations Mus musculus` and *mouse_brain_vs_heart.difgenes.txt* file to `Mouse diff`.
> 3. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus`
>    - {% icon param-file %} *"Study set File"*: `Mouse diff`
>    - {% icon param-file %} *"Population set File"*: `Mouse population`
>    - Select **'No'** in the Summarize Output option.
> ![GO Enrichment Mouse Diff](../../images/transcriptomics_images/goenrichment_galaxyMouseDiff.png)
>
> 4. This will generate 6 files with the names: `goenrichment on Mouse diff MF Table`, `goenrichment on Mouse diff BP Table`, `goenrichment on Mouse diff CC Table`, `goenrichment on Mouse diff MF Graph`, `goenrichment on Mouse diff BP Graph` and `goenrichment on Mouse diff CC Graph`.
> 5. Analyze the table and graph from *Biological Process*.
>
>
> ![GO Enrichment Mouse Diff BP](../../images/transcriptomics_images/goenrichment_bpMouseDiff.png)
>
>    > <comment-title></comment-title>
>    > As you can see the three graphs are very complex and difficult to analyze.
>    {: .comment}
{: .hands_on}

As you see, the number of enriched GO Terms is very high, with graphs that are too extensive to analyze manually. And this is despite the fact that GOEnrichment ignores singletons and skips dependent tests by default, precisely to avoid enrichment results that are too extensive and not informative.

The Summarize Output option in the GOEnrichment tool addresses this problem by conflating branches/families of enriched GO terms and selecting the most representative term(s) from them (usually 1-2 term per family). The greatly simplifies the results while retaining branch information, and thus ensuring that every enriched family of functions is present in the results. Some specificity is necessarily lost, but the trade-off is that the results become easier and more intuitive to analyze.

> <hands-on-title></hands-on-title>
>
> 1. **GOEnrichment** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus`
>    - {% icon param-file %} *"Study set File"*: `Mouse diff`
>    - {% icon param-file %} *"Population set File"*: `Mouse population`
>    - Use the default options for the rest (notice that by default the Summarize Option is on).
> ![GO Enrichment Mouse Diff Summarize](../../images/transcriptomics_images/goenrichment_galaxyMouseDiffSum.png)
> 2. Analyze again the table and graph from *Biological Process*.
>
>
> ![GO Enrichment Mouse Diff Summarize BP](../../images/transcriptomics_images/goenrichment_bpMouseDiffSum.png)
>
>    > <question-title></question-title>
>    > 1. Are there differences in complexity comparing the graph with and without the summarize output option?
>    > > <solution-title></solution-title>
>    > > 1. Yes, there are differences. As you can see, the activation of the Summarize option reduces the size of the graph because this parameter causes families of GO terms to be conflated. Each major branch in the full results is still present in the summarized results, but now is reduced to 1 or 2 most representative terms, leading to a graph that is much easier to interpret while still containing all the key functional information.
>    > {: .solution}
>    {: .question}
{: .hands_on}

Another approach to reduce the complexity of the results is to use a shallower version of GO, the GO slims. GO slims are transversal cuts of GO that cover all key branches but lack specific terms. Thus, using them leads to much simpler results than using the full GO, but also leads to a substantial loss in specificity, which is greater than that of the Summarize Output option. You can download slimmed versions of GO from the [Gene Ontology website](http://geneontology.org/page/go-subset-guide).

To test the GO slim approach, let us use the mouse dataset again. First, however, we need to use GOSlimmer tool to convert the annotations file from full GO to GO slim (as GO annotations are typically made to terms that are too specific to be in the GO slim, and thus need to be extended by the true path rule).

> <hands-on-title></hands-on-title>
>
> 1. **Upload to the Galaxy** the goslim_generic.obo file.
> 2. **Rename** the *goslim_generic.obo* file to `GO Slim`.
> 3. Run **GOSlimmer** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Full Gene Ontology File"*: `GO`
>    - {% icon param-file %} *"GOSlim File"*: `GO Slim`
>    - {% icon param-file %} *"Gene Product Annotation File"*: `GO annotations Mus musculus`
>
> ![Galaxy GO Slimmer](../../images/transcriptomics_images/goenrichment_galaxySlimer.png)
>
> This will generate one file called `Slim Annotations`.
{: .hands_on}

Now we will go use the GOEnrichment tool with the new Slim Annotations file and the same study set.

> <hands-on-title></hands-on-title>
>
> 1. **GOEnrichment** {% icon tool %} with the following parameters:
>    - Use the `GO Slim`, `Slim Annotations`, `Mouse diff` and `Mouse population` files.
>    - Use the default options.
> ![GO Enrichment Mouse Slim](../../images/transcriptomics_images/goenrichment_galaxyMouseSlim.png)
>
>    > <question-title></question-title>
>    >
>    > 1. What differences do you observe when comparing the results obtained with the GO Slim to those obtained with the full GO, with the Summarize Output option?
>    >
>    > *Component Cellular* with full GO
>    > ![GO Enrichment Mouse Summarize CC](../../images/transcriptomics_images/goenrichment_ccSum.png)
>    > *Component Cellular* with GO Slim
>    > ![GO Enrichment Mouse Slim CC](../../images/transcriptomics_images/goenrichment_ccSlimSum.png)
>    >
>    > > <solution-title></solution-title>
>    > > 1. The differences that you observe are due to the ontology used. When we apply the summarize option with the full GO, the GOEnrichment tool will return a summarized output (as we have seen previously). When we opted for GO Slim, the original annotation was already summarized, resulting in an even more summarized output, but with a consequent loss of specificity.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


# Interpretation of the results
The interpretation of the results will depend on the biological information that we intend to extract. Enrichment analysis can be used in validation (e.g., of a protocol for extracting membrane proteins), characterization (e.g., of the effects of a stress in a organism) and elucidation (e.g., of the functions impacted by the knock-out of a transcription factor).

There is one important point to keep in mind during the analysis: statistically significant is different from biologically meaningful. That said, it is typically possible to obtain some biological or technical insight about the underlying experiment from statistically enriched terms, even if it isn’t readily apparent.

Terms that are very generic tend to be difficult to interpret, because the meaning they convey is shallow. On the other hand, very specific terms are generally not integrative and thus not useful in interpreting a gene set collectively. The interesting terms are those that are sufficiently specific to transmit substantial biological meaning, yet generic enough to integrate multiple genes.

For the second exercice, we will continue to work with the same study set as before but now we analyze separately genes that are over- and under-expressed, and see the enriched GO terms presents in the brain and heart from the mouse.

> <hands-on-title></hands-on-title>
>
> 1. **Upload to Galaxy** the mouseOverexpressed.txt and the mouseUnderexpressed.txt files.
>
>    > <comment-title></comment-title>
>    >
>    > The differentially expressed genes can be identified using the adjusted p-value (also known as FDR). The logFC values indicate whether genes are more expressed (logFC>0) or less expressed (logFC<0) in one condition when comparing with another condition.
>    >
>    {: .comment}
>
> 2. **GOEnrichment** {% icon tool %} with the following parameters for the both study files (*mouseOverexpressed.txt* and the *mouseUnderexpressed.txt*).
>    - Use the `GO`, the `GO annotations Mus musculus` and the `Mouse population` files.
>    - Use the default options.
> ![GO Enrichment Mouse Overexpress](../../images/transcriptomics_images/goenrichment_galaxyMouseOver.png)
> ![GO Enrichment Mouse Underexpress](../../images/transcriptomics_images/goenrichment_galaxyMouseUnder.png)
>
> 3. This will generate 12 files, 6 for each sample file, like in previous cases.
>
>    > <question-title></question-title>
>    >
>    > Analyze both Biological Process tables. According to the study, which tissues are over- and underexpressed?
>    >
>    > > <solution-title></solution-title>
>    > > The samples correspond to the expressions that occur in the tissues referring to the brain and heart, so the results in the tables (and also in the graphs) should correspond to the specific functions of each organ. When we analyze the tables of enriched functional terms, we can see that the results from underexpressed genes reveal functions related to the brain. While in the case of the genes overexpressed, we identify functions related to muscle / heart function.
>    > {: .solution}
>    {: .question}
{: .hands_on}


# Conclusion

Functional enrichment is a good way to look for patterns in gene lists, but interpretation of results can become a complicated process. One way to reduce this complexity is to use the GOEnrichment tool. This tool not only performs the GO Enrichment test, showing us enriched GO terms from our sets, but also contains functionality to simplify the results and make them more easily interpretable. Independently of this, we need to be careful when choosing our genes of interest, but also the background set of genes against which we want to compare.


topic: transcriptomics
tutorial: full-de-novo
title: De novo transcriptome assembly, annotation, and differential expression analysis
difficulty: intermediate
time_estimation: 3H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 5c64eb561e4ce1ccd3c97347dceb34bb
last_modified: 2026-05-15T22:28:17.599979
zenodo_link: https://zenodo.org/record/3541678
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/full-de-novo/tutorial.html

content:
As a result of the development of novel sequencing technologies, the years between 2008 and 2012 saw a large drop in the cost of sequencing. Per megabase and genome, the cost dropped to 1/100,000th and 1/10,000th of the price, respectively. Prior to this, only transcriptomes of organisms that were of broad interest and utility to scientific research were sequenced; however, these developed in 2010s high-throughput sequencing (also called next-generation sequencing) technologies are both cost- and labor- effective, and the range of organisms studied via these methods is expanding.

Examining non-model organisms can provide novel insights into the mechanisms underlying the "diversity of fascinating morphological innovations" that have enabled the abundance of life on planet Earth. In animals and plants, the "innovations" that cannot be examined in common model organisms include mimicry, mutualism, parasitism, and asexual reproduction. De novo transcriptome assembly is often the preferred method to studying non-model organisms, since it is cheaper and easier than building a genome, and reference-based methods are not possible without an existing genome. The transcriptomes of these organisms can thus reveal novel proteins and their isoforms that are implicated in such unique biological phenomena.

[(source)](https://en.wikipedia.org/wiki/De_novo_transcriptome_assembly)

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Read cleaning (20 minutes)

Known sequencing biases:
- Unknown nucleotides (Ns)
- Bad quality nucleotides
- Hexamers biases (Illumina. Now corrected ?)

Why do we need to correct those?
- To remove a lot of sequencing errors (detrimental to the vast majority of assemblers)
- Because most de-bruijn graph based assemblers can’t handle unknown nucleotides

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the 12 `fq.gz` into a `List of Pairs` collection named `fastq_raw`
>    - Option 1: from a shared data library (ask your instructor)
>    - Option 2: from Zenodo using the URLs given below
>
>      [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3541678.svg)](https://doi.org/10.5281/zenodo.3541678)
>
>    ```
>    https://zenodo.org/record/3541678/files/A1_left.fq.gz
>    https://zenodo.org/record/3541678/files/A1_right.fq.gz
>    https://zenodo.org/record/3541678/files/A2_left.fq.gz
>    https://zenodo.org/record/3541678/files/A2_right.fq.gz
>    https://zenodo.org/record/3541678/files/A3_left.fq.gz
>    https://zenodo.org/record/3541678/files/A3_right.fq.gz
>    https://zenodo.org/record/3541678/files/B1_left.fq.gz
>    https://zenodo.org/record/3541678/files/B1_right.fq.gz
>    https://zenodo.org/record/3541678/files/B2_left.fq.gz
>    https://zenodo.org/record/3541678/files/B2_right.fq.gz
>    https://zenodo.org/record/3541678/files/B3_left.fq.gz
>    https://zenodo.org/record/3541678/files/B3_right.fq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md collection=true collection_type="List of Pairs" collection_name="fastq_raw" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to ...
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

## Quality control

> <hands-on-title>Task description</hands-on-title>
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>   - *"Short read data from your current history"*: `fastq_raw` (collection)
>
{: .hands_on}

<!-- ## Quality control with **MultiQC** - step 2/2

> <hands-on-title>Task description</hands-on-title>
>
> 1. **MultiQC** {% icon tool %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `FastQC`
>                - In *"FastQC output"*:
>                    - *"Type of FastQC output?"*: `Raw data`
>                    - *"FastQC output"*: `data XX, data XX, and others (flattened)`
>
>    > <comment-title></comment-title>
>    >
>    > We agree that it's not comfortable. The wrapper of MultiQC must be improved
>    {: .comment}
>
{: .hands_on} -->

## Read cleaning with **Trimmomatic**

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trimmomatic** {% icon tool %} with the following parameters:
>    - *"Single-end or paired-end reads?"*: `Paired-end (as collection)`
>    - *"Select FASTQ dataset collection with R1/R2 pair"*: `fastq_raw`
>    - *"Perform initial ILLUMINACLIP step?"*: `Yes`
>    - *"Adapter sequences to use"*: `TruSeq3 (additional seqs) (paired-ended, for MiSeq and HiSeq)`
>    - In *"Trimmomatic Operation"*:
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Cut bases off end of a read, if below a threshold quality (TRAILING)`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Cut bases off start of a read, if below a threshold quality (LEADING)`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Sliding window trimming (SLIDINGWINDOW)`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Drop reads with average quality lower than a specific level (AVGQUAL)`
>                - *"Minimum length of reads to be kept"*: `25`
>        - {% icon param-repeat %} *"Insert Trimmomatic Operation"*
>            - *"Select Trimmomatic operation to perform"*: `Drop reads below a specified length (MINLEN)`
>                - *"Minimum length of reads to be kept"*: `50`
>    - *"Output trimmomatic log messages?"*: `Yes`
> 2. **Rename** the Dataset Collection
>    - `Trimmomatic on collection XX: paired` -> `fastqc_cleaned`
>
>    > <comment-title></comment-title>
>    >
>    > You can check the Trimmomatic log files to get the number of read before and after the cleaning
>    > ```
>    > Input Read Pairs: 10000
>    > Both Surviving: 8804 (88.04%)
>    > Forward Only Surviving: 491 (4.91%)
>    > Reverse Only Surviving: 456 (4.56%) Dropped: 249 (2.49%)
>    > ```
>    {: .comment}
>
>    {% snippet faqs/galaxy/collections_rename.md %}
>
{: .hands_on}

## Quality control after cleaning

> <hands-on-title>Task description</hands-on-title>
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>   - *"Short read data from your current history"*: `fastqc_cleaned` (collection)
>
{: .hands_on}

# Assembly (120 minutes - computing)

## Assembly with **Trinity**

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinity** {% icon tool %} with the following parameters:
>    - *"Are you pooling sequence datasets?"*: `Yes`
>        - *"Paired or Single-end data?"*: `Paired-end collection`
>            - *"Strand specific data"*: `No`
>    - *"Run in silico normalization of reads"*: `No`
>    - In *"Additional Options"*:
>        - *"Use the genome guided mode?"*: `No`
> 2. **Rename** the Trinity output
>    - `Trinity on data 52, data 51, and others: Assembled Transcripts` -> `transcriptome_raw.fasta`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

# Assembly assessment / cleaning

## Checking of the assembly statistics

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinity Statistics** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_raw.fasta`
>
>    > <comment-title></comment-title>
>    > This step, even with this toy dataset, will take around 2 hours
>    {: .comment}
>
{: .hands_on}

## Remapping on the raw transcriptome

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Align reads and estimate abundance** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_raw.fasta`
>    - *"Paired or Single-end data?"*: `Paired`
>        - *"Left/Forward strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `left`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Right/Reverse strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `right`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Strand specific data"*: `Yes`
>    - *"Abundance estimation method"*: `Salmon`
>    - In *"Additional Options"*:
>        - *"Trinity assembly?"*: `Yes`
> 2. **Rename** the 6 `* isoforms counts` :(
>    - Check in the information panel (**i** icon) the lineage of your file (ex: `A1_left.fq.gz` ... )
>    - Rename the datasets: `A1_raw`, `A2_raw`, `A3_raw`, `B1_raw`, `B2_raw`, `B3_raw`.
>
>    > <comment-title></comment-title>
>    >
>    > If you check at the Standard Error messages of your outputs. You can get the `Mapping rate`
>    > 1. Click on one dataset
>    > 2. Click on the little **i** icon
>    > 3. Click on *Tool Standard Error:	stderr*
>    > ```
>    > [2019-11-14 15:44:21.500] [jointLog] [info] Mapping rate = 44.4358%
>    > ```
>    {: .comment}
>
>    > <comment-title></comment-title>
>    >
>    > At this stage, you can now delete some useless datasets
>    > - `Trimmomatic on collection XX: unpaired`
>    > - `Align reads and estimate abundance on *: genes counts`
>    > Note that the dataset are just hidden. You can delete them permanently and make some room in the history options (the little wheel icon)
>    {: .comment}
>
>
{: .hands_on}

## Merge the mapping tables and compute normalizations

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Build expression matrix** {% icon tool %} with the following parameters:
>    - *"Abundance estimates"*: `A1_raw`, `A2_raw`, `A3_raw`, `B1_raw`, `B2_raw`, `B3_raw`
>    - *"Abundance estimation method"*: `Salmon`
>
{: .hands_on}

> <question-title></question-title>
>
> What are the three tables?
>
> > <solution-title></solution-title>
> >
> > 1. `estimated RNA-Seq fragment isoform counts (raw counts)``
> > 2. `matrix of isoform TPM expression values (not cross-sample normalized)`
> > 3. `matrix of TMM-normalized expression values`
> >
> {: .solution}
>
{: .question}

## Compute contig Ex90N50 statistic and Ex90 transcript count

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Compute contig Ex90N50 statistic and Ex90 transcript count** {% icon tool %} with the following parameters:
>    - *"Expression matrix"*: `Build expression matrix: matrix of TMM-normalized expression values`
>    - *"Transcripts"*: `transcriptome_raw.fasta`
> 2. Click on the visulization icon on the dataset `Compute contig Ex90N50 statistic and Ex90 transcript count: ExN50 statistics`
>    1. **Scatterplot - Creates a 2D-scatterplot from tabular datapoints**
>    2. *"X Column"*: select the Columns `1`
>    3. *"Y Column"*: select the Columns `2`
>
{: .hands_on}

### What we get
![ExN50_plot_toy_dataset](../../images/full-de-novo/ExN50_plot_toy_dataset.png)

### What we should get with a real dataset
![ExN50_plot](../../images/full-de-novo/ExN50_plot.png)
[(source)](https://github.com/trinityrnaseq/trinityrnaseq/wiki/Transcriptome-Contig-Nx-and-ExN50-stats)

## Transcriptome annotation completeness

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Busco** {% icon tool %} with the following parameters:
>    - *"Sequence to analyse"*: `transcriptome_raw.fasta`
>    - *"Mode"*: `transcriptome`
>    - *"Lineage"*: `eukaryota_odb9`
>
{: .hands_on}

![RNASeq samples quality check Graphs](../../images/full-de-novo/rnaseq_samples_quality_check.png)

## Filter low expression transcripts

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Filter low expression transcripts** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_raw.fasta`
>    - *"Expression matrix"*: `Build expression matrix: matrix of isoform TPM expression values (not cross-sample normalized)`
>    - *"Minimum expression level required across any sample"*: `1.0`
>    - *"Isoform filtering method"*: `Keep all isoforms above a minimum percent of dominant expression`
>        - *"Minimum percent of dominant isoform expression"*: `1`
>
>    > <comment-title></comment-title>
>    >
>    > If you check at the Standard Error messages of your outputs. You can get the `Retained` rate
>    > 1. Click on one dataset
>    > 2. Click on the little **i** icon
>    > 3. Click on *Tool Standard Error:	stderr*
>    > ```
>    > 	Retained 2096 / 2102 = 99.71% of total transcripts.
>    > ```
>    {: .comment}
>
> 2. **Rename** the output
>    - `Filter low expression transcripts on data 42 and data 14: filtered low expression transcripts` -> `transcriptome_filtered.fasta`
>
{: .hands_on}

## Checking of the assembly statistics after cleaning

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinity Statistics** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_filtered.fasta`
>
{: .hands_on}


# Annotation
## Generate gene to transcript map

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Generate gene to transcript map** {% icon tool %} with the following parameters:
>    - *"Trinity assembly"*: `transcriptome_filtered.fasta`
>
{: .hands_on}

## Peptide prediction

> <hands-on-title>Task description</hands-on-title>
>
> 1. **TransDecoder** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_filtered.fasta`
>    - In *"Training Options"*:
>        - *"Select the training method"*: `Train with the top longest ORFs`
>
{: .hands_on}

## Similarity search

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Diamond** {% icon tool %} with the following parameters:
>    - *"What do you want to align?"*: `Align amino acid query sequences (blastp)`
>    - *"Input query file in FASTA or FASTQ format"*: `TransDecoder on data XXX: pep`
>    - *"Select a reference database"*: `Uniprot Swissprot`
>    - *"Format of output file"*: `BLAST Tabular`
>    - In *"Method to restrict the number of hits?"*: `Maximum number of target sequences`
>        - *"The maximum number of target sequence per query to report alignments for"*: `1`
> 3. **Rename** the Diamond output
>    - `Diamond on data XXX` -> `Diamond (blastp)`
> 2. **Diamond** {% icon tool %} with the following parameters:
>    - *"What do you want to align?"*: `Align DNA query sequences (blastx)`
>    - *"Input query file in FASTA or FASTQ format"*: `transcriptome_filtered.fasta`
>    - *"Select a reference database"*: `Uniprot Swissprot`
>    - *"Format of output file"*: `BLAST Tabular`
>    - In *"Method to restrict the number of hits?"*: `Maximum number of target sequences`
>        - *"The maximum number of target sequence per query to report alignments for"*: `1`
> 4. **Rename** the Diamond output
>    - `Diamond on data XXX` -> `Diamond (blastx)`
>
>    > <comment-title></comment-title>
>    >
>    > Note that you can both use **Diamond** {% icon tool %} or the **NCBI BLAST+ blastp** {% icon tool %} and **NCBI BLAST+ blast** {% icon tool %}
>    {: .comment}
>
{: .hands_on}

## Find signal peptides

> <hands-on-title>Task description</hands-on-title>
>
> 1. **SignalP 3.0** {% icon tool %} with the following parameters:
>    - *"Fasta file of protein sequences"*: `TransDecoder on data XXX: pep`
>
{: .hands_on}

## Find transmembrane domains

> <hands-on-title>Task description</hands-on-title>
>
> 1. **TMHMM 2.0** {% icon tool %} with the following parameters:
>    - *"FASTA file of protein sequences"*: `TransDecoder on data XXX: pep`
>
{: .hands_on}

## Search again profile database

> <hands-on-title>Task description</hands-on-title>
>
> 1. **hmmscan** {% icon tool %} with the following parameters:
>    - *"Sequence file"*: `TransDecoder on data XXX: pep`
>
{: .hands_on}

## Transcriptome annotation using **Trinotate**

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Trinotate** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_filtered.fasta`
>    - *"Peptides"*: `TransDecoder on data XXX: pep`
>    - *"Genes to transcripts map"*: `Generate gene to transcript map on data XXX: Genes to transcripts map`
>    - *"BLASTP: Peptides vs Uniprot.SwissProt"*: `Diamond (blastp)`
>    - *"BLASTX: Transcripts vs Uniprot.SwissProt"*: `Diamond (blastx)`
>    - *"HMMER hmmscan: Peptides vs PFAM"*: `Table of per-domain hits from HMM matches of TransDecoder on data XXX: pep against the profile database`
>    - *"TMHMM on Peptides"*: `TMHMM results`
>    - *"SignalP on Peptides"*: `SignalP euk results`
>    - *"Let Galaxy downloading the Trinotate Pre-generated Resource SQLite database"*: `Yes`
>
{: .hands_on}

# Differential Expression (DE) Analysis

## Remapping on the filtered transcriptome using

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Align reads and estimate abundance** {% icon tool %} with the following parameters:
>    - *"Transcripts"*: `transcriptome_filtered.fasta`
>    - *"Paired or Single-end data?"*: `Paired`
>        - *"Left/Forward strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `left`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Right/Reverse strand reads"* -> `Multiple datasets`
>            - Click on the *Folder* button at the right
>                - *Type to Search*: `right`
>                - Select the 6 `Trimmomatic on ..._left.fq.gz`
>        - *"Strand specific data"*: `Yes`
>    - *"Abundance estimation method"*: `Salmon`
>    - In *"Additional Options"*:
>        - *"Trinity assembly?"*: `Yes`
> 2. **Rename** the 6 `* isoforms counts` :(
>    - Check in the information panel (**i** icon) the lineage of your file (ex: `A1_left.fq.gz` ... )
>    - Rename the datasets: `A1`, `A2`, `A3`, `B1`, `B2`, `B3`.
>
>    > <comment-title></comment-title>
>    >
>    > If you check at the Standard Error messages of your outputs. You can get the `Mapping rate`
>    > 1. Click on one dataset
>    > 2. Click on the little **i** icon
>    > 3. Click on *Tool Standard Error:	stderr*
>    > ```
>    > [2019-11-14 15:44:21.500] [jointLog] [info] Mapping rate = 44.4358%
>    > ```
>    {: .comment}
>
>    > <comment-title></comment-title>
>    >
>    > At this stage, you can now delete some useless datasets
>    > - `Align reads and estimate abundance on *: genes counts`
>    > Note that the dataset are just hidden. You can delete them permanently and make some room in the history options (the little wheel icon)
>    {: .comment}
>
{: .hands_on}

## Merge the mapping tables and compute a TMM normalization

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Build expression matrix** {% icon tool %} with the following parameters:
>    - *"Abundance estimates"*: `A1`, `A2`, `A3`, `B1`, `B2`, `B3`
>    - *"Abundance estimation method"*: `Salmon`
> 2. **Describe samples and replicates**  {% icon tool %} with the following parameters:
>    - *"Samples"*
>        - *"1: Samples"*:
>            - *"Full sample name"*: `A1`
>            - *"Condition"*: `A`
>        - *"2: Samples"*:
>            - *"Full sample name"*: `A2`
>            - *"Condition"*: `A`
>        - ...:
>        - *"6: Samples"*:
>            - *"Full sample name"*: `B3`
>            - *"Condition"*: `B`
>
{: .hands_on}

## RNASeq samples quality check

> <hands-on-title>Task description</hands-on-title>
> 1. **RNASeq samples quality check** {% icon tool %} with the following parameters:
>    - *"Expression matrix"*: `Build expression matrix: estimated RNA-Seq fragment isoform counts (raw counts)`
>    - *"Samples description"*: `Describe samples`
>
{: .hands_on}

## Differential expression analysis

> <hands-on-title>Task description</hands-on-title>
> 1. **Differential expression analysis** {% icon tool %} with the following parameters:
>    - *"Expression matrix"*: `Build expression matrix: estimated RNA-Seq fragment isoform counts (raw counts)`
>    - *"Sample description"*: `Describe samples` (the last one)
>    - *"Differential analysis method"*: `DESeq2`
>
{: .hands_on}

## Extract and cluster differentially expressed transcripts

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Extract and cluster differentially expressed transcripts** {% icon tool %} with the following parameters:
>    - In *"Additional Options"*:
>        - *"Expression matrix"*: `Build expression matrix: estimated RNA-Seq fragment isoform counts (raw counts)`
>        - *"Sample description"*: `Describe samples`
>        - *"Differential expression results"*: `Differential expression results on data XXX and data XXX`
>        - *"p-value cutoff for FDR"*: `1`
>        - *"Run GO enrichment analysis"*: `No`
>
>    > <comment-title></comment-title>
>    >
>    > *"p-value cutoff for FDR"*: `1`
>    > Don't do this at home! It's because we have a Toy Dataset. The cutoff should be around `0.001`
>    {: .comment}
>
{: .hands_on}

## Partition genes into expression clusters

> <hands-on-title>Task description</hands-on-title>
>
> 1. **Partition genes into expression clusters** {% icon tool %} with the following parameters:
>    - *"RData file"*: `Extract and cluster differentially expressed transcripts: RData file`
>    - *"Method for partitioning genes into clusters"*: `Cut tree based on x percent of max(height) of tree`
>
{: .hands_on}

# Conclusion


Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.


topic: transcriptomics
tutorial: rna-seq-bash-star-align
title: RNA-seq Alignment with STAR
difficulty: intermediate
time_estimation: 90M
tools_json: []
requirements_json: []
tags_json: []
content_hash: eb9fe90a771b482110043a9a536a50ca
last_modified: 2026-05-15T22:28:17.647516
zenodo_link: https://zenodo.org/record/6457007
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-bash-star-align/tutorial.html

content:
In recent years, RNA sequencing (in short RNA-Seq) has become a very widely used technology to analyze the continuously changing cellular transcriptome, i.e. the set of all RNA molecules in one cell or a population of cells. One of the most common aims of RNA-Seq is the profiling of gene expression by identifying genes or molecular pathways that are differentially expressed (DE) between two or more biological conditions. This tutorial demonstrates a computational workflow for counting and locating the genes in RNA sequences. The first and most critical step in an RNA-seq analysis.

In the study of {% cite brooks2011conservation %}, the authors identified genes and pathways regulated by the *Pasilla* gene (the *Drosophila* homologue of the mammalian splicing regulators Nova-1 and Nova-2 proteins) using RNA-Seq data. They depleted the *Pasilla* (*PS*) gene in *Drosophila melanogaster* by RNA interference (RNAi). Total RNA was then isolated and used to prepare both single-end and paired-end RNA-Seq libraries for treated (PS depleted) and untreated samples. These libraries were sequenced to obtain RNA-Seq reads for each sample. The RNA-Seq data for the treated and the untreated samples can be compared to identify the effects of *Pasilla* gene depletion on gene expression.

In this tutorial, we illustrate the analysis of the gene expression data step by step using 7 of the original datasets:

- 4 untreated samples: [GSM461176](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461176), [GSM461177](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461177), [GSM461178](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461178), [GSM461182](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461182)
- 3 treated samples (*Pasilla* gene depleted by RNAi): [GSM461179](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461179), [GSM461180](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461180), [GSM461181](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM461181)

Each sample constitutes a separate biological replicate of the corresponding condition (treated or untreated). Moreover, two of the treated and two of the untreated samples are from a paired-end sequencing assay, while the remaining samples are from a single-end sequencing experiment.

> <comment-title>Full data</comment-title>
>
> The original data are available at NCBI Gene Expression Omnibus (GEO) under accession number [GSE18508](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE18508). The raw RNA-Seq reads have been extracted from the Sequence Read Archive (SRA) files and converted into FASTQ files.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title></comment-title>
>
> This tutorial is significantly based on Galaxy's ["Reference-based RNA-Seq data analysis"]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}) tutorial.
>
{: .comment}

# Data Upload

At first, we need to put our hands on our data. In this tutorial we will use the files for 2 out of the 7 samples to demonstrate how to calculate read counts (a measure of the gene expression) from FASTQ files (quality control, mapping, read counting). The FASTQ files for the other 5 samples, are provided, in case you want to reproduce the whole analysis later.
The "Data Upload" process is the only one in this tutorial that takes place directly into Galaxy's interface.


> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the FASTQ file pairs from [Zenodo]({{ page.zenodo_link }}) or a data library:
>    - `GSM461177` (untreated): `GSM461177_1` and `GSM461177_2`
>    - `GSM461180` (treated): `GSM461180_1` and `GSM461180_2`
>
>    ```text
>    {{ page.zenodo_link }}/files/GSM461177_1.fastqsanger
>    {{ page.zenodo_link }}/files/GSM461177_2.fastqsanger
>    {{ page.zenodo_link }}/files/GSM461180_1.fastqsanger
>    {{ page.zenodo_link }}/files/GSM461180_2.fastqsanger
>    ```
>
> 2. {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    > <comment-title></comment-title>
>    >
>    > Note that these are the full files for the samples and ~1.5Gb each so it may take some minutes to import.
>    >
>    > For a quicker run-through of the FASTQ steps a small subset of each FASTQ file (~5Mb) can be found here on [Zenodo]({{ page.zenodo_link }}):
>    >
>    > ```text
>    > {{ page.zenodo_link }}/files/GSM461177_1_subsampled.fastqsanger
>    > {{ page.zenodo_link }}/files/GSM461177_2_subsampled.fastqsanger
>    > {{ page.zenodo_link }}/files/GSM461180_1_subsampled.fastqsanger
>    > {{ page.zenodo_link }}/files/GSM461180_2_subsampled.fastqsanger
>    > ```
>    >
>    {: .comment}
>
> 4. Change the datatype from `fastqsanger` to `fastq`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastq" %}
>
> 5. We also need to import two more files, essential for the alignment operation (and basically every alignment procedure): the organism's reference genome (here *D. melanogaster*) and the organism's gene annotation.
>    Those can be aquired directly via link and Galaxy's data library as described above. For this tutorial we are going to use the files [dm6.fa.gz](https://hgdownload.soe.ucsc.edu/goldenPath/dm6/bigZips/dm6.fa.gz) and [Drosophila_melanogaster.BDGP6.87.gtf (dm6)](https://usegalaxy.eu/libraries/folders/F30cab321d898d2fb/dataset/02c5f7fcdb6bf41f). Note that it is essential to convert genome's file from `*.fa.gz` to `*.fa`. That is easy now that we have already used the same method to convert `fastqsanger` to `fastq`. Remember to change the name of the file, too, in your working history as Galaxy will not do it automatically. Doing so will prevent any confusions later on.
>
> {% snippet faqs/galaxy/datasets_rename.md name="dm6.fa" %}
>
{: .hands_on}

{% include topics/sequence-analysis/tutorials/quality-control/fastq_question.md %}

Keep in mind that the reads are raw data from the sequencing machine without any pretreatments. They need to be assessed for their quality.

# Tools to use

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

> <hands-on-title>Software to use</hands-on-title>
>
> The R Console and other interactive tools like RStudio are great for prototyping code and exploring data, but sooner or later we will want to use our program in a pipeline or run it in a shell script to process thousands of data files. This is one of those cases and, in order to do that, we will use the terminal provided by the RStudio itself.
> We go to "Tools" and pick the "Shell..." option and we are good to go. Our workspace is the left, terminal window that just opened.
>
> Fortunately, [miniconda](https://docs.conda.io/en/latest/miniconda.html) is already installed. Miniconda is a package manager that simplifies the installation processes. We can and will use it to install every essential package for our tutorial. However, it is of critical importance that we do that in an new environment within our existing base and install our packages in said environment.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ conda create -n name_of_your_env fastqc cutadapt star samtools subread
> > $ conda activate name_of_your_env
> > ```
> {: .code-in}
>
>
> | Software | Version | Manual | Available for | Description |
> | -------- | ------------ | ------ | ------------- | ----------- |
> | [fastqc](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) | 0.12.1 | [fastqc Manual](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/) | Linux, MacOS, Windows | A quality control tool for high throughput sequence data. |
> | [Cutadapt](https://cutadapt.readthedocs.io/en/stable/index.html) | 4.4 | [Cutadapt Manual](https://cutadapt.readthedocs.io/en/stable/guide.html) | Linux, MacOS | Finds and removes adapter sequences, primers, poly-A tails and other types of unwanted sequence from your high-throughput sequencing reads. |
> | [STAR](https://github.com/alexdobin/STAR) | 2.7.10b | [STAR Manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf) | Linux, MacOS | Universal RNA-seq aligner. |
> | [SAMtools](http://samtools.sourceforge.net/) | 1.17 | [SAMtools Manual](http://www.htslib.org/doc/samtools.html) | Linux, MacOS | Utilities for manipulating alignments in the SAM format. |
> | [Subread](https://subread.sourceforge.net/) | 2.0.3 | [featureCounts Manual](https://subread.sourceforge.net/featureCounts.html) | Linux, MacOS, Windows | Subread is a general-purpose read aligner which can align both genomic DNA-seq and RNA-seq reads and it includes **featureCounts**, a highly efficient general-purpose read summarization program that counts mapped reads for genomic features.  |
>
{: .hands_on}

# Data from Galaxy to RStudio 

Okay, our data are imported into our Gaxaly History we are working on, but we are gonna need to access them through RStutio in order to work with them.
Fortunately, this step is pretty simple with Galaxy:

> <hands-on-title>Get Data</hands-on-title>
> > <code-in-title> The gx_get() Function</code-in-title>
> > ```R
> > gx_get(number_of_file_you_want_to_import_as_given_in_Galaxy_History)
> > ```
> {: .code-in}
> The above function will import the files selected in RStudio and it will return the path where they will be stored. We will execute it multiple times, one for every distinct file (6 in our occasion). And now every time we are going to need one of the files imported we are going to use the path returned by the function.
>
{: .hands_on}

# Quality Control

During sequencing, errors are introduced, such as incorrect nucleotides being called. These are due to the technical limitations of each sequencing platform. Sequencing errors might bias the analysis and can lead to a misinterpretation of the data. Adapters may also be present if the reads are longer than the fragments sequenced and trimming these may improve the number of reads mapped.

Sequence quality control is therefore an essential first step in your analysis. We will use similar tools as described in the ["Quality control" tutorial]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}): [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) to create a report of sequence quality and [Cutadapt](https://cutadapt.readthedocs.io/en/stable/guide.html) ({% cite marcel2011cutadapt %}) to improve the quality of sequences via trimming and filtering.

> <hands-on-title>FastQC</hands-on-title>
>
> When using fastqc to check the quality of our reads, it would be nice to have a folder where our results will be saved. From there we can access them and assess the quality of the reads given.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ mkdir qualityRaw
> > $ fastqc /import/number_of_dataset_1_as_described_above -o qualityRaw/
> > $ fastqc /import/number_of_dataset_2_as_described_above -o qualityRaw/
> > $ fastqc /import/number_of_dataset_3_as_described_above -o qualityRaw/
> > $ fastqc /import/number_of_dataset_4_as_described_above -o qualityRaw/
> > ```
> {: .code-in}
>
{: .hands_on}

> <hands-on-title>Cutadapt</hands-on-title>
>
>  Now, it would be a good idea to trim the reads in order to get rid of bases that were sequenced with high uncertainty (i.e. low quality bases) at the read ends and also remove the reads of overall bad quality. We can also use this opportunity to restore the trimmed files names to make our job easier down the road. 
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ mkdir trimmedData
> > $ cutadapt /import/2 /import/3_reverse_file_of_2 -o /trimmedData/GSM461177_R1 -p /trimmedData/GSM461177_R2
> > ```
> {: .code-in}
> The same trimming procedure should take place for the second pair of reads (forward and reverse as above). After that, the files we are going to work with are the ones located in the **trimmedData** folder (4 in our case). 
>
{: .hands_on}

> <hands-on-title>FastQC on trimmed data</hands-on-title>
>
> A second quality check can take place, same as before, only this time applied on the trimmed data files located in the **trimmedData** folder.
>
> > <code-in-title>Environment and Packages</code-in-title>
> > ```bash
> > $ mkdir qualityTrimmed
> > $ fastqc /trimmedData/GSM641177_R1 -o qualityRaw/
> > $ fastqc /trimmedData/GSM641177_R2 -o qualityRaw/
> > $ fastqc /trimmedData/GSM641180_R1 -o qualityRaw/
> > $ fastqc /trimmedData/GSM641180_R2 -o qualityRaw/
> > ```
> {: .code-in}
>
{: .hands_on}

# Alignment to a reference genome

![Simplified workflow that starts with the raw sequence reads and ends up to genome alignment](../../images/bash-star-rnaseq/workflow_align.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

We perform read alignment or mapping to determine where in the genome our reads originated from. There are a number of tools to choose from and, while there is no gold standard, there are some tools that are better suited for particular NGS analyses. We will be using the [STAR Aligner (STAR)](https://github.com/alexdobin/STAR), which is a universal aligner for mapping sequences against a large reference genome.

The alignment process consists of two steps:

1. Indexing the reference genome
2. Aligning the reads to the reference genome

# Setting up

> <hands-on-title>Index the reference genome</hands-on-title>
> Our first step is to index the reference genome for use by STAR. Indexing allows the aligner to quickly find potential alignment sites for query sequences in a genome, which saves time during alignment. Indexing the reference only has to be run once. The only reason you would want to create a new index is if you are working with a different reference genome or you are using a different tool for alignment.
>
> > <code-in-title>Indexing with `STAR`</code-in-title>
> > ```bash
> > $ mkdir index
> > $ STAR --runThreadN 16 --runMode genomeGenerate --genomeDir ~/index --genomeFastaFiles /import/14 --sjdbGTFfile /import/15 --sjdbOverhang 100 --genomeSAindexNbases 12
> > ```
> {: .code-in}
>
> While the index is created and saved in the **index** folder, you will see output that looks something like this:
>
> > <code-out-title></code-out-title>
> > ```
> > STAR version: 2.7.10b   compiled: 2022-11-01T09:53:26-04:00 :/home/dobin/data/STAR/STARcode
> > STAR.master/source
> > Date Time ..... started STAR run
> > Date Time ... starting to generate Genome files
> > Date Time ..... processing annotations GTF
> > Date Time ... starting to sort Suffix Array. This may take a long time...
> > Date Time ... sorting Suffix Array chunks and saving them to disk...
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Align reads to reference genome</hands-on-title>
>
> The alignment process consists of choosing an appropriate reference genome to map our reads against and then deciding on an aligner. We will use the STAR algorithm as said.
>
> An example of what a `STAR` command looks like is below. This command will not run, as we do not have the files: `input_file_R1` or `input_file_R2`.
>
>
> ```
> $ STAR --genomeDir /index_folder --runThreadN 16 --readFilesIn /trimmedData/input_file_R1 /trimmedData/input_file_R2
> ```
>
>
> Have a look at the [STAR manual](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf), while we are running STAR with the default
>
> parameters here, your use case might require a change of parameters. *NOTE: Always read the manual page for any tool before using
> and make sure the options you use are appropriate for your data.*
>
> We are going to start by aligning the reads from just one of the
> samples in our dataset (GSM461177). After that, we will be
> iterating the following process on the second pair of our sample files (GSM461180).
>
> > <code-in-title>Aligning with STAR</code-in-title>
> > ```bash
> > $ STAR --genomeDir ~/index --runThreadN 16 -- readFilesIn /trimmedData/GSM461177_R1 /trimmedData/GSM461177_R2 --outFileNamePrefix GSM461177
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > ```
> > Date Time ..... started STAR run
> > Date Time ..... loading genome
> > Date Time ..... started mapping
> > ```
> {: .code-out}
>
{: .hands_on}

# SAM/BAM format

The [SAM file](https://genome.sph.umich.edu/wiki/SAM),
is a tab-delimited text file that contains information for each individual read and its alignment to the genome. While we do not
have time to go into detail about the features of the SAM format, the paper by
[Heng Li et al.](http://bioinformatics.oxfordjournals.org/content/25/16/2078.full) provides a lot more detail on the specification.

**The compressed binary version of SAM is called a BAM file.** We use this version to reduce size and to allow for *indexing*, which enables efficient random access of the data contained within the file.

The file begins with a **header**, which is optional for SAM files. The header is used to describe the source of data, reference sequence, method of
alignment, etc., this will change depending on the aligner being used. Following the header is the **alignment section**. Each line
that follows corresponds to alignment information for a single read. Each alignment line has **11 mandatory fields** for essential
mapping information and a variable number of other fields for aligner specific information. An example entry from a SAM file is
displayed below with the different fields highlighted.

![sam file example with fields description, showing qname, flag, rname, pos, maqp, and cigar as the first fields](../../images/bash-star-rnaseq/sam_bam.png)

Image from [Data Wrangling and Processing for Genomics](https://datacarpentry.org/wrangling-genomics/)

![sam file example with fields description, showing seq, qual, mrnm, mpos, and size as the second set of fields.](../../images/bash-star-rnaseq/sam_bam3.png)

Image from ["Data Wrangling and Processing for Genomics"](https://datacarpentry.org/wrangling-genomics/)

> <hands-on-title>SAM to BAM</hands-on-title>
> > <code-in-title>bash</code-in-title>
> > We will convert the SAM file to BAM format using the `samtools` program with the `view` command and tell this command that the input is in SAM format (`-S`) and to output BAM format (`-b`). We will start with the first sample (GSM461177) and repeat the process with the second (GSM461180):
> >
> > ```bash
> > $ samtools view -S -b GSM461177Aligned.out.sam > GSM461177Aligned.out.bam
> > ```
> {: .code-in}
> > <code-out-title></code-out-title>
> > ```
> > [samopen] SAM header is present: 1 sequences.
> > ```
> {: .code-out}
>
{: .hands_on}

> <hands-on-title>Sort BAM file by coordinates</hands-on-title>
>
> Next we sort the BAM file using the `sort` command from `samtools`. `-o` tells the command where to write the output.
> > <code-in-title>`sort` command</code-in-title>
> > ```bash
> > $ samtools sort -o GSM461177Aligned.out.sorted.bam GSM461177Aligned.out.bam
> > ```
> {: .code-in}
>
> Our files are pretty small, so we will not see this output. If you run the workflow with larger files, you will see something like this:
> > <code-out-title></code-out-title>
> >
> > ```
> > [bam_sort_core] merging from 2 files...
> > ```
> {: .code-out}
>
> SAM/BAM files can be sorted in multiple ways, e.g. by location of alignment on the chromosome, by read name, etc. It is important to be aware that different alignment tools will output differently sorted SAM/BAM, and different downstream tools require differently sorted alignment files as input.
>
> You can use samtools to learn more about this bam file as well.
>
> 
{: .hands_on}

# Counting reads per genes

Time to tie things up! To compare the expression of single genes between different conditions (e.g. with or without PS depletion), an essential first step is to quantify the number of reads per gene, or more specifically the number of reads mapping to the exons of each gene.

![Counting the number of reads per annotated gene](../../images/transcriptomics_images/gene_counting.png "Counting the number of reads per annotated gene")

For the final step of this tutorial we will use featureCounts to count the number of reads per annotated gene.

> <hands-on-title>FeatureCounts</hands-on-title>
>
> The main output of featureCounts is a table with the counts, i.e. the number of reads (or fragments in the case of paired-end reads) mapped to each gene (in rows, with their ID in the first column) in the provided annotation. FeatureCount generates also the feature length output datasets. Keep in mind that the "/import/7/" corresponds to *D. melanogaster's* gene annotation file we imported in RStudio way back. 
>
> > <code-in-title>featureCounts</code-in-title>
> > ```bash
> > $ featureCounts -a /import/7 -T 8 -o featurecounts.txt -p GSM461177Aligned.out.sorted.bam GSM461180Aligned.out.sorted.bam
> > ```
> {: .code-in}
>
> The **featurecounts.txt** produced contains all the information needed for further downstream analysis of the sequences we aligned (e.g. Differential Expression).
>
{: .hands_on}


topic: transcriptomics
tutorial: rna-seq-viz-with-heatmap2
title: Visualization of RNA-Seq results with heatmap2
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: []
content_hash: 4c964f1d18dcbc3e96f9a61d8322bf8d
last_modified: 2026-05-27T09:01:49.988144
zenodo_link: https://zenodo.org/record/2529926
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-viz-with-heatmap2/tutorial.html

content:
Heatmaps are commonly used to visualize RNA-Seq results. They are useful for visualizing the expression of genes across the samples. In this tutorial we show how the **heatmap2** tool in Galaxy can be used to generate heatmaps. The heatmap2 tool uses the heatmap.2 function from the R gplots package. Here we will demonstrate how to make a heatmap of the top differentially expressed (DE) genes in an RNA-Seq experiment, similar to what is shown for the fruitfly dataset in the [RNA-seq ref-based tutorial]({% link topics/transcriptomics/tutorials/ref-based/tutorial.md %}). We will also show how a heatmap for a custom set of genes an be created.

To generate a heatmap of RNA-seq results, we need a file of normalized counts. This file is provided for you here. The expression values have been normalized for differences in sequencing depth and composition bias between the samples. To generate this file yourself, see the [RNA-seq counts to genes]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %}) tutorial, and run limma-voom selecting *"Output Normalised Counts Table?"*: `Yes`. You could also use a file of normalized counts from other RNA-seq differential expression tools, such as edgeR or DESeq2. We also need some genes to plot in the heatmap.

The data for this tutorial comes from a Nature Cell Biology paper, [EGF-mediated induction of Mcl-1 at the switch to lactation is essential for alveolar cell survival](https://www.ncbi.nlm.nih.gov/pubmed/25730472)), Fu et al. 2015. This study examined the expression profiles of basal and luminal cells in the mammary gland of virgin, pregnant and lactating mice. Six groups are present, with one for each combination of cell type and mouse status.

![Tutorial Dataset](../../images/rna-seq-reads-to-counts/mouse_exp.png "Tutorial Dataset")


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing the inputs

We will use three files for this analysis:

 * **Normalized counts file** (genes in rows, samples in columns)
 * **Differentially expressed results file** (genes in rows, columns for P values and log fold change)
 * **Genes of interest** (custom list of genes to be visualized in heatmap)

## Import data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-seq exercise e.g. `RNA-seq heatmap`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the normalized counts table.
>
>    To import the file, there are two options:
>    - Option 1: From a shared data library if available (ask your instructor)
>    - Option 2: From [Zenodo](https://zenodo.org/record/2529926)
>
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
>    - You can paste the links below into the **Paste/Fetch** box:
>
>      ```
>      https://zenodo.org/record/2529926/files/limma-voom_normalised_counts
>      https://zenodo.org/record/2529926/files/limma-voom_luminalpregnant-luminallactate
>      https://zenodo.org/record/2529926/files/heatmap_genes
>      ```
>
>    - Select *"Genome"*: `mm10`
>
> 2. Rename the counts dataset as `normalized counts`, the limma-voom_luminalpregnant-luminallactate file as `DE results` and the list of genes as `heatmap genes` using the {% icon galaxy-pencil %} (pencil) icon.
> 3. Check that the datatype is `tabular`.
>    If the datatype is not `tabular`, please change the file type to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

Click on the {% icon galaxy-eye %} (eye) icon of the the `DE results` file and take a look. It should look like below with 8 columns (just the first few rows are shown).

![DE results](../../images/rna-seq-viz-with-heatmap2/DE_results.png "DE results")

# Create heatmap of top DE genes

First we'll demonstrate how to create a heatmap of the top differentially expressed genes in an RNA-seq dataset. To do this we need to extract the differentially expressed genes from the `DE results` file. This file contains the results from comparing gene expression in the luminal cells in the pregnant versus lactating mice. It includes genes that are not significantly differentially expressed. As in the [RNA-seq counts to genes tutorial]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %})), we'll call genes significantly differentially expressed in this dataset if they pass the thresholds of adjusted P value < 0.01 and fold change of > 1.5 (log2FC of 0.58).

## Extract all significant genes

> <hands-on-title>Extract the significant genes</hands-on-title>
>
> 1. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters to extract genes with adj P < 0.01:
>    - {% icon param-file %} *"Filter"*: `DE results`
>    - {% icon param-text %} *"With following condition"*: `c8<0.01`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
> 2. {% tool [Filter data on any column using simple expressions](Filter1) %} with the following parameters to extract genes with absolute fold change > 1.5 (log2fc of 0.58):
>    - {% icon param-file %} *"Filter"*: output of Filter {% icon tool %}
>    - {% icon param-text %} *"With following condition"*: `abs(c4)>0.58`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
> 3. Rename file to `Significant genes`
{: .hands_on}

## Extract the top significant genes

This gives us a file with all the significant genes, the genes that pass our thresholds for statistical and biological significance. As we can see there are many genes (~1,610), too many to plot in one heatmap, so we'll select the top 20 by P value.

> <hands-on-title>Extract the top significant genes by P value</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} data in ascending or descending order, with the following settings to sort by P value:
>    - {% icon param-file %} *"Sort Query"*: `Significant genes` (output of 2nd Filter {% icon tool %})
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>    - {% icon param-select %} *"on Column"*: `Column: 7`
>    - {% icon param-select %} *"in"*: `Ascending order`
>    - {% icon param-select %} *"Flavor"*: `General numeric sort ( scientific notation -g)`
> 2. {% tool [Select first](Show beginning1) %}  lines from a dataset with the following parameters to select top genes:
>    - {% icon param-text %} *"Select first"*: `21` (20 genes plus header row)
>    - {% icon param-file %} *"from"*: output of Sort {% icon tool %}
> 3. Rename file as `top 20 by Pvalue`
{: .hands_on}

## Extract the normalized counts for top genes

Now we have a file that contains only the top 20 genes from the DE results. Next we need to get the normalized counts for these genes, from the file containing the normalized counts for all genes in the experiment, and then extract just the columns we need for the heatmap (the normalized counts and gene labels).

First click on the {% icon galaxy-eye %} (eye) icon and take a look at the `normalized counts` file that we imported. It should look like below (just the first few rows and columns are shown). Note that the normalized count values are log2. We will join our `top 20 by Pvalue` file to the `normalized counts` file, matching on the ENTREZID columns.

![Normalized counts](../../images/rna-seq-viz-with-heatmap2/normalized_counts.png "Normalized counts")

> <hands-on-title>Extract the normalized counts for the top genes</hands-on-title>
> 1. {% tool [Join two Datasets side by side on a specified field](join1) %} with the following parameters to join on the ENTREZID column:
>    - {% icon param-file %} *"Join"*: the `top 20 by Pvalue` file
>    - {% icon param-select %} *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `normalized counts` file
>    - {% icon param-select %} *"and column"*: `Column: 1`
>    - {% icon param-select %} *"Keep the header lines"*: `Yes`
>
>    The generated file has more columns than we need for the heatmap. In addition to the columns with normalized counts (in log2), there is the $$log_{2} FC$$ and other information. We need to remove the extra columns.
>
> 2. {% tool [Cut](Cut1) %} columns from a table to extract the columns with the gene symbols and normalized counts
>    - {% icon param-text %} *"Cut columns"*: `c2,c12-c23`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the joined dataset (output of **Join two Datasets** {% icon tool %})
{: .hands_on}

The file should look like below with the 20 genes in the rows and the 12 samples in the columns (just the first few columns are shown).

![Heatmap input](../../images/rna-seq-viz-with-heatmap2/heatmap_input.png "Heatmap input")

Now that we have our file with just the normalized counts for the genes we want, we can create a heatmap.

## Create heatmap of top genes

> <hands-on-title>Plot the heatmap of top genes</hands-on-title>
>
> 1. {% tool [heatmap2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1) %} to plot the heatmap:
>    - {% icon param-file %} *"Input should have column headers"*: output of **Cut** {% icon tool %}
>    - {% icon param-select %} *"Data transformation"*: `Plot the data as it is`
>    - {% icon param-select %} *"Compute z-scores prior to clustering"*: `Compute on rows` (scale genes)
>    - {% icon param-select %} *"Enable data clustering"*: `No`
>    - {% icon param-select %} *"Labeling columns and rows"*: `Label my columns and rows`
>    - {% icon param-select %} *"Type of colormap to use"*: `Gradient with 3 colors`
{: .hands_on}

You should see a heatmap like below. Note that here we are plotting the top genes differentially expressed in the luminal cells from the pregnant mice (`MCL1.LC` and `MCL1.LD`) versus the luminal cells from the lactating mice (`MCL1.LE` and `MCL1.LF`). This heatmap enables us to see the expression of these genes in all the samples from the different groups in the experiment (basal virgin, basal pregnant, basal lactating, luminal virgin, luminal pregnant, luminal lactating).

![Heatmap of top DE genes](../../images/rna-seq-viz-with-heatmap2/heatmap2_top20.png "Heatmap of top DE genes"){:width="150%"}


> <question-title></question-title>
> 1. Why do we not use clustering here?
> 2. Why do we compute z-scores on rows (genes)? Try rerunning heatmap2 changing the *"Compute z-scores prior to clustering"* parameter to `Do not compute z-scores`.
> 3. The genes are ordered by P value. Can you make the heatmap with the genes ordered by fold change? Hint: Sort by the logFC column in ascending order to have the genes downregulated in the luminal pregnant vs lactating (negative fold change) at the top and the upregulated genes (positive fold change) at the bottom.
> 4. How could we make a heatmap of the top 10 most upregulated and top 10 most downregulated significant genes?
>
>    > <solution-title></solution-title>
>    > 1. We don't use clustering here as we want to keep the genes in the order we input (ordered by P value).
>    > 2. We scale the genes as otherwise large expression values from highly expressed genes would dominate the plot, see below.
>    >     ![Heatmap with no scaling](../../images/rna-seq-viz-with-heatmap2/heatmap2_top20_noscale.png "Heatmap without scaling genes")
>    >
>    > 3. To make the heatmap with these genes ordered by logFC, we could {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} the output of the **Join** {% icon tool %} step on the logFC column (Column 4), then {% tool [Cut](Cut1) %} the columns as before and remake the heatmap. It should look like below.
>    >     ![Heatmap sorted by logFC](../../images/rna-seq-viz-with-heatmap2/heatmap2_top20_sorted_by_lfc.png "Heatmap sorted by logFC")
>    > 4. To make a heatmap of the most upregulated and downregulated significant genes, we could {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} the `Significant genes` on the logFC column (instead of P value above), then use the {% tool [Select first](Show beginning1) %} and {% tool [Select last](Show tail1) %} tools to select the genes with the largest and smallest fold changes (10 genes for each) and {% tool [Concatenate multiple datasets or collections](cat1) %} to combine the outputs of the first and last Selects into one file. We would then repeat the {% tool [Join two Datasets side by side on a specified field](join1) %} and {% tool [Cut](Cut1) %} steps as before to get the normalized counts and make the heatmap as below (Note that one gene name in the middle is missing as it is a gene that has NA for gene symbol)
>    >     ![Heatmap for top logFC](../../images/rna-seq-viz-with-heatmap2/heatmap2_top_up_and_down.png "Heatmap of top 10 up and down")
>    {: .solution}
{: .question}


# Create heatmap of custom genes

You can also create a heatmap for a custom set of genes. To demonstrate this, we will create a heatmap for the 31 genes in Figure 6b from the original paper using this dataset (see below). These 31 genes include the authors' main gene of interest in the paper, Mcl1, and a set of cytokines/growth factors identified as differentially expressed. We will recreate this heatmap here. To see how to visualize these genes in a volcano plot see [this tutorial]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %}).

![Fu heatmap](../../images/rna-seq-viz-with-heatmap2/fu_heatmap.png "Fu et al, Nat Cell Biol 2015")

These 31 genes are in the file we imported called `heatmap genes`, shown below.

![Heatmap genes](../../images/rna-seq-viz-with-heatmap2/heatmap_genes.png "Heatmap genes")

As in the previous example, we need to extract the normalized counts for just these 31 genes. To do that we will join the `heatmap genes` file with the `normalized counts` file, on the Gene Symbol columns this time (instead of ENTREZID), and then extract the columns we need.

## Extract normalized counts for custom genes

> <hands-on-title>Extract the normalized counts for the genes of interest</hands-on-title>
>
> 1. {% tool [Join two Datasets side by side on a specified field](join1) %} with the following parameters:
>    - {% icon param-file %} *"Join"*: the `heatmap genes` file
>    - {% icon param-select %} *"using column"*: `Column: 1`
>    - {% icon param-file %} *"with"*: `normalized counts` file
>    - {% icon param-select %} *"and column"*: `Column: 2`
>    - {% icon param-select %} *"Keep the header lines"*: `Yes`
> 2. {% tool [Cut](Cut1) %} columns from a table to extract the columns with the gene ids and normalized counts:
>    - {% icon param-text %} *"Cut columns"*: `c1,c5-c16`
>    - {% icon param-select %} *"Delimited by"*: `Tab`
>    - {% icon param-file %} *"From"*: the joined dataset (output of **Join two Datasets** {% icon tool %})
>
>    The genes are in rows and the samples in columns, we could leave the genes in rows but we will transpose to have genes in columns and samples in rows as in the Figure in the paper.
>
> 3. {% tool [Transpose](toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0) %} to have samples in rows and genes in columns:
>    - *"Input tabular dataset"*:
>        - {% icon param-file %} *"From"*: the `Cut` dataset (output of **Cut** {% icon tool %})
{: .hands_on}

We now have a table with the 31 genes in columns and the normalized counts for the 12 samples in rows, similar to below (just the first few columns are shown).

![Transposed input](../../images/rna-seq-viz-with-heatmap2/transposed_input.png "Transposed input")

## Create heatmap of custom genes

> <hands-on-title>Plot the heatmap of custom genes</hands-on-title>
>
> 1. {% tool [heatmap2](toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input should have column headers"*: the generated table (output of **Transpose** {% icon tool %})
>    - {% icon param-select %} *"Data transformation"*: `Plot the data as it is`
>    - {% icon param-select %} *"Compute z-scores prior to clustering"*: `Compute on columns` (scale genes)
>    - {% icon param-select %} *"Enable data clustering"*: `No`
>    - {% icon param-select %} *"Labeling columns and rows"*: `Label my columns and rows`
>    - {% icon param-select %} *"Type of colormap to use"*: `Gradient with 3 colors`
{: .hands_on}

You should see a heatmap like below.

![Fu heatmap regenerated](../../images/rna-seq-viz-with-heatmap2/fu_heatmap_regenerated.png "Fu heatmap regenerated"){: width="30%"}

> <question-title></question-title>
>
> How does the heatmap compare to the one from the Fu paper Fig 6 (above)?
>
>    > <solution-title></solution-title>
>    >
>    > The heatmap looks similar to the heatmap in the paper.
>    >
>    {: .solution}
{: .question}


# Conclusion


In this tutorial we have seen how heatmaps can be used to visualize RNA-seq results using the **heatmap2** tool in Galaxy. We use the same dataset from the tutorials, [RNA-seq reads to counts]({% link topics/transcriptomics/tutorials/rna-seq-reads-to-counts/tutorial.md %}), [RNA-seq counts to genes]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %}), [RNA-seq genes to pathways]({% link topics/transcriptomics/tutorials/rna-seq-genes-to-pathways/tutorial.md %}) and [Visualization of RNA-Seq results with Volcano Plot]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.md %}).

tools_json:
["Filter data on any column using simple expressions", "Filter1", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2", "Select first", "Show beginning1", "Join two Datasets side by side on a specified field", "join1", "Cut", "Cut1", "heatmap2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_heatmap2/ggplot2_heatmap2/3.2.0+galaxy1", "Select last", "Show tail1", "Concatenate multiple datasets or collections", "cat1", "Transpose", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_transpose/datamash_transpose/1.9+galaxy0"]


topic: transcriptomics
tutorial: rna-interactome
title: RNA-RNA interactome data analysis
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 43fa92ce5345e30919303449fb2aa5f1
last_modified: 2026-05-15T22:28:17.635816
zenodo_link: https://zenodo.org/record/3709188
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-interactome/tutorial.html

content:
With the advances in the next-generation sequencing technologies, genome-wide RNA-RNA interaction predictions are now
possible. The most recent line of development has been to ligate the microRNA to the site-specific interaction region of
 the target, selecting these interactions via cross-linking to one of the Argonaute proteins required for microRNA-based
  regulation, and to sequence the resulting chimeric RNA molecule, for example, the CLASH and CLEAR-CLIP protocols.
Going beyond microRNAs, these protocols can be applied to RNA interactions that involve a regulatory protein
other than Argonaute. To generalize even further, researchers have applied the same idea to the detection of all
transcriptome-wide RNA-RNA interactions, which include both inter- and intramolecular base pairing without the necessity
 of choosing a specific regulatory protein for cross-linking. These protocols include LIGR-Seq that maps the human
 RNA-RNA interactome and PARIS that focused on long-range structures in human and mouse.

The reads from these experiments are chimeric with each arm generated from one of the interaction
partners. Due to short lengths, often these sequenced arms ambiguously map to multiple locations and  inferring the
origin of these can be quite complicated. Theoretically, alignment tools like `HISAT2` and `STAR` can be used to align
chimeric reads, but they are not efficient at this task. The other alignment tools like `BWA-MEM` or `Bowtie2` can be
used in local alignment settings to map these chimeric reads. In this case user needs to adjust the alignment parameters
to match the read lengths and there needs to be a lot of post-processing to be done to choose the best hits. Recently,
there is also an alignment tool called `CLAN` published to specifically map the chimeric reads from CLASH experiments.

In this tutorial, we will learn the analysis of a CLEAR-CLIP data set using a tool suite called `ChiRA`. The data used
is from mouse cortex sample
([GSM1881541](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM1881541)) prepared using CLEAR-CLIP protocol. It is
a complete analysis framework that can be used starting from raw sequencing reads to analysis and visualization of
results. `ChiRA` uses `BWA-MEM` or `CLAN` to map the reads. Subsequently, it also merges the overlappig alignments and
 chooses the best alignments per read by quantifying the all the loci that reads map to. In the end, it scores each
  alignment and outputs only the best alignments per read. The final part of this tutorial gives an insight into how to
  filter, export and visualize your results using the visualization framework `ChiRAViz`.


![ChiRA workflow](../../images/rna-interactome/chira.png "ChiRA workflow. First the reads deduplicated and mapped to transcriptome. Then the mapped loci are merged based on overlapping. The merged loci are quantified and the interactions are scored and reported.")

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from [Zenodo](https://zenodo.org/record/3709188) or from the shared data library
>
>    ```
>    https://zenodo.org/record/3709188/files/miRNA_mature.fa.gz
>    https://zenodo.org/record/3709188/files/Mus_musculus.GRCm38.dna.fa.gz
>    https://zenodo.org/record/3709188/files/SRR2413302.fastq.gz
>    https://zenodo.org/record/3709188/files/transcriptome.fa.gz
>    https://zenodo.org/record/3709188/files/whole_transcriptome.gff.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 3. Rename the datasets
> 4. Check that the datatype
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 5. Add to each database a tag corresponding to ...
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
{: .hands_on}

# Preprocessing

Before starting with the analysis of data it is always good to check the sequenced reads for low quality bases and
adapters.

## Quality control

> <hands-on-title>Quality check</hands-on-title>
> First use `FastQC` to assess the read quality
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Short read data from your current history"*: `SRR2413302.fastq.gz` (Input dataset)
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Why do you think `FastQC` failed to find any adapters?
>
> > <solution-title></solution-title>
> >
> > 1. Because `FastQC` it uses a set of standard adapters to screen for adapters. The "special" adapters used in this
> >    library preparation are not present in the `FastQC` standard adapters list.
> >
> {: .solution}
>
{: .question}

## Adapter trimming
Due to the inefficiency of the current RNA interactome protocols, not all reads are not made up of RNA hybrids. In some
cases, reads contain single RNA fragments with adapters or nothing but only adapters. Hence adapter removal is a very important step
in this analysis. In this step, we use `cutadapt` to trim the adapters. As the adapters used in this library are not
standard Illumina adapters, we need to provide them manually.

> <hands-on-title>Adapter trimming</hands-on-title>
> We use `cutadapt` to trim the adapter content
>
> 1. **cutadapt** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"FASTQ/A file"*: `SRR2413302.fastq.gz` (Input dataset)
>    - In *"Read 1 Options"*
>       - *"3' (End) Adapters"* -> *"Insert 3' (End) Adapters"*
>           - *"Source"*: `Enter Custom sequence`
>           - *"Enter custom 3' adapter sequence"*: `GTGTCAGTCACTTCCAGCGG`
>       - *"5' (Front) Adapters"* -> *"Insert 5' (Front) Adapters"*
>           - *"Source"*: `Enter Custom sequence`
>           - *"Enter custom 5' adapter sequence"*: `GCATAGGGAGGACGATGCGG`
>    - In *"Filter Options"*
>        - *"Minimum length"*: `16`
>
{: .hands_on}


> <hands-on-title>Post adapter trimming quality check</hands-on-title>
> It is interesting to see whether our manually entered adapters were trimmed
>
> 1. **FastQC** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Short read data from your current history"*: `Read 1 Output` (output of **cutadapt** {% icon tool %})
>    - Observe the **Per base sequence content**
> ![FastQC per base sequence content](../../images/rna-interactome/chira_fastqc_seq_content.png)
{: .hands_on}

> <question-title></question-title>
>
> Would you be concerned about the abnormal "Per base sequence content towards the end"?
>
> > <solution-title></solution-title>
> >
> > Normally yes, but in this case not. Always look at this plot in combination with "Sequence Length Distribution"
> > plot. It looks like there is huge difference in base composition after 55th base. But the number of
> > sequences that constitute this is very important. From the sequence length distribution, most of the sequences are
> > between 53 and 57 bases long. We see the abnormality in the per base sequence content because it is from
> > very few sequences.
> > ![FastQC sequence length distribution](../../images/rna-interactome/chira_fastqc_seq_length.png)
> >
> {: .solution}
>
{: .question}

# Analysis of interactome data using `ChiRA` tool suite

The analysis includes several steps that deal with deduplication mapping, quantification and extraction of interacting
partners.

## Remove duplicate sequences

First, we eliminate the duplicate sequences from the library to reduce the computational effort. This will also have an
impact on the quantification of the loci because often these identical sequences might be PCR duplicates. There is also
a 5' degenerate linker of length 5 nucleotides present in the reads. Hence we have to strip that too.

> <hands-on-title></hands-on-title>
>
> 1. **ChiRA collapse** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input FASTQ file"*: `Read 1 Output` (output of **cutadapt** {% icon tool %})
>    - *"Length of the UMI if present at the 5' end of your reads"*: `5`
>
>    > <tip-title>Dealing with UMIs</tip-title>
>    >
>    > * If you have UMIs (at the 5' end) in the sequenced reads, please set *"Length of the UMI if present at the 5' end of your reads"*.
>    > * The UMI will be trimmed and put in the unique sequence id.
>    {: .tip}
>
{: .hands_on}


## Map reads to the reference transcriptome

> <hands-on-title>Map chimeric reads from fasta file</hands-on-title>
> Here we use `BWA-MEM` aligner in local alignment mode to locate the chimeric arms on the
> transcriptome. Your reference can be single or split in two. Two references are
> ideal for example if you have CLASH experimental data where you have separate
> miRNA and target references.
>
> 1. **ChiRA map** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input FASTA file"*: `fasta file` (output of **ChiRA collapse** {% icon tool %})
>    - *"Single or split reference?"*: `Split reference`
>        - {% icon param-file %} *"Reference FASTA file"*: `miRNA_mature.fa.gz` (Input dataset)
>        - {% icon param-file %} *"Second reference FASTA file"*: `transcriptome.fa.gz` (Input dataset)
>    - *"aligner"*: `BWA-MEM`
>
{: .hands_on}


## Merge overlapping alignment information

In this step, we merge the overlapping aligned positions to define the read concentrated loci. If an annotation GTF file
produced, the transcriptomic alignment positions are first converted to their corresponding genomic positions. The
merging is also done on reads defining which parts of the reads are mapping that indicates potential interacting segments
of read.

> <hands-on-title></hands-on-title>
>
> 1. **ChiRA merge** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Input BED file of alignments"*: `ChiRA aligned BED` (output of **ChiRA map** {% icon tool %})
>    - *"Do you have an annotation in GTF format?"*: `Yes`
>        - {% icon param-file %} *"Annotations in GTF format"*: `whole_transcriptome.gff.gz` (Input dataset)
>    - *"Did you use single or split reference for alignment?"*: `Split reference`
>        - {% icon param-file %} *"Reference FASTA file"*: `miRNA_mature.fa.gz` (Input dataset)
>        - {% icon param-file %} *"Second reference FASTA file"*: `transcriptome.fa.gz` (Input dataset)
>
>    > <tip-title>Parameters for samples with high coverage.</tip-title>
>    >
>    > * In samples with a very high coverage, the likelihood of having overlapping alignments increases. Hence the
>    >   default `Overlap based` merging may results in very long loci merged by some random alignments.
>    > * Therefore, use the `blockbuster` merging mode and adjust the paramertes accordingly.
>    >   - From *"Select the mode of merging"*: `Gaussian based (blockbuster)`
>    > * Working with only the chimeric reads further reduces the computation time fr subsequent steps.
>    >   - *"chimeric_only"*: `Yes`
>    {: .tip}
>
{: .hands_on}

## Quantify aligned loci to score the alignments

Now we have the loci where the potential interacting read segments are mapped to. Due to the small length of these arms, there is a
very high chance of multi mapping. Another reason for this is the lenient mapping parameters that are used to increase
the mapping sensitivity. Quantification needs 2 files containing read segements and loci where they are mapping to. From
this information, `ChiRA quanitify` tries to infer the correct origin of reads and calculates the expression of the loci
using a simple expectation-maximization algorithm.

> <hands-on-title>Task description</hands-on-title>
>
> 1. **ChiRA qauntify** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"BED file of aligned segments"*: `ChiRA aligned read segments` (output of **ChiRA merge** {% icon tool %})
>    - {% icon param-file %} *"Tabular file of merged alignments"*: `ChiRA merged alignments` (output of **ChiRA merge** {% icon tool %})
>
{: .hands_on}

## Extract the best scoring chimeras

After having the information about the loci expression, the final step extracts only the best scoring interacting
partners for each read. All the combinations of the transcripts that are overlapping with the interacting loci are
reported. If there is more than one locus with the equal best score then all the best hits are reported. If you have the
genomic fasta file the tool can hybridize the interacting loci sequences using `IntaRNA`.

> <hands-on-title>Task description</hands-on-title>
>
> 1. **ChiRA extract** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"File containing CRLs information"*: `ChiRA quantified loci` (output of **ChiRA qauntify** {% icon tool %})
>    - *"Have genomic information?"*: `Yes`
>        - {% icon param-file %} *"Annotations in GTF format"*: `whole_transcriptome.gff.gz` (Input dataset)
>        - *"Choose the source for the FASTA file"*: `History`
>            - {% icon param-file %} *"FASTA file"*: `Mus_musculus.GRCm38.dna.fa.gz` (Input dataset)
>    - *"Did you use single or split reference for alignment?"*: `Split reference`
>        - {% icon param-file %} *"Reference FASTA file"*: `miRNA_mature.fa.gz` (Input dataset)
>        - {% icon param-file %} *"Second reference FASTA file"*: `transcriptome.fa.gz` (Input dataset)
>    - *"Hybridize chimeric loci?"*: `Yes`
>    - *"Summarize interactions at loci level?"*: `Yes`
>
{: .hands_on}

# Visualization

The output tabular file generated in the above step can be huge with up to some millions of rows depending on the
library size and more than 30 columns. Extracting useful data from this can be very tedious. For example, extracting and
visualizing the distribution of target biotypes of your favorite miRNA can be very tricky and might need more than a
hand full of galaxy tools to achieve. For this reason, there exists a visualization and filtering tool for this data along
with `ChiRA` known as `ChiRAViz`. It is a galaxy visualization framework to work with the output of `ChiRA`. But it does
not directly work with the tabular output we have. Rather it needs a "sqlite" database. For this reason, we first build a
sqlite database from the `ChiRA` output.

> <hands-on-title>Data preperation</hands-on-title>
>
> 1. **Query Tabular** {% icon tool %} with the following parameters:
>    - In *"Database Table"*:
>        - {% icon param-repeat %} *"Insert Database Table"*
>            - {% icon param-file %} *"Tabular Dataset for Table"*: `ChiRA chimeric reads` (output of **ChiRA extract** {% icon tool %})
>    - In *"Table Options"*
>        - *"Use first line as column names"*: `Yes`
>    - *"Save the sqlite database in your history"*: `Yes`
>
> 2. Change the datatype to `chira.sqlite`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="chira.sqlite" %}
>
{: .hands_on}

> <hands-on-title>Visualize interactions</hands-on-title>
>
> 1. Please click on {% icon galaxy-barchart %} *"Visualize this data"*. Then click on the `ChiRAViz` visualization.
>    This loads the data into the visualization framework and shows some basic plots from the data.
>    - The visualization split into two to show the left and the right arms information.
>    - On home page pie charts of left and right chimeric arms, types of interactions and top 50 expressed RNAs are shown.
>    ![ChiRAViz home page](../../images/rna-interactome/chiraviz_view1.png)
> 2. Now choose the bio types of interactions that you want to work with. Here we first get all available interactions,
>    and then filter the interactions we are interested in next page. To get all interactions, choose `all` in both dropdowns
>    on the top and then click on **"Get interactions"**.
>
>    ![ChiRAViz selector](../../images/rna-interactome/chiraviz_choose.png)
>
{: .hands_on}

> <hands-on-title> Filter and summarize interactions and export the results</hands-on-title>
> `ChiRAViz` provides filters to search for keywords like gene symbols, sort interactions by score, filter by score or
> hybridization energy. Then the filtered interactions can be summarized or exported to a file. In this step, we filter
> the interactions that `mmu-miR-190a` involved in and consider those which have an `IntaRNA` predicted hybrid.
>    - To search, type `mir-190a` in the search field and click on search icon
>      or hit enter. Search is case insensitive and can search for sub-phrases
>      too. This results in 27 records.
>    - We now further filter the records that contain `IntaRNA` hybrid. If
>      there is no hybrid predicted by `IntaRNA`,
>      then the hybrid filed contains an `NA` value.
>       - From **"--filter--"** dropdown choose `Hybrid`
>       - From **"--operator--"** choose `<>`
>       - Enter `NA` in the value field and hit the enter key.
>    This filters out 10 more records and results in 17 records.
>    - At this point, you can select individual interactions by clicking the individual checkboxes or by clicking **"Check all"**. Both the possibilities are highlighted in red color in the following figure.
>    - Click on **Summary** to view the summary plots for the selected interactions.
>    - Clicking on **Export** to export the selected interactions to a file.
> ![ChiRAViz filter page](../../images/rna-interactome/chiraviz_view2.png)
{: .hands_on}

> <question-title></question-title>
>
> 1. Which strand of `mmu-miR-190a` is the most expressed?
>
> > <solution-title></solution-title>
> >
> > Both strands of `mmu-miR-190a` participated in the interactions, but `mmu-miR-190a-5p` is the most abundant strand between the two.
> >
> {: .solution}
>
{: .question}


> <hands-on-title> Viewing individual interaction information</hands-on-title>
> - From the list of interactions in the left panel expand the interaction `mmu-miR-190a-5p:Myo5a` by clicking on "+" (highlighted in red). There are 4 sub-records
>   corresponds to 4 different transcripts of the target gene `Myo5a`.
> - Click on one of the records to view following information.
>   - **"Chimera"** panel in the middle depicts the mapping positions on the read with read length.
>   - **"Interacting partners"** panel shows the information on which transcripts the left and right arm are mapping to with their alignment positions on the transcripts.
>   - **"Alignment Information"** panel shows the alignment if present with a possibility to download the alignment.
> ![ChiRAViz single interaction](../../images/rna-interactome/chiraviz_view3.png)
{: .hands_on}

# Conclusion


Though chimeric reads look normal when inspected in a FASTQ file, the origin of each read is from two different RNA fragments. Limitations of the current sequencing protocols limit the length of each sequenced interacting RNA fragment. These smaller RNA fragments are often harder to map considering that the boundaries of each RNA fragment in the read are unknown. In this tutorial, we have seen how to map these reads and infer the true origins of them by quantifying the mapped loci. The visualization framework gives flexibility in filtering and searching output files, visualize the summaries of filtered data as well as exporting them.


topic: transcriptomics
tutorial: network-analysis-with-heinz
title: Network analysis with Heinz
difficulty: intermediate
time_estimation: 1h30m
tools_json: []
requirements_json: []
tags_json: ["metatranscriptomics", "network analysis"]
content_hash: 98bc0ae6d8ec3a1fbbc13013568a2ec0
last_modified: 2026-05-15T22:28:17.630219
zenodo_link: https://doi.org/10.5281/zenodo.1344105
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/network-analysis-with-heinz/tutorial.html

content:
# Overview


The human microbiome plays a key role in health and disease. Thanks to comparative metatranscriptomics,
the cellular functions that are deregulated by the microbiome in disease can now be computationally
explored. Unlike gene-centric approaches, pathway-based methods provide a systemic view of such
functions; however, they typically consider each pathway in isolation and in its entirety.
They can therefore overlook the key differences that (i) span multiple pathways, (ii) contain
bidirectionally deregulated components, (iii) are confined to a pathway region. To capture these
properties, computational methods that reach beyond the scope of predefined pathways are needed.

In this tutorial, we will perform a network analysis using Heinz ([GitHub](https://github.com/ls-cwi/heinz),
[publication](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2718639/)) in Galaxy. The data comes from
the study [May et al.](https://academic.oup.com/bioinformatics/article/32/11/1678/2240171),
and we will reproduce some of the computational steps from this study with simplified data and
parameters to speed up the analysis for the purposes of this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


{% snippet faqs/galaxy/analysis_results_may_vary.md %}


In this tutorial, we will run the Heinz workflow step by step to understand how each step works. To get an overview of what this workflow does
and what kind of calculation is involved, view the flowchart below.

![Heinz flowchart](../../images/transcriptomics_images/flowchart-heinz.png)

# Obtaining and preparing data  [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1344105.svg)](https://doi.org/10.5281/zenodo.1344105)


The study [May et al.](https://academic.oup.com/bioinformatics/article/32/11/1678/2240171) includes the computation
steps starting from the raw RNAseq datasets. The operations that processed raw data into the interpreted data are
beyond the scope of this tutorial. To learn that, please refer to the relevant topics in the Galaxy training
material. In this tutorial, we start with the interpreted data, which are KO
([KEGG Orthology](https://www.genome.jp/kegg/ko.html)) count data. All the data needed for this tutorial are
available from Zenodo.

> <details-title>Background: What is KO count?</details-title>
>
> KOs are organism-independent identifiers that group together proteins of similar biochemical functions.
> It is a term specific to [KEGG](https://www.genome.jp/kegg/) database. It is a group concept, similar to the concept "pathway", which
> includes a bunch of molecules and their interactions. Take a random line in our dataset, "K01369 7" as an example, the first column represents the KO ID,
> and the second column --- the numeric value is the KO counts. If we understand it by the face value, it is 7 protein molecules that are counted from our dataset as this KO "K01369"; but we usually mingle proteins with genes because we often assume that each protein corresponds to a gene, so it can be understood as 7 genes. Now we are working on the microbial RNAseq data, and the assumption is also used subconsciously.
>
{: .details}

## Understanding our input data

According to the study [May et al.](https://academic.oup.com/bioinformatics/article/32/11/1678/2240171),
dental caries (DC) dataset in this tutorial came from an experiment that comprised of supragingival plaque
samples collected from all dental surfaces of 36 individuals who had either a caries-positive (disease) or
a caries-negative (health) oral profile. Each of the 36 samples was sequenced, pre-processed and transformed
into KO counts. We will use these count data as the starting point to perform the network analysis.

> <comment-title>Dataset details</comment-title>
> The count data of the 36 samples are separated into 36 files, organized into two groups:
> 'CP' (caries-positive) and 'CN' (caries-negative).
{: .comment}


## Importing the data into Galaxy

After knowing what our input data are like, let's get them into Galaxy history:

> <hands-on-title>Obtaining our data</hands-on-title>
>
> 1. Make sure we have an empty Galaxy history. Give it a sensible name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. **Upload Disease Dataset**
>    - Open the file upload menu
>    - Click on  **Collection** tab
>    - Click on the **Paste/Fetch data** button
>    - Copy the Zenodo links for the Disease Datasets
>      > <details-title>View list of Zenodo URLs for Dental Caries Dataset (Disease, CP)</details-title>
>      > ```
>      > https://zenodo.org/record/1344105/files/2241_CP_DZ_PairTo_2242.txt
>      > https://zenodo.org/record/1344105/files/2126_CP_MZ_PairTo_2125.txt
>      > https://zenodo.org/record/1344105/files/2991_CP_DZ_PairTo_2992.txt
>      > https://zenodo.org/record/1344105/files/2931_CP_DZ_PairTo_2930.txt
>      > https://zenodo.org/record/1344105/files/2284_CP_DZ_PairTo_2283.txt
>      > https://zenodo.org/record/1344105/files/2125_CP_MZ_PairTo_2126.txt
>      > https://zenodo.org/record/1344105/files/4131_CP_DZ_PairTo_4132.txt
>      > https://zenodo.org/record/1344105/files/2954_CP_DZ_PairTo_2955.txt
>      > https://zenodo.org/record/1344105/files/2170_CP_MZ_PairTo_2169.txt
>      > https://zenodo.org/record/1344105/files/2955_CP_DZ_PairTo_2954.txt
>      > https://zenodo.org/record/1344105/files/2011_CP_DZ_PairTo_2012.txt
>      > https://zenodo.org/record/1344105/files/2012_CP_DZ_PairTo_2011.txt
>      > https://zenodo.org/record/1344105/files/2269_CP_DZ_PairTo_2270.txt
>      > https://zenodo.org/record/1344105/files/3215_CP_MZ_PairTo_3214.txt
>      > https://zenodo.org/record/1344105/files/2354_CP_DZ_PairTo_2355.txt
>      > https://zenodo.org/record/1344105/files/3306_CP_DZ_PairTo_3307.txt
>      > https://zenodo.org/record/1344105/files/2061_CP_DZ_PairTo_2062.txt
>      > https://zenodo.org/record/1344105/files/2355_CP_DZ_PairTo_2354.txt
>      > https://zenodo.org/record/1344105/files/2242_CP_DZ_PairTo_2241.txt
>      > ```
>      {: .details}
>    - Click on **Start** to Upload the files
>    - Click **Build** once upload has completed
>    - Enter **Name**: `CP`
>    - Click **Create list** to make the collection
>
> 3. **Upload control (healthy) datasets**
>    - Repeat the previous steps with the samples from healthy individuals:
>      > <details-title>View list of Zenodo URLs for Dental Caries Dataset (Healthy, CN)</details-title>
>      > ```
>      > https://zenodo.org/record/1344105/files/2310_CN_DZ_PairTo_2309.txt
>      > https://zenodo.org/record/1344105/files/2062_CN_DZ_PairTo_2061.txt
>      > https://zenodo.org/record/1344105/files/2191_CN_MZ_PairTo_2192.txt
>      > https://zenodo.org/record/1344105/files/2052_CN_MZ_PairTo_2051.txt
>      > https://zenodo.org/record/1344105/files/2051_CN_MZ_PairTo_2052.txt
>      > https://zenodo.org/record/1344105/files/2192_CN_MZ_PairTo_2191.txt
>      > https://zenodo.org/record/1344105/files/2234_CN_DZ_PairTo_2233.txt
>      > https://zenodo.org/record/1344105/files/2233_CN_DZ_PairTo_2234.txt
>      > https://zenodo.org/record/1344105/files/2270_CN_DZ_PairTo_2269.txt
>      > https://zenodo.org/record/1344105/files/2225_CN_MZ_PairTo_2226.txt
>      > https://zenodo.org/record/1344105/files/4132_CN_DZ_PairTo_4131.txt
>      > https://zenodo.org/record/1344105/files/2309_CN_DZ_PairTo_2310.txt
>      > https://zenodo.org/record/1344105/files/2992_CN_DZ_PairTo_2991.txt
>      > https://zenodo.org/record/1344105/files/3214_CN_MZ_PairTo_3215.txt
>      > https://zenodo.org/record/1344105/files/2169_CN_MZ_PairTo_2170.txt
>      > https://zenodo.org/record/1344105/files/2930_CN_DZ_PairTo_2931.txt
>      > https://zenodo.org/record/1344105/files/3307_CN_DZ_PairTo_3306.txt
>      > ```
>      {: .details}
>    - Name this collection: `CN`
>
>    > <question-title></question-title>
>    >
>    > 1. How many samples do you have in our disease collection (CP)? How many healthy samples (CN)?
>    > 2. How many columns in each file? What are these columns?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. You should have 19 samples in the disease collection (CP), and 17 in the negative collection (CN).
>    > > 2. There are two columns, one is the KO IDs, the other is the count.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

> <tip-title>Creating a collection from files already in your history</tip-title>
>
> *Dataset collections* enables us to easily run tools on multiple datasets at once, you probably
> have done that in the file upload menu using the `Collection` tab. If not, you can still create
> *dataset collections* manually.
>
> > <hands-on-title>Manually organizing our data into a collection</hands-on-title>
> >
> > If you have all the datasets in the history, but they are not organized into a collection yet,
> > you can follow these steps to create a collection:
> >
> > 1. Click on the **checkmark icon** at top of your history.
> >   ![Checkmark icon in history menu](../../../../shared/images/history_menu_buttons2.png)
> >
> > 2. Select all the files whose name contains `CP`, then click on **n of N selected** and select
> >   **Build Dataset List** from the dropdown menu.
> >
> > 3. In the next dialog window, you need to give a name, here we just set it to `CP`, then click **Create list**.
> >
> > 4. Redo the Step 2, 3 for CN, set the name of the data list as 'CN'.
> {: .hands_on}
{: .tip}



# Differential Expression Analysis (DEA) by DESeq2

## What is differential expression analysis?

> <comment-title>A definition</comment-title>
>
> The definition of differential expression analysis given by
> [EBI](https://www.ebi.ac.uk/training/online/course/functional-genomics-ii-common-technologies-and-data-analysis-methods/differential-gene)
> means taking the normalised read count data and performing statistical analysis to discover quantitative
> changes in expression levels between experimental groups. For example, we use statistical testing to decide
> whether, for a given gene, an observed difference in read counts is significant, that is, whether it is greater
> than what would be expected just due to natural random variation.
>
{: .comment}

In principle, DEA is a causal analysis; but in reality, it is hampered by the complexity of the experimental
situation and measurement. Back to our datasets, CP and CN, they are from two experimental groups. By DEA,
we hope to pinpoint the candidate genes relevant to dental caries first, then we will use Heinz to infer the
related pathways.

## Which tools are available for DEA?

There are a few canned tools commonly used for DEA, like [Limma](https://bioconductor.org/packages/release/bioc/html/limma.html)
and [DESeq2](https://bioconductor.org/packages/release/bioc/html/DESeq2.html). If you are interested, you
may look up the pros and cons of each tool. Here we use DESeq2.


## Perform Differential Expression Analysis (DEA)

After learning about differential expression analysis, let's get some hands-on experience.

> <hands-on-title>DEA via DESeq2</hands-on-title>
>
> - **DESeq2** {% icon tool %} with the following parameters
>   - *"Specify a factor name"*: `dental_caries` (under `1: Factor`)
>   - *"Specify a factor level"*: to `CP` (under `1: Factor level`)
>       - {% icon param-files %} *"Counts file(s)"*: `CP` (collection)
>   - *"Specify a factor level"*: `CN` (under `2: Factor level`)
>       - {% icon param-files %} *"Counts file(s)"*: `CN` (collection)
>   - *"Files have header"*: `No`
>   - *"Visualising the analysis results"*: `No`
>   - Leave all other parameters to the default settings <br><br>
>
{: .hands_on}

It takes a few minutes to finish DESseq2. After the analysis is finished, have a look at the file, it should look like something this:

```
GeneID  Base mean   log2(FC)	        StdErr	            Wald-Stats	        P-value	                P-adj
K15661  55.0733128361562    2.49892393773464	0.508475451930939	4.91454194739384	8.89902710599613e-07	0.00491938218419466
K15666  48.4532561772761    2.22805029428311	0.493391718412049	4.51578372951607	6.30830206230236e-06	0.00820667379798327
K10213  25.1966742274619    -2.45624670858868	0.550895251183889	-4.45864563782342	8.24791475064012e-06	0.00820667379798327
K03732  563.63634258472 -1.41961721984814	0.316992240900184	-4.47839737596338	7.52055094378821e-06	0.00820667379798327
K01792  112.923146882195    -1.26925892659617	0.285732234964578	-4.44212717810268	8.90738834802815e-06	0.00820667379798327
```


# Fit a BUM model (a mixture model)

From a statistical point of view, p-values are uniformly distributed under null hypothesis; in other words,
under alternative hypothesis, the noise component (which holds under null hypothesis) will be adequately
modeled by a uniform distribution. With this knowledge, we can fit these p-values to a mixture model
(BUM model), as the figure below shows.

![p-values are fitted to a mixture model](../../images/transcriptomics_images/bum.jpeg){:width="50%"}

> <comment-title>Why do we need a mixture model?</comment-title>
>
> To avoid making this tutorial sound like a math class, let's focus on the philosophy of using a statistical model.
> In the graph above, we have visualized the p-values; those bins have some shapes; by viewing the figure, it
> is possible for us to cherrypick the data points (KO) with mesh eyes, but it must be tedious and boring, nobody wants to
> spend a few days on doing this. In this situation, computers come into play; however, unlike humans, computers cannot understand the graph but
> mathematical formulas, that's why we need to fix the data into a statistical model (denoted by the red curve and blue line in the graph),
> which computers can understand easily.
>
{: .comment}


Before fitting to BUM model in Galaxy, we need to prepare the input data for the tool
**Fit a BUM model**, that's a file that only contains p-values.

> <hands-on-title>extract p-values from DESeq2 output</hands-on-title>
>
> - **cut** columns from a table {% icon tool %} with the following parameters:
>   - *"Cut columns"*: `c6`
>   - *"Delimited by"*: to `TAB`
>   - *"From"*: to the output of **DESeq2**
>
{: .hands_on}

Now we can **Fit a BUM model**:

> <hands-on-title>fit the BUM model</hands-on-title>
>
> - **Fit a BUM model** {% icon tool %} with the following parameters
>   - *"Input file"*: the output of **cut**
>
{: .hands_on}

# Pinpoint the key pathways with Heinz

After getting the parameters of the BUM model from the last step, we will use Heinz to pinpoint
the optimal subnetwork. Then we could look for the key pathways relevant to the subnetwork.
Before we continue, let's figure out what Heinz is actually doing.

Heinz is an algorithm for searching an optimal subnetwork from a bigger network. You may wonder what
the network is here. Through the previous steps, we have got a list of identities, that is a list
of gene IDs with p-values, which form the nodes  of 'the bigger network', the relations between the
nodes, that is the edges, needs to be obtained from a background network, which represents a pathway
relation databases, such as [Reactome](https://reactome.org/) and [STRING](https://string-db.org/).
In this tutorial, we only use a small sample background network for demonstration purposes. The
background network is represented as edges in a TXT file where each line denotes an edge as follows:

```
ACTR1B	ACVR2B
ZSWIM9	FOXP3
LGALS4	PRKX
NPTX1	CIAO1
```

Upload this edge file (hereafter we call it edge file) into the Galaxy instance.

> <hands-on-title>Upload edge file</hands-on-title>
>
> 1. **Upload the edge file from Zenodo**
>    - Open the upload menu
>    - Choose **Paste/Fetch data** option
>    - Paste the URL into the box:
>      ```
>      https://zenodo.org/record/1344105/files/edge.txt
>      ```
>    - Click **Start**
{: .hands_on}


## Calculate Heinz scores

As the first step, we need to calculate a Heinz score for each node, using the BUM model parameters
we obtained; meanwhile, we also need to specify an FDR value as input.

> <comment-title>What is an FDR value?</comment-title>
>
> FDR is short for false discovery rate, which is a method of conceptualizing the rate of type I errors
> in null hypothesis testing when conducting multiple comparisons, if you are interested, view the detail
> in [Wikipedia](https://en.wikipedia.org/wiki/False_discovery_rate).
>
{: .comment}


For different datasets and problems, we probably need to pick up an FDR value separately. Here we set FDR to 0.11.

Similar to **Fit a BUM model**, we also need to prepare the input data for the tool
**Calculate a Heinz score**.

> <question-title></question-title>
>
> What is the requirement of the input data format for **Calculate a Heinz score**?
>
> > <solution-title></solution-title>
> >
> > In the user interface of the tool "Calculate a Heinz score", we see that "A node file with p-values" is needed.
> > It should contain two columns delimited by tab, one is KO ID; the other, p-value.
> >
> {: .solution}
{: .question}

> <hands-on-title>extract geneID and p-values from DESeq2 output</hands-on-title>
>
> - **cut** columns from a table {% icon tool %} with the following parameters
>   - *"Cut columns"*: `c1,c6`
>   - *"Delimited by"*: `TAB`
>   - *"From"*: the output of **DESeq2**
>
{: .hands_on}

> <hands-on-title>calculate Heinz scores</hands-on-title>
>
> - **Calculate a Heinz score** {% icon tool %} with the following parameters
>   - *"A node file with p-values"*: the output of the previous **cut** step
>   - *"FDR value"*: to 0.11
>   - *"Choose your input type for BUM parameters"*: `The output file of BUM model`
>   - *"Output file of BUM model as input: lambda on the first line and alpha, the second"*: the output of **Fit a BUM model**
{: .hands_on}


> <comment-title>What is the Heinz score?</comment-title>
>
> To figure out this score without reading the formula, we can understand Heinz score in this way. FDR is involved in calculating a p-value threshold
> and any KO whose p-value is below this threshold is regarded as significant, which means the Heinz score is positive (another calculation in the formula). If we pick a higher FDR value, then we will have a higher p-value threshold, and more KOs are probably regarded as significant. In this situation, we probably have many false positive (those regarded as significant are actually not) on the one hand; on the other hand, Heinz will deliver a bigger subnetwork, which might be exhausting to analyze. Therefore, we need to pick up an FDR value properly.
>
{: .comment}


## Run Heinz: pinpoint the optimal subnetwork

After getting Heinz scores, let's run Heinz program to find the optimal subnetwork from the background network which we mentioned earlier.

> <hands-on-title>pinpoint the optimal subnetwork</hands-on-title>
>
> - **Identify optimal scoring subnetwork** {% icon tool %} with the following parameters
>   - *"File containing Heinz scores"*: the output of **Calculate a Heinz score**
>   - *"Edge file"*: the edge file uploaded
>
{: .hands_on}

It usually takes a few minutes to get the result, but mind you, for some tasks, it might take a few hours to get a result in practice.

> <comment-title>The running time of the program</comment-title>
>
> * Graph problem is way more complicated than we thought.
> * It might take a much longer time for some complicated datasets.
> * We can use multiple CPUs to accelerate the computation (for now, this function is not available in Galaxy yet),
>   but, to use that, you can install Heinz directly via [Bioconda](https://anaconda.org/bioconda/heinz) in a Linux environment.
{: .comment}

## Visualize the output: visualize the optimal subnetwork

The result we got from the last step is not very human-readable, is it? It is a little painful to understand the Heinz result directly. Therefore we need to visualize the output by making it into graphs.

> <hands-on-title>visualize the optimal subnetwork</hands-on-title>
>
> - **Visualize** the optimal scoring subnetwork {% icon tool %} with the following parameters
>   - *"Heinz output file"*: the output of **Identify optimal scoring subnetwork**
>
{: .hands_on}

In this tutorial, you probably get a similar graph to the following:

![p-values are fitted to a mixture model](../../images/transcriptomics_images/Heinz_visualisation.png){:width="30%"}

> <question-title></question-title>
>
> According to the figure above, Why are some shapes are round, others square?
>
> > <solution-title></solution-title>
> >
> > Round shape corresponds to positive Heinz score; square shape corresponds to negative Heinz score.
> >
> {: .solution}
{: .question}


In addition to the tool we will use in Galaxy, you may consider using eXamine plugin in Cytoscape for a richer visualization, if we use the original data from the study [May et al.](https://academic.oup.com/bioinformatics/article/32/11/1678/2240171), perform all of these steps and visualize the result by eXamine, we can get a visualization as follows:

![metaModules Result](../../images/transcriptomics_images/metaModules.png){:width="60%"}

If you are interested, you may try to make sense of the result yourself. To answer the question "which pathways are potentially contributing to dental caries?", we need to find the key pathways relevant to the subnetwork. To achieve that, we probably need to combine information from other bioinformatics services and literature.


# Save the history into a workflow

Let's assume that these are the operations we always need for different datasets, to make life easier, we can save all of these operations into a workflow in Galaxy, which you can reuse for different datasets next time.

> <hands-on-title>Extract the Heinz workflow</hands-on-title>
>
> 1. Extract the workflow from your history
>
>    {% snippet faqs/galaxy/workflows_extract_from_history.md %}
>
> 2. Run the workflow on the dataset in this tutorial
>
{: .hands_on}

![Heinz workflow](../../images/transcriptomics_images/heinz-workflow.png)

As a self practice, you ran the newly created workflow with the dataset in this tutorial. Do you feel the convenience of running a workflow directly?

Congrats! You have finished all the tools in Heinz workflow! You have successfully run each tool and understood how it works. In real research, running these tools are only part of the effort, we still need to invest a huge amount of intelligence in making sense of the results and converting them to knowledge, which is fraught with uncertainties and confounders, where perhaps luck will come into play. So good luck!


topic: transcriptomics
tutorial: small_ncrna_clustering
title: Small Non-coding RNA Clustering using BlockClust
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 1b94e50d804131122df3e0590d260f37
last_modified: 2026-05-15T22:28:17.659872
zenodo_link: https://zenodo.org/record/1491876
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/small_ncrna_clustering/tutorial.html

content:
Small Non-coding RNAs (ncRNAs) play a vital role in many cellular processes such as RNA splicing, translation, gene regulation. The small RNA-seq is a type of RNA-seq in which RNA fragments are size selected to capture only short RNAs. One of the most common applications of the small RNA-seq is discovering novel small ncRNAs. Mapping the small RNA-seq data reveals interesting patterns that represent the traces of the small RNA processing.

For example, consider the miRNA biogenesis. The primary miRNA transcripts are processed by Drosha-complexes and results in hairpin precursor miRNAs. Then after they transported to the cytoplasm, Dicer slices off the hairpin. One of the RNA strands bound by Argonaute proteins regulates the target mRNA while the other strand is degraded.

The following figure (bottom part of the figure is taken from {% cite Pasquinelli2012 %}) represents one of the small RNA processing patterns. The vertical block bars represent a mapped reads on reference genome. The height of the bars represents the number of reads mapped at that location. Reads in each bubble represent a *read profile*. In this case, they both are from miRNAs. From the small RNA-seq, we often see the two processed miRNA strands after mapping. The miRNA strand which targets the mRNA is expressed and we see more reads compared to the degraded strand. The gap between those two piles of reads represents the missing hairpin.

![read profiles](../../images/transcriptomics_images/blockclust_profiles.png "Patterns of processing in read profiles")

In this tutorial, we will learn how to use **BlockClust** to cluster similar processing patterns together.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preprocessing of the data
The data is we use is from human cell line MCF-7 sequenced in single-end on Illumina Genome Analyzer II. Reads are 36nt long and should be sufficiently long enough for this type of data. We start with a BAM file as an input for this tutorial. This BAM file is a result of mapping the reads on human reference genome build hg19 using **segemehl**. If you want to use your own raw sequencing data, please clip the adapters and map to the reference genome.

In the first step, we convert the BAM file into BED file. This is not a plain file conversion. Hence we cannot use any kind of BAM to BED conversion tools. We use BlockClust tool in the pre-processing mode for this purpose. The resulting BED file contains tags (a tag is a unique read sequence in a deep-sequencing library) and their normalized expression (in column 5), i.e. the ratio of the read count per tag to the number of mappings on the reference genome.

{: .hands_on}
## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo](https://zenodo.org/record/2172221/files/GSM769512.bam)
>
>    ```
>    https://zenodo.org/record/2172221/files/GSM769512.bam
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
{: .hands_on}

> <question-title></question-title>
>
> Why is it mandatory to clip the adapters from the small RNA-seq data?
>
> > <solution-title></solution-title>
> >
> > Generally, the processed RNA fragments from the small RNAs are about 18-30nt long. If the sequenced read length is longer than the fragment (which is usually the case), the sequencer reads into the 3' adapter.
> >
> {: .solution}
>
{: .question}


## Sort BAM file
Before continue to BAM to BED conversion we need to sort the alignments in the input BAM file by their positions.
> <hands-on-title>Sort BAM</hands-on-title>
> 1. **Samtools sort** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"BAM File"*: `GSM769512.bam`
>
{: .hands_on}

## BAM to BED of tags
Now it is time to do the actual conversion.
> <hands-on-title><b>BlockClust</b> preprocessing</hands-on-title>
> 1. **BlockClust** {% icon tool %} with the following parameters:
>    - *"Select mode of operation"*: `Pre-processing `
>    - {% icon param-file %} *"BAM file containing alignments"*: output of **Samtools sort** {% icon tool %}
>
{: .hands_on}

# Grouping of the adjacent reads into read profiles

Now we group the adjacent reads into so-called blocks and blockgroups using blockbuster tool. In general, each blockgroup should represent a single ncRNA read profile.

![blocks of a blockgroup](../../images/transcriptomics_images/blockclust_blocks.png "Representation of blocks and blockgroups of a read profile")

> <details-title>More details about the theory</details-title>
>
> The idea is to perform peak detection on the signal obtained by counting the number of reads per nucleotide. This signal, spanning adjacent loci, is then modeled with a mixture of Gaussians. An iterative greedy procedure is then used to collect reads that belong to the same block, starting from the largest Gaussian component, and removing them in successive iterations. The tool further assembles a sequence of adjacent blocks into a blockgroup if the blocks are either overlapping or are at a distance smaller than a user-defined threshold. A more detailed explanation is at [the blockbuster website](http://hoffmann.bioinf.uni-leipzig.de/LIFE/blockbuster.html)
>
{: .details}


## Sort the BED file of tags

In order to run the blockbuster successfully the input BED file need to be sorted by chromosome, strand, start position and then end positions.

> <hands-on-title>Sort the BED file</hands-on-title>
>
> 1. **Sort** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: output of **BlockClust** {% icon tool %}
>    - In *"Column selections"*:
>        - *"on column"*: `Column: 1`
>        - *"Flavor"*: `Alphabetical sort`
>    - Click on *"Insert Column selections"*:
>    - In *"Column selections"*:
>        - *"on column"*: `Column: 6`
>        - *"Flavor"*: `Alphabetical sort`
>    - Click on *"Insert Column selections"*:
>    - In *"Column selections"*:
>        - *"on column"*: `Column: 2`
>        - *"Flavor"*: `Fast numeric sort`
>    - Click on *"Insert Column selections"*:
>    - In *"Column selections"*:
>        - *"on column"*: `Column: 3`
>        - *"Flavor"*: `Fast numeric sort`
>    - *"Ignore case"*: `Yes`
>
{: .hands_on}

## Group reads into blocks and blockgroups using **blockbuster**

> <hands-on-title>blockbuster</hands-on-title>
>
> 1. **blockbuster** {% icon tool %} with the following parameters:
>    - {% icon param-file %} *"BED file containing read expressions"*: output of **Sort** {% icon tool %}
>    - *"Type of output"*: `reads (blockbuster format)`
>
{: .hands_on}


# Clustering with **BlockClust**
Here we use **BlockClust** in clustering mode. All you need here is the output of the **blockbuster**.

Apart from clustering, **BlockClust** has built-in class specific discriminative models for C/D box snoRNA, H/ACA box snoRNA, miRNA, rRNA, snRNA, tRNA and Y_RNA. So it can also be used to predict if a read profile might belong to one of the known ncRNA class.
> <hands-on-title>clustering</hands-on-title>
>
> 1. **BlockClust** {% icon tool %} with the following parameters:
>    - *"Select mode of operation"*: `Clustering and classification`
>        - {% icon param-file %} *"Input blockgroups file"*: output of **blockbuster** {% icon tool %}
>        - *"Select reference genome"*: `Human (hg19)`
>        - *"Would you like to perform classification?"*: `Yes`
>        - *"Mode of classification"*: `Model based`
>
{: .hands_on}

The tool produces the following four output files.

The `BlockClust: BED of predicted clusters` file is the result of Markov cluster algorithm. Each entry in this file represents a blockgroup. The 4th column contains is in the form of *annotation:blockgroup_id:cluster_id*. The *cluster_id* represents which cluster the blockgroup belongs to.

The `BlockClust: Model based predictions BED` file is the result of classification. Each entry in this file represents a blockgroup. The 4th column contains the annotation of the ncRNA which it overlaps. For the blockgroups which are not overlapped with any known ncRNAs, the prefix `predicted_` is added indicating that it is predicted by the **BlockClust** classification models.

The `BlockClust: Hierarchical clustering plot` file shows a dendrogram construced based on average linkage clustering.

The `BlockClust: Pairwise similarities` file contains the pairwise similarities of all input blockgroups.

Next, we will visualize the read profiles (from the BAM file) and the predictions (BED from **BlockClust**) together.

> <hands-on-title>visualization</hands-on-title>
>
> 1. Install [IGV](https://software.broadinstitute.org/software/igv/download) (if not already installed)
> 2. Start IGV locally
> 3. Expand the {% icon param-file %} `GSM769512.bam` file
> 4. Click on the `local` in `display with IGV` to load the reads into the IGV browser
> 5. Expand the {% icon param-file %} `BlockClust: Model based predictions BED` file
> 6. Click on the `local` in `display with IGV` to load the reads into the IGV browser
> 7. Go to the location `chr4:90653059-90653141` in IGV.
>
{: .hands_on}

> <question-title></question-title>
>
> 1. Do you see any already annotated transcript in that location?
> 2. What does the read profile resemble?
> 3. What did **BlockClust** predict?
>
> > <solution-title></solution-title>
> >
> > 1. There is no annotated gene from RefSeq annotation on the same strand.
> > 2. It resembles the miRNA profiles mentioned in the introduction section. One of the read piles looks like mature miRNA and the other like miRNA*.
> > 3. Predicted as a miRNA.
> >
> {: .solution}
>
{: .question}

> <comment-title></comment-title>
>
> In order for this step to work, you will need to have either IGV or [Java web start](https://www.java.com/en/download/faq/java_webstart.xml)
> installed on your machine. However, the questions in this section can also be answered by inspecting the IGV screenshots below.
>
> Check the [IGV documentation](https://software.broadinstitute.org/software/igv/AlignmentData) for more information.
>
{: .comment}



# Conclusion


In this tutorial, we learned how to use **BlockClust** to cluster real small RNA sequencing read profiles based on their similarity. We also learned that the **BlockClust** can also be used to classify the read profiles based on pre-built classification models.


topic: transcriptomics
tutorial: srna
title: Differential abundance testing of small RNAs
difficulty: intermediate
time_estimation: 3h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 0178e80983559c5f60aa06f78c40f749
last_modified: 2026-05-15T22:28:17.660625
zenodo_link: https://zenodo.org/record/826906
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/srna/tutorial.html

content:
Small, noncoding RNA (sRNA) molecules, typically 18-40nt in length, are key features of post-transcriptional regulatory mechanisms governing gene expression. Through interactions with protein cofactors, these tiny sRNAs typically function by perfectly or imperfectly basepairing with substrate RNA molecules, and then eliciting downstream effects such as translation inhibition or RNA degradation. Different subclasses of sRNAs - *e.g.* microRNAs (miRNAs), Piwi-interaction RNAs (piRNAs), and endogenous short interferring RNAs (siRNAs) - exhibit unique characteristics, and their relative abundances in biological contexts can indicate whether they are active or not. In this tutorial, we will examine expression of the piRNA subclass of sRNAs and their targets in *Drosophila melanogaster*.

The data used in this tutorial are from polyphosphatase-treated sRNA sequencing (sRNA-seq) experiments in *Drosophila*. The goal of this study was to determine how siRNA expression changes in flies treated with RNA interference (RNAi) to knock down Symplekin, which is a component of the core cleaveage completx. To that end, in addition to sRNA-seq, mRNA-seq experiments were performed to determine whether targets of differentially expressed siRNAs were also differentially expressed. The original published study can be found [here](https://www.ncbi.nlm.nih.gov/pubmed/28415970). Because of the long processing time for the large original files - which contained 7-22 million reads each - we have downsampled the original input data to include only a subset of usable reads.

# Analysis strategy

In this exercise we will identify what small RNAs are present in *Drosophila* Dmel-2 tissue culture cells treated with either RNAi against  core cleavage complex component Symplekin or blank (control) RNAi (published data available in GEO at [GSE82128](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE82128)). In this study, biological triplicate sRNA- and mRNA-seq libraries were sequenced for both RNAi conditions. After removing contaminant ribosomal RNA (rRNA) and miRNA reads, we will identify and quantify endogenous siRNAs from sequenced reads and test for differential abundance. We will follow a popular small RNA analysis pipeline developed for piRNAs by the Phillip Zamore Lab and the ZLab (Zhiping Weng) at UMass Med School called [PiPipes](https://github.com/bowhan/piPipes). Although PiPipes was developed for analysis of piRNAs, many of the basical principles can be applied to other classes of small RNAs.

It is of note that this tutorial uses datasets that have been de-multiplexed so that the input data files are a single FASTQ formatted file for each sample. Because sRNAs are typically much smaller than fragments generated for RNA-seq or other types of deep sequencing experiments, single-end sequencing strategies are almost always used to sequence sRNAs. This tutorial uses the *Collections* feature of Galaxy to orgainze each set of replicates into a single group, making tool form submission easier and ensuring reproducibility.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Data upload and organization

Due to the large size of the original sRNA-seq datasets, we have downsampled them to only inlcude a subset of reads. These datasets are available at [`Zenodo`](https://doi.org/10.5281/zenodo.826906) where you can find the FASTQ files corresponding to replicate sRNA-seq experiments and additional annotation files for the *Drosophila melanogaster* genome version dm3.

> <hands-on-title>Data upload and organization</hands-on-title>
>
> 1. Create a new history and name it something meaningful (*e.g.* sRNA-seq tutorial)
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the 3 `Blank_RNAi_sRNA-seq` and 3 `Symp_RNAi_sRNA-seq` FASTQ files from [Zenodo](https://zenodo.org/record/1324070) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/826906/files/Blank_RNAi_sRNA-seq_rep1_downsampled.fastqsanger.gz
>    https://zenodo.org/record/826906/files/Blank_RNAi_sRNA-seq_rep2_downsampled.fastqsanger.gz
>    https://zenodo.org/record/826906/files/Blank_RNAi_sRNA-seq_rep3_downsampled.fastqsanger.gz
>    https://zenodo.org/record/826906/files/Symp_RNAi_sRNA-seq_rep1_downsampled.fastqsanger.gz
>    https://zenodo.org/record/826906/files/Symp_RNAi_sRNA-seq_rep2_downsampled.fastqsanger.gz
>    https://zenodo.org/record/826906/files/Symp_RNAi_sRNA-seq_rep3_downsampled.fastqsanger.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    Set the datatype of the read (.fastqsanger) files to **fastq**
>    As default, Galaxy takes the link as name, so rename them.
>
> 2. Build a dataset list for both sets (`Blank` and `Symp`) of replicate FASTQ files
>
>    {% snippet faqs/galaxy/collections_build_list.md %}
>
>    ![History section with all the loaded datasets](../../images/sRNA/Fig3_dataset_lists_made.png)
>
> 3. Import the remaining files from [Zenodo](https://zenodo.org/record/1324070) or from the data library (ask your instructor)
>
>    ```
>    https://zenodo.org/record/826906/files/dm3_transcriptome_Tx2Gene_downsampled.tab.gz
>    https://zenodo.org/record/826906/files/dm3_transcriptome_sequences_downsampled.fa.gz
>    https://zenodo.org/record/826906/files/dm3_miRNA_hairpin_sequences.fa.gz
>    https://zenodo.org/record/826906/files/dm3_rRNA_sequences.fa.gz
>    ```
>
>    - Set the datatype of the annotation file to **tab** and assign the Genome as **dm3**
>    - Set the datatype of the reference files to **fasta** and assign the Genome as **dm3**
>
{: .hands_on}

## Read quality checking

Read quality scores (phred scores) in FASTQ-formatted data can be encoded by one of a few different encoding schemes. Most Galaxy tools assume that input FASTQ files are using the Sanger/Illumina 1.9 encoding scheme. If the input FASTQ files are using an alternate encoding scheme, then some tools will not interpret the quality score encodings correctly. It is good practice to confirm the quality encoding scheme of your data and then convert to Sanger/Illumina 1.9, if necessary. We can check the quality encoding scheme using the [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) tool (further described in the [NGS-QC tutorial]({{site.baseurl}}/topics/sequence-analysis)).


> <hands-on-title>Quality checking</hands-on-title>
>
> 1. Run **FastQC** {% icon tool %} on the `Blank` collection
>    - {% icon param-collection %} *"Short read data from your current history"*: `Blank` collection
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
> 2. Run **FastQC** {% icon tool %} for the *Symplekin* RNAi dataset collection
> 3. Inspect the **FastQC** {% icon tool %} reports
>
>    > <question-title></question-title>
>    >
>    > 1. What quality score encoding scheme is being used for each sample?
>    > 2. What is the read length for each sample?
>    > 3. What does the base/read quality look like for each sample?
>    >
>    >    ![The output of FastQC as a boxplot](../../images/sRNA/Fig5a_fastqc_result_stats_boxplot.png)
>    >
>    > 4. Are there any adaptors present in these reads? Which one(s)?
>    >
>    >    ![The output of FastQC stats adapted](../../images/sRNA/Fig5b_fastqc_result_stats_adapter.png)
>    >
>    > > <solution-title></solution-title>
>    > > 1. All samples use the Illumina 1.9 quality encoding scheme, so we do **not** need to convert.
>    > > 2. All samples have a read length of 50 nt.
>    > > 3. The base quality across the entire length of the reads is good (phred score > 28 for the most part).
>    > > 4. Yes, the "Illumina Universal Adapter" is present.
>    > {: .solution }
>    {: .question}
{: .hands_on}

> <hands-on-title>Formatting </hands-on-title>
>
> If your data are not in **Sanger / Illumina 1.9 format**:
> 1. **FASTQ Groomer** {% icon tool %} on each collection
>    - {% icon param-collection %} *"File to groom"*: input collection
>    - *"Input FASTQ quality scores type"*: `Illumina 1.3-1.7`
> 2. Check that the format is **fastqsanger** instead of **fastq**
>
> If your data are already in **Sanger / Illumina 1.9 format**:
> 1. Change the datatype to `fastqsanger`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="fastqsanger" %}
>
{: .hands_on}

Go back to the **FastQC** {% icon tool %} and scroll down to the "Adapter Content" section. You can see that Illumina Universal Adapters are present in **>80%** of the reads. The next step is to remove these artificial adaptors because they are not part of the biological sRNAs. If a different adapter is present, you can update the **Adapter sequence to be trimmed off** in the **Trim Galore!** {% icon tool %} step described below.

## Adaptor trimming

sRNA-seq library preparation involves adding an artificial adaptor sequence to both the 5' and 3' ends of the small RNAs. While the 5' adaptor anchors reads to the sequencing surface and thus are not sequenced, the 3' adaptor is typically sequenced immediately following the sRNA sequence. In the example datasets here, the 3' adaptor sequence is identified as the Illumina Universal Adapter, and needs to be removed from each read before aligning to a reference. We will be using the Galaxy tool **Trim Galore!** {% icon tool %} which implements the [`cutadapt`](https://cutadapt.readthedocs.io/en/stable/) tool for adapter trimming.

> <hands-on-title>Adaptor trimming</hands-on-title>
>
> 1. **Trim Galore!** {% icon tool %} on each collection of FASTQ read files to remove Illumina adapters from the 3' ends of reads with the following parameters:
>    - *"Is this library paired- or single-end"*: `Single-end`
>       - {% icon param-collection %} *"Reads in FASTQ format?"*: `Blank` collection
>       - *"Adapter sequence to be trimmed"*: `Illumina universal`
>    - *"Trim Galore! advanced settings"*: `Full parameter list`
>       - *"Trim low-quality ends from reads in addition to adapter removal"*: `0`
>       - *"Overlap with adapter sequence required to trim a sequence"*: `6`
>       - *"Discard reads that became shorter than length N"*: `12`
>       - *"Generate a report file"*: `Yes`
>
> 2. **Trim Galore!** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
>
{: .hands_on}

We don't want to trim for quality because the adapter-trimmed sequences represent a full small RNA molecule, and we want to maintain the integrity of the entire molecule. We increase the minimum read length required to keep a read because small RNAs can potentially be shorter than 20 nt (the default value). We can check out the generated report file for any sample and see the command for the tool, a summary of the total reads processed and number of reads with an adapter identified, and histogram data of the length of adaptor trimmed. We also see that a very small percentage of low-quality bases have been trimmed

> <hands-on-title>Adaptor trimming check</hands-on-title>
>
> 1. **FastQC** {% icon tool %} on each collection of trimmed FASTQ read files to assess whether adapters were successfully removed.
> 2. Inspect the **FastQC** {% icon tool %} reports
>
>    > <question-title></question-title>
>    >
>    > 1. What is the read length?
>    >
>    >   ![FastQC output after trimming](../../images/sRNA/Fig8a_fastqc_post_trimming_result.png)
>    >
>    > 2. Are there any adaptors present in these reads? Which one(s)?
>    >
>    >   ![FastQC output after trimming](../../images/sRNA/Fig8b_fastqc_post_trimming_lengths.png)
>    > > <solution-title></solution-title>
>    > > 1. The read lengths range from 12 to 50 nt after trimming.
>    > > 2. No, Illumina Universal Adaptors are no longer present. No other adapters are present.
>    > {: .solution }
>    {: .question}
{: .hands_on}

An interesting thing to note from our **FastQC** {% icon tool %} results is the *Sequence Length Distribution* results. While many RNA-seq experiments have normal distribution of read lengths, an unusual spike at 22nt is observed in our data. This spike represents the large set of endogenous siRNAs that occur in the cell line used in this study, and is actually confirmation that our dataset captures the biological molecule we are interested in.

Now that we have converted to *fastqsanger* format and trimmed our reads of the Illumina Universal Adaptors, we will align our trimmed reads to reference *Drosophila* rRNA and miRNA sequences (dm3) to remove these artifacts. Interestingly, *Drosophila* have a short 2S rRNA sequence that is 30nt long and typically co-migrates with sRNA populations during gel electrophoresis. rRNAs make up a very large proportion of all non-coding RNAs, and thus need to be removed. Oftentimes, experimental approaches can be utilized to deplete or avoid capture of rRNAs, but these methods are not always 100% efficient. We also want to remove any miRNA sequences, as these are not relevant to our analysis. After removing rRNA and miRNA reads, we will analyze the remaining reads, which may be siRNA or piRNA sequences.

## Hierarchical read alignment to remove rRNA/miRNA reads

To first identify rRNA-originating reads (which we are not interested in in this case), we will align the reads to a reference set of rRNA sequences using [**HISAT2** {% icon tool %}](https://ccb.jhu.edu/software/hisat2/index.shtml), an accurate and fast tool for aligning reads to a reference.

> <hands-on-title>Hierarchical alignment to rRNA and miRNA reference sequences</hands-on-title>
>
> 1. **HISAT2** {% icon tool %} on each collection of trimmed reads to align to reference rRNA sequences with the following parameters
>    - *"Source for the reference genome to align against"*: `Use a genome from history`
>       - {% icon param-file %} *"Select the reference genome"*: `Dmel_rRNA_sequences.fa`
>    - *"Is this a single or paired library?"*: `Single-end`
>       - {% icon param-collection %} *"FASTA/Q file"*: output of **Trim Galore!** {% icon tool %} for `Blank` collection
>       - *"Specify strand-specific information"*: `Forward (F)`
>
> 2. **HISAT2** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
{: .hands_on}

We now need to extract the *unaligned* reads from the output BAM file for aligning to reference miRNA sequences. We can do this by using the `Filter SAM or BAM, output SAM or BAM` tool to obtain reads with a bit flag = 4 (meaning the read is unaligned) and then converting the filtered BAM file to FASTQ format with the `Convert from BAM to FastQ` tool.

> <hands-on-title>Hierarchical alignment to rRNA and miRNA reference sequences (2)</hands-on-title>
> 1. **Filter SAM or BAM, output SAM or BAM** {% icon tool %} on collection of HISAT2 output BAM files with the following parameters
>    - {% icon param-collection %} *"SAM or BAM file to filter"*: output of **HISAT2** {% icon tool %} for `Blank` collection
>    - *"Filter on bitwise flag"*: `Yes`
>       - *"Only output alignments with all of these flag bits set"*: `The read in unmapped`
>
> 2. **Filter SAM or BAM, output SAM or BAM** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
> 3. **Convert from BAM to FastQ** {% icon tool %} on each collection of filtered HISAT2 output BAM files with the following parameters:
>    - {% icon param-collection %} *"Convert the following BAM file to FASTQ"*: output of **Filter SAM or BAM** {% icon tool %} for `Blank` collection
>
> 4. **Convert from BAM to FastQ** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
{: .hands_on}

Next we will align the non-rRNA reads to a known set of miRNA hairpin sequences to identify miRNA reads.

> <hands-on-title>Hierarchical alignment to rRNA and miRNA reference sequences (3)</hands-on-title>
> 1. **HISAT2** {% icon tool %} on each collection of filtered HISAT2 output FASTQ files to align non-rRNA reads to reference miRNA hairpin sequences using the following parameters:
>    - *"Source for the reference genome to align against"*: `Use a genome from history`
>       - {% icon param-file %} *"Select the reference genome"*: `Dmel_miRNA_sequences.fa`
>    - *"Is this a single or paired library?"*: `Single-end`
>       - {% icon param-collection %} *"FASTA/Q file"*: output of **Convert from BAM to FastQ** {% icon tool %} for `Blank` collection
>       - *"Specify strand-specific information"*: `Forward (F)`
> 2. **HISAT2** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
{: .hands_on}

For this tutorial we are not interested in miRNA reads, so we need to extract *unaligned* reads from the output BAM files. To do this, repeat the **Filter SAM or BAM, output SAM or BAM** {% icon tool %} steps for each dataset collection. Finally, rename the converted FASTQ files something meaningful (*e.g.* "non-r/miRNA control RNAi sRNA-seq").

In some instances, miRNA-aligned reads are the desired output for downstream analyses, for example, if we are investigating how loss of key miRNA pathway components affect levels of pre-miRNA and mature miRNAs. In this case, the BAM output of HISAT2 can directly be used in subsequent tools as it contains the subset of reads aligned to miRNA sequences.

## Small RNA subclass distinction

In *Drosophila*, non-miRNA small RNAs are typically divided into two major groups: endogenous siRNAs which are 20-22nt long and piRNAs which are 23-29nt long. We want to analyze these sRNA subclasses independently, so next we are going to filter the non-r/miRNA reads based on length using the **Manipulate FASTQ** {% icon tool %} tool.

> <hands-on-title>Extract subclasses</hands-on-title>
>
> 1. **Manipulate FASTQ** {% icon tool %} on each collection of non-r/miRNA reads to identify siRNAs (20-22nt) using the following parameters.
>    - {% icon param-collection %} *"FASTQ File"*: output of last **HISAT** {% icon tool %} for `Blank` collection
>    - Click on *"Insert Match Reads"*
>       - In "1: Match Reads"
>         - *"Match Reads by"*: `Sequence Content`
>           - *"Match by"*: `^.{12,19}$|^.{23,50}$`
>    - Click on *"Manipulate Reads"*
>       - In "1: Manipulate Reads"
>         - *"Manipulate Reads on"*: `Miscellaneous actions`
>           - *"Miscellaneous Manipulation Type"*: `Remove Read`
> 2. **Manipulate FASTQ** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
> 3. Rename each resulting dataset collection something meaningful (*i.e.* "blank RNAi - siRNA reads (20-22nt)")
{: .hands_on}

The regular expression in the **Match by** parameter tells the tool to identify sequences that are length 12-19 or 23-50 (inclusive), and the **Miscellaneous Manipulation Type** parameter tells the tool to remove these sequences. What remains are sequences of length 20-22nt. We will now repeat these steps to identify sequences in the size-range of piRNAs (23-29nt).

> <hands-on-title>Extract subclasses</hands-on-title>
> 1. **Manipulate FASTQ** {% icon tool %}: Run `Manipulate FASTQ` on each collection of non-r/miRNA reads to identify sequences in the size-range of piRNAs (23-29nt)
>    - {% icon param-collection %} *"FASTQ File"*: output of last **HISAT** {% icon tool %} for `Blank` collection
>    - Click on *"Insert Match Reads"*
>       - In "1: Match Reads"
>         - *"Match Reads by"*: `Sequence Content`
>           - *"Match by"*: `^.{12,22}$|^.{30,50}$`
>    - Click on *"Manipulate Reads"*
>       - In "1: Manipulate Reads"
>         - *"Manipulate Reads on"*: `Miscellaneous actions`
>           - *"Miscellaneous Manipulation Type"*: `Remove Read`
> 2. **Manipulate FASTQ** {% icon tool %}: Repeat for the *Symplekin* RNAi dataset collection
> 3. Rename each resulting dataset collection something meaningful (*i.e.* "blank RNAi - piRNA reads (23-29nt)")
{: .hands_on}

> <hands-on-title>Check the read length</hands-on-title>
> 1. **FastQC** {% icon tool %}: Run `FastQC` on each collection of siRNA and piRNA read files to confirm the correct read lengths.
{: .hands_on}

![FastQC result on both collections](../../images/sRNA/Fig13_FastQC_piRNA_siRNA_result_Blank_rep1.png)

We see in the above image (for Blank RNAi, replicate 3) that we have ~32k reads that are 20-22nt long in our Reads_20-22_nt_length_siRNAs file and ~16k reads that are 23-29nt long in our Reads_23-29_nt_length_piRNAs file, as expected. Given that we had ~268k trimmed reads in the Blank RNAi replicate 3 file, siRNAs represent ~12.1% of the reads and piRNAs represent ~6.2% of the reads in this experiment. There is no distinct peak in the piRNA range (you can find the example of the real piRNA profile in Figure 1 from [Colin D. Malone et al, 2009](http://dx.doi.org/10.1016/j.cell.2009.03.040).

The next step in our analysis pipeline is to identify which RNA features - e.g. protein-coding mRNAs, transposable elements - the siRNAs and piRNAs align to. Most fly piRNAs originate from and thus align to transposable elements (TEs) but some also originate from genome piRNA clusters or protein-coding genes. siRNAs are thought to inhibit TE mobility and potentially target mRNAs for degradation via an RNA-interference (RNAi) mechanism. Ultimately, we want to know whether an RNA feature (TE, piRNA cluster, mRNA, etc.) has significantly different numbers of siRNAs or piRNAs targeting it. To determine this, we will use **Salmon** {% icon tool %} to simultaneously align and quantify siRNA reads against known target sequences to estimate siRNA abundance per target. For the remainder of the tutorial we will be focusing on siRNAs, but similar approaches can be done for piRNAs, miRNAs, or any other subclass of small, non-coding RNAs.

## siRNA abundance estimation

We want to identify which siRNAs are differentially abundance between the blank and *Symplekin* RNAi conditions. To do this we will implement a counting approach using **Salmon** {% icon tool %} to quantify siRNAs per RNA feature. Specifically, we will analyzing mRNA and TE elements by counting siRNAs against a FASTA file of transcript sequences, not the reference genome. This approach is especially useful in this case because we are interested in TEs, which occur at many copies (100s - 1000s) in the genome, and thus will result in multiple alignments if we align to the genome with a tool like **HISAT2** {% icon tool %}. Further, we will be counting abundance of siRNAs that align *sense* or *antisense* to an RNA feature independently, as opposite sense alignments are correlated with unique downstream silencing effects. Then, we will provide this information to **DESeq2** {% icon tool %} to generate normalized counts and significance testing for differential abundance of siRNAs per feature.

> <hands-on-title>siRNA abundance estimation</hands-on-title>
>
> 1. **Salmon** {% icon tool %} on each collection of siRNA reads (20-22nt) to quantify the abundance of *antisense* siRNAs at relevant targets
>    - *"Select a reference transcriptome from your history or use a built-in index?"*: `Use one from the history`
>      - {% icon param-file %} *"Select the reference transcriptome"*: `the reference mRNA and TE fasta file`
>      - *"The size should be odd number"*: `19`
>    - *"Is this library mate-paired?"*: `Single-end`
>      - {% icon param-collection %} *"FASTQ/FASTA file"*: collection with the Blank RNAi siRNA (20-22nt) reads
>      - *"Specify the strandedness of the reads"*: `read 1 (or single-end read) comes from the reverse strand (SR)`
>    - In *"Additional Options"*
>      - *"Incompatible Prior"*: `0`
>
> 1. **Salmon** {% icon tool %}: Repeat for the *Symplekin* RNAi siRNAs (20-22nt) reads dataset collection
> 1. **Salmon** {% icon tool %}: Repeat on each collection of siRNA reads (20-22nt) to quantify the abundance of *sense* siRNAs at relevant targets by changing the following parameters:
>    - *"Specify the strandedness of the reads"*: `read 1 (or single-end read) comes from the forward strand (SF)`
{: .hands_on}

The output of **Salmon** {% icon tool %} includes a table of RNA features, estimated counts, transcripts per million, and feature length. We will be using the TPM quantification column as input to **DESeq2** {% icon tool %} in the next section.

## siRNA differential abundance testing

[**DESeq2** {% icon tool %}](https://bioconductor.org/packages/release/bioc/html/DESeq2.html) is a great tool for differential expression analysis, but we also employ it here for estimation of abundance of reads targeting each of our RNA features. As input, **DESeq2** {% icon tool %} can take *t*ranscripts *p*er *m*illion (TPM) counts produced by **Salmon** {% icon tool %} for each feature. TPMs are estimates of the relative abundance of a given transcript in units, and analogously can be used here to quantify siRNAs that align either sense or antisense to transcripts.

> <hands-on-title>siRNA differential abundance testing</hands-on-title>
>
> 1. **DESeq2** {% icon tool %} to test for differential abundance of *antisense* siRNAs at mRNA and TE features using the following parameters:
>    - In *"Factor"*:
>       - In "1: Factor"
>           - *"Specify a factor name"*: `RNAi`
>           - In *"Factor level"*:
>               - In *"1: Factor level"*:
>                   - *"Specify a factor level"*: `Symplekin`
>                   - {% icon param-collection %} *"Counts file(s)"*: the Symplekin RNAi siRNA counts from reverse strand (SR) collection
>               - In *"2: Factor level"*:
>                   - *"Specify a factor level"*: `RNAi`
>                   - {% icon param-collection %} *"Counts file(s)"*: the Symplekin RNAi siRNA counts from reverse strand (SR)
>    - *"Choice of Input data"*: `TPM values (e.g. from sailfish and salmon)`
>       - *"Program used to generate TPMs"*: `Salmon`
>       - *"Gene mapping format"*: `Transcript-ID and Gene-ID mapping file`
>         - {% icon param-file %} *"Tabular file with Transcript - Gene mapping"*: `reference tx2g (transcript to gene) file` (available through zenodo link)
>    - *"Output normalized counts table"*: `Yes`
>
> 1. **DESeq2** {% icon tool %} for the *sense* siRNAs at mRNA and TE features:
>    - In *"Factor"*:
>       - In "1: Factor"
>           - *"Specify a factor name"*: `RNAi`
>           - In *"Factor level"*:
>               - In *"1: Factor level"*:
>                   - *"Specify a factor level"*: `Symplekin`
>                   - {% icon param-collection %} *"Counts file(s)"*: the Symplekin RNAi siRNA counts from forward strand (SF)
>               - In *"2: Factor level"*:
>                   - *"Specify a factor level"*: `RNAi`
>                   - {% icon param-collection %} *"Counts file(s)"*: the Symplekin RNAi siRNA counts from forward strand (SF)
>    - *"Choice of Input data"*: `TPM values (e.g. from sailfish and salmon)`
>       - *"Program used to generate TPMs"*: `Salmon`
>       - *"Gene mapping format"*: `Transcript-ID and Gene-ID mapping file`
>         - {% icon param-file %} *"Tabular file with Transcript - Gene mapping"*: `reference tx2g (transcript to gene) file` (available through zenodo link)
>    - *"Output normalized counts table"*: `Yes`
>
> 1. **Filter** {% icon tool %} to extract features with a significantly different *antisense* siRNA abundance (adjusted *p*-value less than 0.05).
>    - {% icon param-file %} *"Filter"*: Select the `DESeq2` result file from testing *antisense* (SR) siRNA abundances
>    - *"With following condition"*: `c7<0.05`
>
> 1. **Filter** {% icon tool %} for the *sense* siRNAs at mRNA and TE features:
>    - {% icon param-file %} *"Filter"*: Select the `DESeq2` result file from testing *sense* (SF) siRNA abundances
>    - *"With following condition"*: `c7<0.05`
{: .hands_on}

> <question-title></question-title>
>
> How many features have a significant difference in *antisense* and *sense* siRNA abundance in the *Symplekin* RNAi condition?
>
> > <solution-title></solution-title>
> > There are 87 *antisense* and 15 *sense* features that have significanlty different siRNA abundances.
> {: .solution }
{: .question}

For more information about `DESeq2` and its outputs, have a look at [`DESeq2` documentation](https://www.bioconductor.org/packages/release/bioc/manuals/DESeq2/man/DESeq2.pdf).

## Visualization

Coming soon!

# Conclusion


Analysis of small RNAs is a complicated and intricate process due to the diversity in characteristics, functionality, and nuances of small RNA subclasses. The goal of this tutorial is to introduce you to a common small RNA workflow that specifically identifies changes in endogenous siRNA abundances that target protein-coding mRNAs and transposable elements. The steps presented here can be rearranged and modified based on small RNA features of specific systems and the needs of the user.

## Small RNA analysis (general) pipeline

![sRNA tutorial workflow](../../images/sRNA/schematic_for_sRNAseq_tutorial.png)


topic: transcriptomics
tutorial: rna-seq-viz-with-volcanoplot
title: Visualization of RNA-Seq results with Volcano Plot
difficulty: introductory
time_estimation: 30m
tools_json: ["Volcano Plot", "toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.7"]
requirements_json: ["type: internal", "rna-seq-counts-to-genes"]
tags_json: []
content_hash: 18aa7d185476f5b7bc9653f0ee69a341
last_modified: 2026-05-27T09:01:49.988491
zenodo_link: https://zenodo.org/record/2529117
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot/tutorial.html

content:
![Volcano plot highlighting significant genes](../../images/rna-seq-viz-with-volcanoplot/volcanoplot.png){: style="float:right;width:60%" }
Volcano plots are commonly used to display the results of RNA-seq or other omics experiments. A volcano plot is a type of scatterplot that shows statistical significance (P value) versus magnitude of change (fold change). It enables quick visual identification of genes with large fold changes that are also statistically significant. These may be the most biologically significant genes. In a volcano plot, the most upregulated genes are towards the right, the most downregulated genes are towards the left, and the most statistically significant genes are towards the top.

To generate a volcano plot of RNA-seq results, we need a file of differentially expressed results which is provided for you here. To generate this file yourself, see the [RNA-seq counts to genes]({% link topics/transcriptomics/tutorials/rna-seq-counts-to-genes/tutorial.md %}) tutorial. The file used here was generated from limma-voom but you could use a file from any RNA-seq differential expression tool, such as edgeR or DESeq2, as long as it has the required columns (see below).

The data for this tutorial comes from {% cite Fu2015 %}. This study examined the expression profiles of basal and luminal cells in the mammary gland of virgin, pregnant and lactating mice. Here, we will visualize the results of the luminal pregnant vs lactating comparison.


> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Preparing the inputs

We will use two files for this analysis:

 * **Differentially expressed results file** (genes in rows, and 4 required columns: raw P values, adjusted P values (FDR), log fold change and gene labels)
 * **Genes of interest file** (list of genes to be plotted in volcano)

## Import data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-seq exercise e.g. `Volcano plot`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Import the result table of differential gene expression analysis, as well as a list of genes that will be annotated in the volcano plot later.
>
>    To import the file, there are two options:
>    - Option 1: From a shared data library if available (ask your instructor)
>    - Option 2: From [Zenodo](https://zenodo.org/record/2529117)
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>    - You can paste the links below into the **Paste/Fetch** box:
>
>      ```
>      https://zenodo.org/record/2529117/files/limma-voom_luminalpregnant-luminallactate
>      https://zenodo.org/record/2529117/files/volcano_genes
>      ```
>
>    - Select *"Type (set all)"*: `tabular`
>
>    After the files import, check that the datatype is `tabular`.
>    If the datatype is not `tabular`, please change the file type to `tabular`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
>
{: .hands_on}

Click on the {% icon galaxy-eye %} (eye) icon and take a look at the `limma-voom_luminalpregnant-luminallactate` file. It should look like below, with 8 columns.

![luminal file](../../images/rna-seq-viz-with-volcanoplot/DE_results.png)

## Create volcano plot highlighting significant genes

First, we will create a volcano plot highlighting all significant genes. We will call genes significant, if they have FDR < 0.01 and a log fold change of 0.58 (equivalent to a fold-change of 1.5). These were the values used in the original paper for this dataset.

> <hands-on-title>Create a Volcano plot</hands-on-title>
>
> 1. {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.7) %} to create a volcano plot:
>    - *"Specify an input file"*: limma-voom file
>    - *"FDR (adjusted P value)"*: `Column 8`
>    - *"P value (raw)"*: `Column 7`
>    - *"Log Fold Change"*: `Column 4`
>    - *"Labels"*: `Column 2`
>    - *"Significance threshold"*: `0.01`
>    - *"LogFC threshold to colour"*: `0.58`
>    - *"Points to label"*: `None`
{: .hands_on}

![Volcano plot highlighting significant genes](../../images/rna-seq-viz-with-volcanoplot/volcanoplot.png){: width="60%"}

In the plot above the genes are coloured if they pass the thresholds for FDR and Log Fold Change. The red dots are upregulated genes and the blue dots are downregulated genes. You can see in this plot that there are many (hundreds) of significant genes in this dataset.

> <question-title></question-title>
>
> Why does the y axis use a negative P value scale?
>
>    > <solution-title></solution-title>
>    >
>    > The negative log of the P values are used for the y axis so that the smallest P values (most significant) are at the top of the plot.
>    >
>    {: .solution}
{: .question}

## Create volcano plot labelling top significant genes

You can also choose to show the labels (e.g. Gene Symbols) for the significant genes with the volcano plot tool. You can select to label all significant or just the top genes. The top genes are those that pass the FDR and logFC thresholds that have the smallest P values. As there are hundreds of significant genes here, too many to sensibly label, let's label the top 10 significant genes.

> <hands-on-title>Create a Volcano plot labelling top 10 significant genes</hands-on-title>
> 1. Use the **Run Job Again** {% icon galaxy-refresh %} button in the History to re-run {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.7) %} with the same parameters as before except:
>    - *"Points to label"*: `Significant`
>        - *"Only label top most significant"*: `10`
{: .hands_on}


![Volcano plot labelling top significant genes](../../images/rna-seq-viz-with-volcanoplot/volcanoplot_top10.png){: width="60%"}

As in the previous plot, genes are coloured if they pass the thresholds for FDR and Log Fold Change, (red for upregulated and blue for downregulated) and the top genes by P value are labelled. Note that in the plot above we can now easily see what the top genes are by P value and also which of them have bigger fold changes.

> <question-title></question-title>
>
> Which gene is the most statistically significant with large fold change?
>
>    > <solution-title></solution-title>
>    >
>    > Csn1s2b, as it is the gene nearest the top of the plot and it is also far to the left. This gene is a calcium-sensitive casein that is important in milk production. As this dataset compares lactating and pregnant mice, it makes sense that it is a gene that is very differentially expressed.
>    >
>    {: .solution}
{: .question}

## Create volcano plot labelling genes of interest

We can also label one or more genes of interest in a volcano plot. This enables us to visualize where these genes are in terms of significance and in comparison to the other genes. In the original paper using this dataset, there is a heatmap of 31 genes in Figure 6b (have a look at this [visualization tutorial]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-heatmap2/tutorial.md %}) if you would like to see how to generate the heatmap). These genes are a set of 30 cytokines/growth factor identified as differentially expressed, and the authors' main gene of interest, Mcl1. These genes are provided in the `volcano_genes` file and shown below. We will label these genes in the volcano plot. We'll add boxes around the labels to highlight the gene names.

![Volcano genes](../../images/rna-seq-viz-with-volcanoplot/volcano_genes.png){: height="25%"}

> <hands-on-title>Create a Volcano plot labelling genes of interest</hands-on-title>
> 1. Use the **Run Job Again** {% icon galaxy-refresh %} button in the History to re-run {% tool [Volcano Plot](toolshed.g2.bx.psu.edu/repos/iuc/volcanoplot/volcanoplot/0.0.7) %} with the same parameters as before except:
>    - *"Points to label"*: `Input from file`
>        - *"File of labels"*: `volcano_genes`
>    - In *"Plot Options"*:
>     -   *"Label Boxes"*: `Yes`
{: .hands_on}

![Volcano plot labelling genes of interest](../../images/rna-seq-viz-with-volcanoplot/volcanoplot_custom_genes.png){: width="60%"}

> <question-title></question-title>
>
> 1. How many of the genes of interest are significant?
> 2. Which gene of interest is the most statistically significant?
>
>    > <solution-title></solution-title>
>    >
>    > 1. 29/31 are significant, the genes that are not in the grey area. 
>    > 2. The Egf gene is the most statistically significant as it is nearest the top of the plot.
>    >
>    {: .solution}
{: .question}


As in the previous plots, genes are coloured if they pass the thresholds for FDR and Log Fold Change. Here, all the genes of interest are significant (red or blue) except for two genes, Mcl1 and Gmfg. Gmfg, has an FDR just very slightly outside the significance threshold we used of 0.01 (0.0105). Mcl1 is the authors' gene of interest and they showed that while it's expression did increase at the protein level, it did not increase at the transcription level, as we can see here, suggesting it is regulated post-transcriptionally. 

> <tip-title>R code</tip-title>
>
> You can get the R code used to generate the plot under *Output Options* in the tool form. You can edit this code in R if you want to customise the plot. See the [Visualization of RNA-Seq results with Volcano Plot in R]({% link topics/transcriptomics/tutorials/rna-seq-viz-with-volcanoplot-r/tutorial.md %}) tutorial for how to do this.
>
{: .tip}

# Conclusion


In this tutorial we have seen how a volcano plot can be generated from RNA-seq data and used to quickly visualize significant genes.


topic: transcriptomics
tutorial: de-novo
title: De novo transcriptome reconstruction with RNA-Seq
difficulty: intermediate
time_estimation: 6h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 0b75b7b0ac00c21b4d7744f8d0f4506f
last_modified: 2026-05-15T22:28:17.595630
zenodo_link: https://zenodo.org/records/583140
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/de-novo/tutorial.html

content:
The data provided here are part of a Galaxy tutorial that analyzes RNA-seq data from a study published by *Wu et al.* in 2014 [DOI:10.1101/gr.164830.113](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4248312/). The goal of this study was to investigate "the dynamics of occupancy and the role in gene regulation of the transcription factor Tal1, a critical regulator of hematopoiesis, at multiple stages of hematopoietic differentiation." To this end, RNA-seq libraries were constructed from multiple mouse cell types including G1E - a GATA-null immortalized cell line derived from targeted disruption of GATA-1 in mouse embryonic stem cells - and megakaryocytes. This RNA-seq data was used to determine differential gene expression between G1E and megakaryocytes and later correlated with Tal1 occupancy. This dataset (GEO Accession: GSE51338) consists of biological replicate, paired-end, poly(A) selected RNA-seq libraries. Because of the long processing time for the large original files, we have downsampled the original raw data files to include only reads that align to chromosome 19 and a subset of interesting genomic loci identified by Wu *et al*.

# Analysis strategy

The goal of this exercise is to identify what transcripts are present in the G1E and megakaryocyte cellular states and which transcripts are differentially expressed between the two states. We will use a *de novo* transcript reconstruction strategy to infer transcript structures from the mapped reads in the absence of the actual annotated transcript structures. This will allow us to identify novel transcripts and novel isoforms of known transcripts, as well as identify differentially expressed transcripts.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Data upload

Due to the large size of this dataset, we have downsampled it to only include reads mapping to chromosome 19 and certain loci with relevance to hematopoeisis. This data is available at [`Zenodo`](https://zenodo.org/record/583140), where you can find the forward and reverse reads corresponding to replicate RNA-seq libraries from G1E and megakaryocyte cells and an annotation file of RefSeq transcripts we will use to generate our transcriptome database.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Create a new history for this RNA-seq exercise
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 2. Open the data upload manager (Get Data -> Upload file)
> 3. Copy and paste the links for the reads and annotation file
> 4. Select **Paste/Fetch Data**
> 5. Paste the link(s) into the text field
> 6. Change the datatype of the read files to **fastqsanger**
> 7. Change the datatype of the annotation file to **gtf** and assign the Genome as **mm10**
> 8. Press **Start**
> 9. Rename the files in your history to retain just the necessary information (*e.g.* "G1E R1 forward reads")
>
>
> 10. Import the files from [Zenodo]({{ page.zenodo_link }})
>
>     ```
>     https://zenodo.org/record/583140/files/G1E_rep1_forward_read_%28SRR549355_1%29
>     https://zenodo.org/record/583140/files/G1E_rep1_reverse_read_%28SRR549355_2%29
>     https://zenodo.org/record/583140/files/G1E_rep2_forward_read_%28SRR549356_1%29
>     https://zenodo.org/record/583140/files/G1E_rep2_reverse_read_%28SRR549356_2%29
>     https://zenodo.org/record/583140/files/Megakaryocyte_rep1_forward_read_%28SRR549357_1%29
>     https://zenodo.org/record/583140/files/Megakaryocyte_rep1_reverse_read_%28SRR549357_2%29
>     https://zenodo.org/record/583140/files/Megakaryocyte_rep2_forward_read_%28SRR549358_1%29
>     https://zenodo.org/record/583140/files/Megakaryocyte_rep2_reverse_read_%28SRR549358_2%29
>     https://zenodo.org/record/583140/files/RefSeq_reference_GTF_%28DSv2%29
>     ```
>
>     You will need to fetch the link to the annotation file yourself ;)
>
>     {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

## Quality control

For quality control, we use similar tools as described in [NGS-QC tutorial]({{site.baseurl}}/topics/sequence-analysis/): [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) and [Trimmomatic](http://www.usadellab.org/cms/?page=trimmomatic).

> <hands-on-title>Quality control</hands-on-title>
>
> 1. **FastQC** {% icon tool %}: Run `FastQC` on the forward and reverse read files to assess the quality of the reads.
>
>    > <question-title></question-title>
>    >
>    > 1. What is the read length?
>    > 2. Is there anything interesting about the quality of the base calls based on the position in the reads?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The read length is 99 bp
>    > > 2. The quality of base calls declines throughout a sequencing run.
>    > {: .solution }
>    {: .question}
>
> 2. **Trimmomatic** {% icon tool %}: Trim off the low quality bases from the ends of the reads to increase mapping efficiency. Run `Trimmomatic` on each pair of forward and reverse reads with the following settings:
>
>    - *"Single-end or paired-end reads?"*: `Paired-end (two separate input files)`
>       - {% icon param-file %} *"Input FASTQ file (R1/first of pair)"*: `G1E_rep1 forward read`
>       - {% icon param-file %} *"Input FASTQ file (R2/second of pair)"*: `G1E_rep1 reverse read`
>    - *"Perform initial ILLUMINACLIP step?"*: `No`
>
> 3. **FastQC** {% icon tool %}: Re-run `FastQC` on trimmed reads and inspect the differences.
>
>    > <question-title></question-title>
>    >
>    > 1. What is the read length?
>    > 2. Is there anything interesting about the quality of the base calls based on the position in the reads?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The read lengths range from 1 to 99 bp after trimming
>    > > 2. The average quality of base calls does not drop off as sharply at the 3' ends of reads.
>    > {: .solution }
>    {: .question}
> ![Before and after trimming comparison](../../images/transcriptomics_images/BeforeAndAfterTrimming.png)
>
> 4. **Trimmomatic** {% icon tool %}: Run `Trimmomatic` on the remaining forward/reverse read pairs with the same parameters.
>
{: .hands_on}

Now that we have trimmed our reads and are fortunate that there is a reference genome assembly for mouse, we will align our trimmed reads to the genome.

> <comment-title></comment-title>
>
> Instead of running a single tool multiple times on all your data, would you rather run a single tool on multiple datasets at once? Check out the [dataset collections](https://galaxyproject.org/tutorials/collections/) feature of Galaxy!
{: .comment}

# Mapping

To make sense of the reads, their positions within mouse genome must be determined. This process is known as aligning or 'mapping' the reads to the reference genome.

> <comment-title></comment-title>
>
> Do you want to learn more about the principles behind mapping? Follow our [training]({{site.baseurl}}/topics/sequence-analysis/)
{: .comment}

In the case of a eukaryotic transcriptome, most reads originate from processed mRNAs lacking introns. Therefore, they cannot be simply mapped back to the genome as we normally do for reads derived from DNA sequences. Instead, the reads must be separated into two categories:

- Reads contained within mature exons - these align perfectly to the reference genome
- Reads that span splice junctions in the mature mRNA - these align with gaps to the reference genome

Spliced mappers have been developed to efficiently map transcript-derived reads against genomes. [`HISAT`](https://ccb.jhu.edu/software/hisat2/index.shtml) is an accurate and fast tool for mapping spliced reads to a genome. Another popular spliced aligner is [`TopHat`](https://ccb.jhu.edu/software/tophat/index.shtml), but we will be using `HISAT` in this tutorial.

> <comment-title></comment-title>
> As it is sometimes quite difficult to determine which settings correspond to those of other programs, the following table might be helpful to identify the library type:
>
> Library type | **Infer Experiment** | **TopHat** | **HISAT** | **htseq-count** | **featureCounts**
> --- | --- | --- | --- | --- | ---
> PE | 1++,1--,2+-,2-+ | FR Second Strand | FR | yes | 1
> PE | 1+-,1-+,2++,2-- | FR First Strand | RF | reverse | 2
> SE | ++,-- | FR Second Strand | F | yes | 1
> SE | +-,-+ | FR First Strand | R | reverse | 2
> SE,PE | undecided | FR Unstranded | default | no | 0
>
{: .comment}

> <hands-on-title>Spliced mapping</hands-on-title>
>
> 1. **HISAT2** {% icon tool %}: Run `HISAT2` on one forward/reverse read pair and modify the following settings:
>    - *"Source for the reference genome"*: `Use a built-in genome`
>       - {% icon param-file %} *"Select a reference genome"*: `Mouse (Mus Musculus): mm10`
>    - *"Single-end or paired-end reads?"*: `Paired-end`
>       - {% icon param-file %} *"FASTA/Q file #1"*: Trimmomatic on G1E_rep1 forward read (R1 paired)
>       - {% icon param-file %} *"FASTA/Q file #2"*: Trimmomatic on G1E_rep1 reverse read (R2 paired)
>       - *"Specify strand information"*: `Forward(FR)`
>    - *"Advanced options"*
>       - *"Spliced alignment options"*
>           - *"Penalty for non-canonical splice sites"*: '3'
>           - *"Penalty function for long introns with canonical splice sites"*: 'Constant [f(x) = B]''
>           - *"Constant term (B)"*: '0.0'
>           - *"Penalty function for long introns with non-canonical splice sites"*: 'Constant [f(x) = B]''
>           - *"Transcriptome assembly reporting"*: 'Report alignments tailored for transcript assemblers including StringTie'
>
> 2. **HISAT2** {% icon tool %}: Run `HISAT2` on the remaining forward/reverse read pairs with the same parameters.
>
{: .hands_on}

# De novo transcript reconstruction
Now that we have mapped our reads to the mouse genome with `HISAT`, we want to determine transcript structures that are represented by the aligned reads. This is called *de novo* transcriptome reconstruction. This unbiased approach permits the comprehensive identification of all transcripts present in a sample, including annotated genes, novel isoforms of annotated genes, and novel genes. While common gene/transcript databases are quite large, they are not comprehensive, and the *de novo* transcriptome reconstruction approach ensures complete transcriptome(s) identification from the experimental samples. The leading tool for transcript reconstruction is `Stringtie`. Here, we will use `Stringtie` to predict transcript structures based on the reads aligned by `HISAT`.

> <hands-on-title>Transcriptome reconstruction</hands-on-title>
>
> 1. **Stringtie** {% icon tool %}: Run `Stringtie` on the `HISAT2` alignments using the default parameters.
>    - Use batch mode to run all four samples from one tool form.
>    - {% icon param-file %} *"Specify strand information"*: `Forward (FR)`
>
{: .hands_on}

# Transcriptome assembly

We just generated four transcriptomes with `Stringtie` representing each of the four RNA-seq libraries we are analyzing. Since these were generated in the absence of a reference transcriptome, and we ultimately would like to know what transcript structure corresponds to which annotated transcript (if any), we have to make a **transcriptome database**. We will use the tool `Stringtie - Merge` to combine redundant transcript structures across the four samples and the RefSeq reference. Once we have merged our transcript structures, we will use `GFFcompare` to annotate the transcripts of our newly created transcriptome so we know the relationship of each transcript to the RefSeq reference.

> <hands-on-title>Transcriptome assembly</hands-on-title>
>
> 1. **Stringtie-merge** {% icon tool %}: Run `Stringtie-merge` on the `Stringtie` assembled transcripts along with the RefSeq annotation file we imported earlier.
>    - {% icon param-file %} *"Transcripts"*: `all four `Stringtie` assemblies`
>    - {% icon param-file %} *"Reference annotation to include in the merging"*: `RefSeq_reference_GTF`
>
> 2. **GFFCompare** {% icon tool %}: Run `GFFCompare` on the `Stringtie-merge` generated transcriptome along with the RefSeq annotation file.
>    - {% icon param-file %} *"GTF inputs for comparison"*: `output of Stringtie-merge`
>    - *"Use Reference Annotation"*: `Yes`
>       - *"Choose the source for the reference annotation"*: `History`
>           - {% icon param-file %} *"Reference Annotation"*: `RefSeq_reference_GTF`
>    - *"Use Sequence Data"*: `Yes`
>       - *"Choose the source for the reference list"*: `Locally cached`
>           - *"Using reference genome"*: 'Mouse (Mus Musculus): mm10'
>
{: .hands_on}

> <comment-title>Note: Transcript categorization used by `GFFcompare`</comment-title>
>
> |**Class code** | **Transcript category**|
> |:---:|:---|
> |= | Annotated in reference|
> |j | Novel isoform of reference|
> |u | Intergenic|
> |x | Anti-sense|
> |r | Repetitive|
> |c | Contained in exon of reference|
> |s | Anti-sense spliced intronic|
> |e | Single exon transfrag overlapping a reference exon and at least 10 bp of a reference intron, indicating a possible pre-mRNA fragment.|
> |i | A transfrag falling entirely within a reference intron|
> |o | Generic exonic overlap with a reference transcript|
> |p | Possible polymerase run-on fragment (within 2Kbases of a reference transcript)|
{: .comment}


# Analysis of the differential gene expression

We just generated a transriptome database that represents the transcripts present in the G1E and megakaryocytes samples. This database provides the location of our transcripts with non-redundant identifiers, as well as information regarding the origin of the transcript.

We now want to identify which transcripts are differentially expressed between the G1E and megakaryocyte cellular states. To do this we will implement a counting approach using `FeatureCounts` to count reads per transcript. Then we will provide this information to `DESeq2` to generate normalized transcript counts (abundance estimates) and significance testing for differential expression.

## Count the number of reads per transcript

To compare the abundance of transcripts between different cellular states, the first essential step is to quantify the number of reads per transcript. [`FeatureCounts`](http://bioinf.wehi.edu.au/featureCounts/ ) is one of the most popular tools for counting reads in genomic features. In our case, we'll be using `FeatureCounts` to count reads aligning in exons of our `GFFCompare` generated transcriptome database.

The recommended mode is "union", which counts overlaps even if a read only shares parts of its sequence with a genomic feature and disregards reads that overlap more than one feature.

> <hands-on-title>Counting the number of reads per transcript</hands-on-title>
>
> 1. **FeatureCounts** {% icon tool %}: Run `FeatureCounts` on the aligned reads (`HISAT2` output) using the `GFFCompare` transcriptome database as the annotation file.
>
>    - {% icon param-file %} *"Alignment file"*: `four HISAT2 aligned read files`
>    - *"Specify strand information"*: `Stranded (Forward)`
>    - {% icon param-file %} *"Gene annotation file"*: `in your history`
>       - *"Gene annotation file"*: `the annotated transcripts GTF file output by GFFCompare`
>    - *"Options for paired-end reads"*
>       - *"Count fragments instead of reads"*: `Enabled; fragments (or templates) wil be counted instead of reads`
>    - *"Advanced options"*
>       - *"GFF gene identifier"*: `transcript_id`
>
{: .hands_on}

## Perform differential gene expression testing

Transcript expression is estimated from read counts, and attempts are made to correct for variability in measurements using replicates. This is absolutely essential to obtaining accurate results. We recommend having at least two biological replicates.

[`DESeq2`](https://bioconductor.org/packages/release/bioc/html/DESeq2.html) is a great tool for differential gene expression analysis. It accepts read counts produced by `FeatureCounts` and applies size factor normalization:

- Computation for each gene of the geometric mean of read counts across all samples
- Division of every gene count by the geometric mean
- Use of the median of these ratios as sample's size factor for normalization

> <hands-on-title></hands-on-title>
>
> 1. **DESeq2** {% icon tool %}: Run `DESeq2` with the following parameters:
>    - *"1: Factor"*
>       - *"1: Factor level"*: `G1E`
>          - {% icon param-file %} *"Counts file(s)"*: `featureCount files corresponding to the two G1E replicates`
>       - *"2: Factor level"*: `Mega`
>          - {% icon param-file %} *"Counts file(s)"*: `featureCount files corresponding to the two Mega replicates`
>
>       > <comment-title></comment-title>
>       >
>       > You can select several files by holding down the CTRL (or COMMAND) key and clicking on the desired files
>       {: .comment}
>    - *"Visualising the analysis results"*: Yes
>    - *"Output normalized counts table"*: Yes
>
{: .hands_on}

The first output of `DESeq2` is a tabular file. The columns are:

1.	Gene identifiers
2.	Mean normalized counts, averaged over all samples from both conditions
3.	Logarithm (base 2) of the fold change (the values correspond to up- or downregulation relative to the condition listed as Factor level 1)
4.	Standard error estimate for the log2 fold change estimate
5.	[Wald](http://data.princeton.edu/wws509/notes/c2s3.html) statistic
6.	*p*-value for the statistical significance of this change
7.	*p*-value adjusted for multiple testing with the Benjamini-Hochberg procedure which controls false discovery rate ([FDR](http://www.biostathandbook.com/multiplecomparisons.html))


> <hands-on-title></hands-on-title>
>
>1. **Filter** {% icon tool %}: Run `Filter` to extract genes with a significant change in gene expression (adjusted *p*-value less than 0.05) between treated and untreated samples
>
>    > <question-title></question-title>
>    >
>    > How many transcripts have a significant change in expression between these conditions?
>    >
>    > > <solution-title></solution-title>
>    > > To filter, use "c7<0.05". And we get 249 transcripts with a significant change in gene expression between the G1E and megakaryocyte cellular states.
>    > {: .solution }
>    {: .question}
>
> 2. **Filter** {% icon tool %}: Determine how many transcripts are up or down regulated in the G1E state.
>
>    > <comment-title></comment-title>
>    > Rename your datasets for the downstream analyses
>    {: .comment}
>
>    > <question-title></question-title>
>    >
>    > Are there more upregulated or downregulated genes in the treated samples?
>    >
>    > > <solution-title></solution-title>
>    > > To obtain the up-regulated genes in the G1E state, we filter the previously generated file (with the significant change in transcript expression) with the expression "c3>0" (the log2 fold changes must be greater than 0). We obtain 102  genes (40.9% of the genes with a significant change in gene expression). For the down-regulated genes in the G1E state, we did the inverse and we find 149 transcripts (59% of the genes with a significant change in transcript expression).
>    > {: .solution }
>    {: .question}
{: .hands_on}

In addition to the list of genes, `DESeq2` outputs a graphical summary of the results, useful to evaluate the quality of the experiment:

1. Histogram of *p*-values for all tests

    ![DESeq2 histogram]({% link topics/transcriptomics/images/transcriptomics_images/Deseq2_histogram3.png %})

2. [MA plot](https://en.wikipedia.org/wiki/MA_plot): global view of the relationship between the expression change of conditions (log ratios, M), the average expression strength of the genes (average mean, A), and the ability of the algorithm to detect differential gene expression. The genes that passed the significance threshold (adjusted p-value < 0.1) are colored in red.

    ![MA plot]({% link topics/transcriptomics/images/transcriptomics_images/Deseq2_MAplot3.png %})

3. Principal Component Analysis ([PCA](https://en.wikipedia.org/wiki/Principal_component_analysis)) and the first two axes

    ![Principal component analysis output]({% link topics/transcriptomics/images/transcriptomics_images/Deseq2_PCA3.png %})

    Each replicate is plotted as an individual data point. This type of plot is useful for visualizing the overall effect of experimental covariates and batch effects.

4. Heatmap of sample-to-sample distance matrix: overview over similarities and dissimilarities between samples

    ![Heatmap of sample-to-sample distance matrix]({% link topics/transcriptomics/images/transcriptomics_images/Deseq2_heatmap3.png %})

5. Dispersion estimates: gene-wise estimates (black), the fitted values (red), and the final maximum a posteriori estimates used in testing (blue)

    ![Dispersion estimates]({% link topics/transcriptomics/images/transcriptomics_images/Deseq2_dispersion3.png %})

    This dispersion plot is typical, with the final estimates shrunk from the gene-wise estimates towards the fitted estimates. Some gene-wise estimates are flagged as outliers and not shrunk towards the fitted value. The amount of shrinkage can be more or less than seen here, depending on the sample size, the number of coefficients, the row mean and the variability of the gene-wise estimates.


For more information about `DESeq2` and its outputs, you can have a look at [`DESeq2` documentation](https://www.bioconductor.org/packages/release/bioc/manuals/DESeq2/man/DESeq2.pdf).

# Visualization
Now that we have a list of transcript expression levels and their differential expression levels, it is time to visually inspect our transcript structures and the reads they were predicted from. It is a good practice to visually inspect (and present) loci with transcripts of interest. Fortunately, there is a built-in genome browser in Galaxy, **Trackster**, that make this task simple (and even fun!).

In this last section, we will convert our aligned read data from BAM format to bigWig format to simplify observing where our stranded RNA-seq data aligned to. We'll then initiate a session on Trackster, load it with our data, and visually inspect our interesting loci.

> <hands-on-title>Converting aligned read files to bigWig format</hands-on-title>
>
> 1. **bamCoverage** {% icon tool %}: Run `bamCoverage` on all four aligned read files (`HISAT2` output) with the following parameters:
>    - *"Bin size in bases"*: '1'
>    - *"Effective genome size"*: 'mm9 (2150570000)''
>    - *"Advanced options"*
>       - *"Only include reads originating from fragments from the forward or reverse strand"*: 'forward'
>
> 2. **Rename** {% icon tool %}: Rename the outputs to reflect the origin of the reads and that they represent the reads mapping to the PLUS strand.
>
> 3. **bamCoverage** {% icon tool %}: Repeat Step 1 except changing the following parameter:
>    - *"Only include reads originating from fragments from the forward or reverse strand"*: 'reverse'
>
> 4. **Rename** {% icon tool %}: Rename the outputs to reflect the origin of the reads and that they represent the reads mapping to the MINUS strand.
{: .hands_on}

> <hands-on-title>Trackster based visualization</hands-on-title>
>
> 1. **Viz** {% icon tool %}: On the center console at the top of the Galaxy interface, choose " Visualization" -> "New track browser"
>    - Name your visualization someting descriptive under "Browser name:"
>    - Choose "Mouse Dec. 2011 (GRCm38/mm10) (mm10)" as the "Reference genome build (dbkey)
>    - Click "Create" to initiate your Trackster session
> ![Trackster opening window](../../images/transcriptomics_images/Trackster_opening_window.png)
>
> 2. **Viz** {% icon tool %}: Click "Add datasets to visualization"
>    - Select the "RefSeq GTF mm10" file
>    - Select the output files from `Stringtie`
>    - Select the output file from `GFFCompare`
>    - Select the output files from `bamCoverage`
>
> 3. {% icon tool %}: Using the grey labels on the left side of each track, drag and arrange the track order to your preference.
>
> 4. {% icon tool %}: Hover over the grey label on the left side of the "RefSeq GTF mm10" track and click the "Edit settings" icon.
>    - Adjust the block color to blue (#0000ff) and antisense strand color to red (#ff0000)
>
> 5. {% icon tool %}: Repeat the previous step on the output files from `StringTie` and `GFFCompare`.
>
> 6. {% icon tool %}: Hover over the grey label on the left side of the "G1E R1 plus" track and click the "Edit settings" icon.
>    - Adjust the color to blue (#0000ff)
>
> 7. {% icon tool %}: Repeat the previous step on the other three bigWig files representing the plus strand.
>
> 8. {% icon tool %}: Hover over the grey label on the left side of the "G1E R1 minus" track and click the "Edit settings" icon.
>    - Adjust the color to red (#ff0000)
>
> 9. {% icon tool %}: Repeat the previous step on the other three bigWig files representing the minus strand.
>
> 10. {% icon tool %}: Adjust the track height of the bigWig files to be consistent for each set of plus strand and minus strand tracks.
> ![Adjusting the track height](../../images/transcriptomics_images/Hoxb13_locus_screenshot.png)
> 11. {% icon tool %}: Direct Trackster to the coordinates: chr11:96191452-96206029, what do you see?
>
>    > <question-title></question-title>
>    > what do you see?
>    >
>    > > <solution-title></solution-title>
>    > > 1. There are two clusters of transcripts that are exclusively expressed in the G1E background
>    > > 2. The left-most transcript is the Hoxb13 transcript
>    > > 3. The center cluster of transcripts are not present in the RefSeq annotation and are determined by `GFFCompare` to be "u" and "x"
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

# Conclusion


In this tutorial, we have analyzed RNA sequencing data to extract useful information, such as which genes are expressed in the G1E and megakaryocyte cellular states and which of these genes are differentially expressed between the two cellular states. In addition, we identified unannotated genes that are expressed in a cell-state dependent manner and at a locus with relevance to differentiation and development. To identify these transcripts, we analyzed RNA sequence datasets using a de novo transcriptome reconstruction RNA-seq data analysis approach. This approach can be summed up with the following scheme:

![RNAseq de novo tutorial workflow](../../images/transcriptomics_images/schematic_for_RNAseq_de_novo_tutorial.png)


topic: transcriptomics
tutorial: mirna-target-finder
title: Whole transcriptome analysis of Arabidopsis thaliana
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: []
content_hash: e43ee27f3f1ad9c9ee2ce76f5865a33d
last_modified: 2026-05-27T09:01:49.977553
zenodo_link: https://zenodo.org/record/4710649
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/transcriptomics/tutorials/mirna-target-finder/tutorial.html

content:
As sessile organisms, the survival of plants under adverse environmental conditions depends, to a large extent, on their ability to perceive stress stimuli and respond appropriately to counteract the potentially damaging effects. Coordination of phytohormones and reactive oxygen species are considered a key element for enhancing stress resistance, allowing fine-tuning of gene expression in response to environmental changes ({% cite PlanasRiverola2019 %}, {% cite Ivashuta2011 %}). These molecules constitute complex signalling networks, endowing with the ability to respond to a variable natural environment.

Brassinosteroids (BRs) are a group of plant steroid hormones essential for plant growth and development, as well as for controlling abiotic and biotic stress. Structurally, BRs are polyhydroxylated sterol derivatives with close similarity to animal hormones (Figure 1). This group of phytohormones is comprised of around 60 different compounds, of which brassinolide (BL), 24-epibrassinolide (EBR), and 28-homobrassinolide (HBR) are considered the most bioactive.

![fig1:Esteroid hormones]({% link topics/transcriptomics/images/transcriptomics_images/brassinosteroids.png %} "Structure of various plant and animal steroid hormones.")

Several recent studies suggest that the BR-mediated gene regulatory networks have the potential to reshape the future of agriculture, not only by alleviating the antagonistic effect diverse abiotic stress conditios, such as drought, but also by enhancing plant growth and yield. For instance, in tomato (_Solanum lycopersicum_), EBR treatment enhances drought tolerance, improving photosynthetic capacity, leaf water status, and antioxidant defense ({% cite Wang2018 %}). In pepper (_Capsicum annuum_), BL treatment increased the efficiency of light utilization under drought ({% cite Hu2013 %}). Gram (_Cicer arietinum_) plants exposed to drought stress and treated with BL showed significant increases in weight ({% cite Anwar2018 %}). However, the mechanisms of BRs action in enhancing plant tolerance to abiotic stresses still remain largely unknown.

MicroRNAs (miRNAs), mainly 20–22 nucleotide small RNAs (sRNAs), are characterized for regulating gene expression at the post-transcriptional level. miRNAs are distinguished from other sRNAs by being generated from precursor harboring an imperfect stem‐loop structure. Unlike in animals, the pre-processing of plant miRNA occurs in the nucleus (Figure 2). The pre-miRNAs are then exported to the cytoplasm after methylation and incorporated into the Argonaute 1 protein to form RISC (RNA-induced silencing complex). The miRNA itself does not have the ability to cleave mRNAs or interfere with translation, but it plays a role in scanning the appropriate target.


![fig2:miRNA biosynthesis](../../images/transcriptomics_images/plant_mirna_synthesis.png "Plant miRNA biosynthesis, homeostasis and mechanisms of action ({% cite Wang2019 %}).")


miRNAs have been found to be important regulators of many physiological processes, such as stress and hormonal responses. Four factors justify the miRNAs to be considered as master regulators of the plant response to the surrounding environment:

- Multiple miRNA genes are regulated under given environmental conditions
- Computational predictions estimate that each miRNA regulates hundreds of genes
- The majority of plant miRNAs regulate genes encoding for transcription factors (TFs)
- Targets include not only mRNAs but also long noncoding RNAs (lncRNAs)

In plants, miRNAs can silence targets through RNA degradation as well as translational repression pathways, and unlike animals, a large proportion of miRNA and their targets have less than four mismatches. This feature has been exploited for developing miRNAs target prediction tools, providing an efficient approach to elucidate the miRNA-mediated regulatory networks, which can contribute to biotechnological solutions to improve crops productivity.

In this tutorial, inspired by {% cite Park2020 %}, we aim to explore the interplay between brassinosteroids and the miRNA-gene silencing pathway, considered one of the most versatile regulatory mechanisms in response to stressful situations in plants.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Experimental design

The main objective of this training is to identify potential targets of miRNAs whose expression is induced by brassinosteroids. Our starting hypothesis is that there must be brassinosteroid-induced miRNAs that have high sequence complementarity with mRNAs whose expression is inhibited in the presence of these hormones (Figure 3).

![fig3:Experimental design](../../images/transcriptomics_images/miRNA_pipeline.png "Experimental design")

# Background on data

The datasets to be used in this training can be classified into three groups: miRNA reads, mRNA reads and additional datasets.

## miRNA reads

The miRNA datasets consist of six FASTQ files, obtained by using the Illumina GAxII sequencing platform. The plant samples were obtained from wild-type Ws-2 seedlings treated with mock or 1 μM EBR for 90 min before harvest. The original datasets are available in the NCBI SRA database, with the accession number [SRP258575](https://www.ncbi.nlm.nih.gov/sra?term=SRP258575). As in the previous case, for this tutorial, we will use a reduced version of the data.


## mRNA reads

The mRNA datasets consist of four FASTQ files, generated through the Illumina HiSeq 2000 sequencing system. The samples were obtained from wild-type Columbia (Col-0) seedlings treated with mock or 100 nM BL for 4 hours. The original datasets are available in the NCBI SRA database, with the accession number [SRP032274](https://www.ncbi.nlm.nih.gov/sra?term=SRP032274). For this tutorial, subsets from the original data were generated in order to reduce the analysis run time.


## Additional datasets

In addition to the RNA-Seq reads obtained from the NCBI database, we will use datasets from two sources:

- [AtRTD2](https://ics.hutton.ac.uk/atRTD/) is a high-quality transcript reference dataset developed to exploit the accuracy of transcript quantification tools such as __Salmon__ and __Kallisto__ in analyzing _Arabidopsis_ RNA-Seq data.
- [PmiREN](http://pmiren.com/) is a comprehensive functional plant miRNA database that includes more than 20,000 annotated miRNAs diverse plant species.

# Get data

The first step of our analysis consists of retrieving the miRNA-Seq datasets from Zenodo and organizing them into collections.

> <hands-on-title>Retrieve miRNA-Seq and mRNA-Seq datasets</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click on __Rule-based__ tab
>    - *"Upload data as"*: `Collection(s)`
>    - Copy the following tabular data, paste it into the textbox and press <kbd>Build</kbd>
>
>      ```
>      SRR11611349	Control miRNA	https://zenodo.org/record/4710649/files/SRR11611349_MIRNASEQ_CTL.fastqsanger.gz	fastqsanger.gz
>      SRR11611350	Control miRNA	https://zenodo.org/record/4710649/files/SRR11611350_MIRNASEQ_CTL.fastqsanger.gz	fastqsanger.gz
>      SRR11611351	Control miRNA	https://zenodo.org/record/4710649/files/SRR11611351.MIRNASEQ_CTLfastqsanger.gz	fastqsanger.gz
>      SRR11611352	BR treated miRNA	https://zenodo.org/record/4710649/files/SRR11611352_MIRNASEQ_BL.fastqsanger.gz	fastqsanger.gz
>      SRR11611353	BR treated miRNA	https://zenodo.org/record/4710649/files/SRR11611353_MIRNASEQ_BL.fastqsanger.gz	fastqsanger.gz
>      SRR11611354	BR treated miRNA	https://zenodo.org/record/4710649/files/SRR11611354_MIRNASEQ_BL.fastqsanger.gz	fastqsanger.gz
>      SRR1019436	Control mRNA	https://zenodo.org/record/4710649/files/SRR1019436_RNASEQ_CTL.fastqsanger.gz	fastqsanger.gz
>      SRR1019437	Control mRNA	https://zenodo.org/record/4710649/files/SRR1019437_RNASEQ_CTL.fastqsanger.gz	fastqsanger.gz
>      SRR1019438	BR treated mRNA	https://zenodo.org/record/4710649/files/SRR1019438_RNASEQ_BL.fastqsanger.gz	fastqsanger.gz
>      SRR1019439	BR treated mRNA	https://zenodo.org/record/4710649/files/SRR1019439_RNASEQ_BL.fastqsanger.gz	fastqsanger.gz
>      ```
>
>    - From **Rules** menu select `Add / Modify Column Definitions`
>       - Click `Add Definition` button and select `List Identifier(s)`: column `A`
>
>         > <tip-title>Can't find <i>List Identifier</i>?</tip-title>
>         > Then you've chosen to upload as a 'dataset' and not a 'collection'. Close the upload menu, and restart the process, making sure you check *Upload data as*: **Collection(s)**
>         {: .tip}
>
>       - Click `Add Definition` button and select `Collection Name`: column `B`
>       - Click `Add Definition` button and select `URL`: column `C`
>       - Click `Add Definition` button and select `Type`: column `D`
>
>    - Click `Apply` and press <kbd>Upload</kbd>
>
> {% snippet faqs/galaxy/datasets_add_tag.md type="name" %}
{: .hands_on}

Next we will retrieve the remaining datasets.

> <hands-on-title>Retrieve the additional datasets</hands-on-title>
>
> 1. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - *"Upload data as"*: `Datasets`
>    - Once again, copy the tabular data, paste it into the textbox and press <kbd>Build</kbd>
>
>      ```
>      annotation_AtRTD2.gtf	https://zenodo.org/record/4710649/files/annotation_AtRTD2_19April2016.gtf.gz
>      transcriptome.fasta	https://zenodo.org/record/4710649/files/transcriptome_AtRTD2_12April2016.fasta.gz
>      star_miRNA_seq.fasta	https://zenodo.org/record/4710649/files/star_miRNA_seq.fasta
>      mature_miRNA_AT.fasta	https://zenodo.org/record/4710649/files/mature_miRNA_AT.fasta
>      miRNA_stem-loop_seq.fasta	https://zenodo.org/record/4710649/files/miRNA_stem-loop_seq.fasta
>      ```
>
>    - From **Rules** menu select `Add / Modify Column Definitions`
>       - Click `Add Definition` button and select `Name`: column `A`
>       - Click `Add Definition` button and select `URL`: column `B`
>    - Click `Apply` and press <kbd>Upload</kbd>
>
>
{: .hands_on}

> <details-title>Dataset subsampling</details-title>
>
> As indicated above, for this tutorial the depth of the samples was reduced in order to speed up the time needed to carry out the analysis. This was done as follows:
>
> > <hands-on-title>Dataset subsampling</hands-on-title>
> >
> > 1. {% tool [Sub-sample sequences files ](toolshed.g2.bx.psu.edu/repos/peterjc/sample_seqs/sample_seqs/0.2.6) %} with the following parameters:
> >    - {% icon param-files %} *"Multiple datasets"*: Each of the fastq files
> >    - *"Subsampling approach"*: `Take every N-th sequence (or pair e.g. every fifth sequence)`
> >    - *"N"*: `100`
> {: .hands_on}
>
> In this way, we will only take 1% of reads at a random sampling rate.
{: .details}


# miRNA data analysis

Once we have imported the data, we can begin to study how brassinosteroid exposure alters gene expression patterns.

## Quality assessment of miRNA reads

Due to technical limitations, sequencing is considered an error-prone process. In Illumina sequencing platforms, substitution type miscalls are the dominant source of errors, which can lead to inconsistent results. Another factor that can interfere with our analyses is the presence of adapter contaminants, which can result in an increased number of unaligned reads, since the adapter sequences are synthetic and do not occur in the genomic sequence.

Sequence quality control is therefore an essential first step in your analysis. We will use two popular tools for evaluating the quality of our raw reads: __Falco__ and __MultiQC__.

> <comment-title></comment-title>
> In order to visualize the data from both collections together in the __MultiQC__ tool, it will be necessary to combine the results generated by __Falco__.
> For more information on the topic of quality control, please see our [dedicated training materials]({% link topics/sequence-analysis/tutorials/quality-control/tutorial.md %}).
{: .comment}

> <hands-on-title>Initial quality check</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Control miRNA`
> 2. Rename the outputs as `Falco unprocessed control miRNA: RawData` and `Falco unprocessed control miRNA: Webpage`
> 3. Repeat the previous steps with the `BR treated miRNA` collection.
> 4. {% tool [Merge Collections](__MERGE_COLLECTION__) %} with the following parameters:
>    - In *"Input collections"*:
>      - *"1.Input Collections"*: `Falco unprocessed control miRNA: RawData`
>      - *"2.Input Collections"*: `Falco unprocessed BR treated mRiNA: RawData`
> 5. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>        - In *"FastQC output"*:
>          - {% icon param-repeat %} *"Insert FastQC output"*
>            - {% icon param-collection %} *"FastQC output"*: select the output of __Merge Collection__ tool
>    - In *"Report title"*: `miRNA initial quality check`
> 6. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
{: .hands_on}


> <question-title></question-title>
>
> Based on the information provided by __MultiQC__, is it necessary to trimming/filtering the reads?
>
> > <solution-title></solution-title>
> >
> > The report generated by __MultiQC__ indicates that three quality parameters show values outside the recommended limits: per sequence G/C content, overrepresented sequences and adapter content.
> >
> > ![fig4:GC content](../../images/transcriptomics_images/miRNA_GC_content.png "Per sequence GC content of the miRNA samples.")
> > ![fig5:Overrepresented sequences](../../images/transcriptomics_images/miRNA_overepresented.png "Overrepresented sequences in miRNA samples.")
> > ![fig6:Adaptor content](../../images/transcriptomics_images/miRNA_adaptors.png "Adaptor content of miRNA samples.")
> >
> > Particularly remarkable is the content of adapters, which are up to 80% of the reads in some samples. Given the abundance of adapters, one would expect that if we eliminate this source of contamination, the rest of the parameters would show a noticeable improvement.
> >
> {: .solution}
{: .question}

To remove the adapters contamination, we will employ the __Trim Galore__ tool, a wrapper script around [__Cutadapt__](https://journal.embnet.org/index.php/embnetjournal/article/view/200) and __Falco__ which automates quality and adapter trimming.

> <hands-on-title>Trimming of adapter sequences</hands-on-title>
>
> 1. {% tool [Trim Galore!](toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.10+galaxy0) %} with the following parameters:
>    - *"Is this library paired- or single-end?"*: `Single-end`
>      -  {% icon param-collection %} *"Reads in FASTQ format"*: `Control miRNA`
>      - *"Adapter sequence to be trimmed"*: `Illumina small RNA adapters`
> 2. Rename the output collection as `Control miRNA trimmed`
> 3. Repeat the previous steps with the `BR treated miRNA` collection.
>
{: .hands_on}

Next, we will reassess the quality of the sequences to check if the adapters have been removed.

> <hands-on-title>Post-processing quality check</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Control miRNA trimmed`
> 2. Rename the outputs as `Falco processed control miRNA: RawData` and `Falco processed control miRNA: Webpage`
> 3. Repeat the previous steps with the `BR treated miRNA trimmed` collection.
> 4. {% tool [Merge Collections](__MERGE_COLLECTION__) %} with the following parameters:
>    - In *"Input collections"*:
>        - *"1.Input Collections"*: `Falco processed control miRNA: RawData`
>        - *"2.Input Collections"*: `Falco processed BR treated miRNA: RawData`
> 5. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>        - In *"FastQC output"*:
>           - {% icon param-repeat %} *"Insert FastQC output"*
>               - {% icon param-collection %} *"FastQC output"*: select the output of __Merge Collection__ tool
>    - *"Report title"*: `miRNA trimmed quality check`
> 6. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
{: .hands_on}

The evaluation of the report generated by __MultiQC__ after having processed the samples by __Trim Galore__ indicates that the G/C content has been successfully corrected, and that the adapter contamination has been eliminated. However, the samples still show a high degree of over-represented sequences (Figure 7).

![fig7:Overexpressed sequences](../../images/transcriptomics_images/miRNA_overepresented_processed.png "Report of overexpressed sequences in the miRNA reads")


> <question-title></question-title>
>
> What do you think could be the cause of the high number of over-represented sequences?
>
> > <solution-title></solution-title>
> >
> >  Two of the factors that may determine the abundance of overrepresented sequences are the existence of highly expressed miRNAs (Seco-Cervera et al. 2018), the existence of highly conserved sequence motifs within the miRNA ({% cite Glazov2008 %}), and contamination with foreign sequences.
> >
> {: .solution}
{: .question}


## miRNA quantification: **MiRDeep2**

Quantification of miRNAs requires to use two different tools:
- The __MiRDeep2 Mapper__ tool for preprocessing the reads.
- The __MiRDeep2 Quantifier__  tool for mapping the deep sequencing reads to predefined miRNA precursors and determining the expression of the corresponding miRNAs. It is carried out in two steps: firstly, the predefined mature miRNA sequences are mapped to the predefined precursors (optionally, predefined star sequences can be mapped to the precursors too). And second, the deep sequencing reads are mapped to the precursors.

> <hands-on-title>Quantification of miRNAs</hands-on-title>
>
> 1. {% tool [MiRDeep2 Mapper](toolshed.g2.bx.psu.edu/repos/rnateam/mirdeep2_mapper/rbc_mirdeep2_mapper/2.0.0.8.1) %} with the following parameters:
>    - {% icon param-collection %} *"Deep sequencing reads"*: `Control miRNA trimmed`
>    - *"Remove reads with non-standard nucleotides"*: `Yes`
>    - *"Clip 3' Adapter Sequence"*: `Don't Clip`
>    - *"Collapse reads and/or Map"*: `Collapse`
> 2. Rename the output as `Collapsed control miRNA`
> 3. Repeat the previous stages by providing `BR treated miRNA trimmed`as input, and rename it as `Collapsed BR treated miRNA`.
>
> 4. {% tool [MiRDeep2 Quantifier](toolshed.g2.bx.psu.edu/repos/rnateam/mirdeep2_quantifier/rbc_mirdeep2_quantifier/2.0.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collapsed deep sequencing reads"*: `Collapsed control miRNA`
>    - {% icon param-file %} *"Precursor sequences"*: `miRNA_stem-loop_seq.fasta`
>    - {% icon param-file %} *"Mature miRNA sequences"*: `mature_miRNA_AT.fasta`
>    - {% icon param-file %} *"Star sequences"*: `star_miRNA_seq.fasta`
> 5. Rename the outputs as `MiRDeep2 control miRNA` and `MiRDeep2 control miRNA (html)`.
> 6. Repeat the fourth step by providing `Collapsed BR treated miRNA` as input, and rename the outputs as `MiRDeep2 BR treated miRNA` and `MiRDeep2 BR treated miRNA (html)`
>
{: .hands_on}

To use the outputs generated by __MiRDeep2 Quantifier__ in the differential expression analysis, it is necessary to modify the datasets.

> <hands-on-title>Edition of MiRDeep2 Quantifier outputs</hands-on-title>
>
> 1. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1,c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-collection %} *"From"*: `MiRDeep2 control miRNA`
> 2. Rename the output as `control miRNA counts`
> 3. {% tool [Cut columns from a table](Cut1) %}  with the following parameters:
>    - *"Cut columns"*: `c1,c2`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-collection %} *"From"*: `MiRDeep2 BR treated miRNA`
> 4. Rename the output as `BR treated miRNA counts`
>
{: .hands_on}

## Differential expression analysis of miRNAs: **DESeq2**

__DESeq2__ is a tool for differential gene expression analysis based on a negative binomial generalized linear model. __DESeq2__ internally corrects the differences in library size, due to which no preliminary normalization of input datasets is required.

> <comment-title></comment-title>
> It is desirable to use at least three replicates of each experimental condition to ensure sufficient statistical power.
>
{: .comment}

> <hands-on-title>Differential expression analysis using DESeq2</hands-on-title>
>
> 1. {% tool [DESeq2](toolshed.g2.bx.psu.edu/repos/iuc/deseq2/deseq2/2.11.40.8+galaxy0) %} with the following parameters:
>    - *"How"*: `Select datasets per level`
>        - In *"Factor"*:
>            - *"Specify a factor name, e.g. effects_drug_x or cancer_markers"*: `effects_brassinolide`
>            - In *"1: Factor level"*:
>                - *"Specify a factor level"*: `brassinolide`
>                - {% icon param-collection %} *"Counts file(s)"*: `BR treated miRNA counts`
>            - In *"2: Factor level"*:
>                - *"Specify a factor level"*: `control`
>                - {% icon param-collection %} *"Counts file(s)"*: `control miRNA counts`
>    - *"Files have header?"*: `No`
>    - *"Choice of Input data"*: `Count data (e.g. from HTSeq-count, featureCounts or StringTie)`
>    - In *"Advanced options"*:
>        - *"Use beta priors"*: `Yes`
> 2. Rename the outputs as `DESeq2 results miRNA` and `DESeq2 plots miRNA`
> 3. Click on the {% icon galaxy-eye %} (eye) icon and inspect the `DESeq2 plots miRNA` file
{: .hands_on}

DESeq2 generated 2 outputs: a table with the normalized counts and a graphical summary of the results. To evaluate the similarity of our samples, we are going to inspect the Principal Component Analysis (PCA) plot. PCA allows evaluating the dominant directions of the highest variability in the data. Thus, the samples subjected to the same conditions should cluster together.
![fig8:PCA miRNA](../../images/transcriptomics_images/miRNA_PCA.png " PCA plot of expression data from control and BR treated miRNA samples.")

As can be seen, the main axes account for only 47% and 19% of the total variation. This suggests that the effect of brassinosteroids on miRNA regulation is limited (Figure 8).

## Filter significantly differentially expressed miRNAs

Next, we will extract those genes whose expression is statistically significantly differentially expressed (DE) due to BR treatment by selecting those whose adjusted p-value is less than or 0.05. A cut-off value of 0.05 indicates that the probability of a false positive result is less than 5%.

> <details-title>p-value vs p-adj</details-title>
>
> The p-value is a measure of the probability that an observed difference could have occurred just by random chance. A small p-value indicates that there is a small chance of getting this data if no real difference existed. A p-value threshold of 0.05 indicates that there is a 5% chance that the result is a false positive. The p-adj (also known as q-value) is an adjusted p-value which taking in to account the false discovery rate (FDR). Applying a FDR becomes necessary when we're measuring thousands of variables from a small sample set.
>
{: .details}

> <hands-on-title>Filter differentially expressed miRNAs</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} data on any column using simple expressions (Galaxy Version 1.1.1) with the following parameters:
>    - {% icon param-file %} *"Filter"*: `DESeq2 results miRNA`
>    - *"With following condition"*: `c7<0.05`
>
{: .hands_on}

> <question-title></question-title>
>
> How many miRNAs show statistically significant differential expression?
>
> > <solution-title></solution-title>
> >
> > Unfortunately, we have not detected any differentially expressed miRNAs. In this case, this is caused by the fact that the subsampled datasets don't have sufficient read depth to test for differential expression.
> {: .solution}
>
{: .question}

To get any sensible results, it is worth analyzing the full dataset. You can analyze the full datasets following the above tutorial. Otherwise, you can import the DESeq2 analysis results that we generated from full dataset into your history.

> <hands-on-title>Retrieve the DESeq2 analysis results on full miRNA dataset</hands-on-title>
>
> 1. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click of the **Paste/Fetch** button
>    - Copy the Zenodo links and press <kbd>Start</kbd>
>
>      ```
>      https://zenodo.org/record/4663389/files/miRNA_DESeq2_results_complete_dataset.tabular
>      ```
>
> 2. Rename each dataset according to the sample id (e.g. `miRNA_DESeq2_results_complete_dataset`)
> 3. Add all miRNA data analysis related tags: #miRNA #BR #control
{: .hands_on}

Repeat the filtering step on the imported DESeq2 result.

> <hands-on-title>Filter and sort differentially expressed miRNAs from the full dataset</hands-on-title>
>
> {: .comment}
> 1. {% tool [Filter](Filter1) %} data on any column using simple expressions (Galaxy Version 1.1.1) with the following parameters:
>    - {% icon param-file %} *"Filter"*: `miRNA_DESeq2_results_complete_dataset`
>    - *"With following condition"*: `c7<0.05`
> 2. Rename the output as `Differentially expressed miRNAs`
> 3. {% tool [Filter](Filter1) %} data on any column using simple expressions (Galaxy Version 1.1.1) with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Differentially expressed miRNAs`
>    - *"With following condition"*: `c3>0`
> 4. Rename the output as `Upregulated miRNAs`
> 5. {% tool [Sort](sort1) %} data in ascending or descending order (Galaxy Version 1.1.0) with the following parameters:
>    - {% icon param-file %} *"Sort Dataset"*: `Upregulated miRNAs`
>    - *"on column"*: `Column: 3`
>    - *"everything in"*: `Descending order`
{: .hands_on}

> <question-title></question-title>
>
> 1. How many miRNAs are differentially expressed?
> 2. How many miRNAs show statistically significant upregulation and what is the most upregulated miRNA?
>
> > <solution-title></solution-title>
> >
> > 1. Out of 442 miRNAs, 39 show significant differential expression.
> > 2. There are 16 upregulated miRNAs and `Ath-miR156g` is the most upregulated one.
> {: .solution}
>
{: .question}

# mRNA data analysis

Once the differential expression analysis of miRNAs has been carried out, the next stage is to analyze how mRNA expression is altered in response to brassinosteroids.

## Quality assessment of mRNA reads

As in the previous section, we shall begin by assessing the quality of our sequences.

> <hands-on-title>Initial quality check</hands-on-title>
>
> 1. {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"Dataset collection"*: `Control mRNA`
> 2. Rename the outputs as `Falco unprocessed control mRNA: RawData` and `Falco unprocessed control mRNA: Webpage`
> 3. Repeat the previous step with the `BR treated mRNA` collection.
> 4. {% tool [Merge Collections](__MERGE_COLLECTION__) %} with the following parameters:
>    - In *"Input collections"*:
>        - *"1.Input Collections"*: `Falco unprocessed control mRNA: RawData`
>        - *"2.Input Collections"*: `Falco unprocessed BR treated mRNA: RawData`
> 5. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>    - In *"Results"*:
>      - *"Which tool was used generate logs?"*: `FastQC`
>        - In *"FastQC output"*:
>           - {% icon param-repeat %} *"Insert FastQC output"*
>               - {% icon param-collection %} *"FastQC output"*: select the output of __Merge Collection__ tool
>    - *"Report title"*: `mRNA initial quality check`
> 6. Click on the {% icon galaxy-eye %} (eye) icon and inspect the generated HTML file
>
{: .hands_on}

> <question-title></question-title>
>
> Are there any stats that indicate the need to process the samples in order to improve their quality?
>
> > <solution-title></solution-title>
> >
> > All the stats are within acceptable limits. However, the adapter content shows the presence of Illumina universal adapters in our reads, which can be removed to avoid possible interferences at later stages (Figure 10).
> >
> >    ![fig10:Adapter content of mRNA samples](../../images/transcriptomics_images/miRNA_adapter_01.png "Quality assessment of mRNA samples")
> >
> {: .solution}
{: .question}

> <comment-title></comment-title>
> Although our samples have adapters, we are not going to trim them in this case. We will explain the reason in the next section.
{: .comment}

## Quantification of gene expression: **Salmon**

After performing the quality assessment of the reads, we can move on to quantifying the gene expression. The aim of this step is to identify which transcript each read comes from and the total number of reads associated with each transcript. In this tutorial, we will use the Salmon tool ({% cite Patro2017 %}) for the quantification of mRNA transcripts.

![fig11:Salmon logo](../../images/transcriptomics_images/salmon_logo.png)

One of the characteristics of __Salmon__ is that it doesn't require performing a base-to-base alignment, which is the time-consuming step of tools such as __STAR__ and __HISAT2__. Salmon relies on the quasi-mapping concept, a new mapping technique that allows the rapid and accurate mapping of RNA-Seq reads to a target transcriptome. Rather than a standard alignment, quasi-mapping seeks to find the best mappings for each read, and does so by finding minimal collections of dynamically sized, right-maximal, matching contexts between target and query positions ({% cite Srivastava2016 %})

The quasi-mapping approach by __Salmon__ requires a reference index to determine the position and orientation information for accurate mapping prior to quantification. It allows providing the transcriptome in a format that optimizes its use in transcript identification and quantification.

After determining the best mapping for each read, __Salmon__  generates the final transcript abundance estimation after modeling sample-specific parameters and biases. Reads that map equally well to more than one transcript will have the count divided between all of the mappings, thus avoiding the loss of information on the different gene isoforms.


> <details-title>Quasi-mapping algorithm</details-title>
> The quasi-mapping algorithm makes use of two main data structures, the generalized suffix array (SA) of the transcriptome T, and a hash table (h) that maps each k-mer in T to its SA interval (by default k = 31). During the quasi-mapping procedure, a read is scanned from left to right until a k-mer (k<sub>i</sub>, starting at position i on the read) is encountered that appears in h. The k<sub>i</sub> is looked up in the hash table and the SA intervals are retrieved, giving all suffixes containing the k-mer k<sub>i</sub>. Then, the Maximal Mappable Prefix (MMP<sub>i</sub>) is computed by finding the longest substring of the read that matches the reference suffixes. Owing to sequencing errors, the MMPs may not span the complete read. In this case, the next informative position (NIP) is determined based on the longest common prefix of the SA intervals of MMP<sub>i</sub>. Suffix array search continues from k-bases before NIP until the final NIP is less than k-bases before the read end. Finally, for each read, the algorithm reports the transcripts it mapped to, location and strand information ({% cite Srivastava2016 %}).
> ![fig12:Quasi-mapping algorithm](../../images/transcriptomics_images/salmon_quasialignment.png "Illustration of the quasi-mapping of a read using k=3. Hash table lookup of k<sub>i</sub> returns the suffix array interval [b, e). The base 'G' at position 6 in the read is a result of sequencing error. Hence, the MMP<sub>i</sub> is 'ATTGA' and the SA interval of MMP<sub>i</sub> is [b', e'). The next k-mer starts at k-bases prior to the NIP which is base after the longest common prefix of the interval [b',e'). In the end, the read in the above example most likely map to the suffix array at [e'-1, e').")
>
{: .details}


> <hands-on-title>Quantify gene expression with Salmon</hands-on-title>
>
> 1. {% tool [Salmon quant](toolshed.g2.bx.psu.edu/repos/bgruening/salmon/salmon/1.10.1+galaxy2) %} with the following parameters:
>    - *"Select salmon quantification mode:"*: `Reads`
>        - *"Select a reference transcriptome from your history or use a built-in index?"*: `Use one from the history`
>            - In *"Salmon index"*:
>                - {% icon param-file %} *"Transcripts fasta file"*: `transcriptome.fasta`
>        - In *"Data input"*:
>            - *"Is this library mate-paired?"*: `Single-end`
>                - {% icon param-collection %} *"FASTQ/FASTA file"*: `Control mRNA`
>        - *"Validate mappings"*: `Yes`
>    - {% icon param-file %} *"File containing a mapping of transcripts to genes"*: `annotation_AtRTD2.gtf`
>
> 2. Rename the outputs as `Salmon control mRNA (Quantification)` and `Salmon control mRNA (Gene Quantification)`
> 3. Repeat the previous procedure by using the `BR treated mRNA` dataset
>
>   > <comment-title>Quasi-mapping sequence requirements</comment-title>
>   > Trimming the reads is not required  when using this method, since if there are k-mers in the reads that are not in the hash table, they are not counted. Quantification of the reads is only as good as the quality of the reference transcriptome.
>   {: .comment}
>
{: .hands_on}

__Salmon__ generates two outputs for each input sample:

- Quantification: summarizes the quantifications by transcript
- Gene quantification: summarizes the quantification by gene

Each output consists of a tabular dataset with five columns:


------------------| ----------
Field             | Description
------------------| ----------
Name              | The name of the target transcript provided in the input transcriptome.
Length            | The length of the target transcript.
EffectiveLength   | The computed effective length.
TPM               | The relative abundance of this transcript in units of Transcripts Per Million.
NumReads          | The number of reads mapping to each transcript that was quantified.
------------------| ----------


> <question-title></question-title>
>
> Could you explain why we did not trim the reads before?
>
> > <solution-title></solution-title>
> >
> > The reason is that, since the k-mers which contain the adaptor sequences are not present in the transcriptome (from which the hash table is generated), they are omitted.
> >
> {: .solution}
>
{: .question}

## Differential expression analysis of mRNAs: **DESeq2**

Now, let's analyze which genes show statistically significant differential expression in response to brassinosteoids.

> <hands-on-title>Differential expression analysis using DEseq2</hands-on-title>
>
> 1. {% tool [DESeq2](toolshed.g2.bx.psu.edu/repos/iuc/deseq2/deseq2/2.11.40.8+galaxy0) %} with the following parameters:
>    - *"How"*: `Select datasets per level`
>        - In *"Factor"*:
>            - *"Specify a factor name, e.g. effects_drug_x or cancer_markers"*: `effects_brassinolide`
>            - In *"1: Factor level"*:
>                - *"Specify a factor level"*: `brassinolide`
>                - {% icon param-collection %} *"Counts file(s)"*: `Salmon BR treated mRNA (Gene Quantification)`
>            - In *"2: Factor level"*:
>                - *"Specify a factor level"*: `control`
>                - {% icon param-collection %} *"Counts file(s)"*: `Salmon control mRNA (Gene Quantification)`
>    - *"Choice of Input data"*: `TPM values (e.g. from kallisto, sailfish or salmon)`
>        - *"Program used to generate TPMs"*: `Salmon`
>        - *"Gene mapping format"*: `GTF/GFF3`
>            - {% icon param-file %} *"GTF/GFF3 annotation file"*: `annotation_AtRTD2.gtf`
> 2. Rename the outputs as `DESeq2 results mRNA` and `DESeq2 plots mRNA`
{: .hands_on}

![fig13:PCA plot of mRNA samples](../../images/transcriptomics_images/mRNA_PCA.png " PCA plot of differential expression data from control and BR treated samples.")

> <question-title></question-title>
>
> What conclusions about the similarity of the samples can be derived from the PCA plot?
>
> > <solution-title></solution-title>
> >
> > From the information provided by the plot, it is possible to state that there is a high similarity between the samples belonging to the same experimental conditions, since the first dimension (x-axis) allows to explain 79% of the variability, and the samples are located at opposite ends of the x-axis.
> >
> {: .solution}
>
{: .question}

## Filter significantly differentially expressed mRNAs

To conclude the analysis of the differential expression of mRNAs, we will extract those transcripts that show a significant induction of expression in response to brassinosteroids. Before continuing with further analysis, similar to miRNA data analysis, import the DESeq2 results generated from full mRNA datasets.

> <hands-on-title>Retrieve the DESeq2 analysis results on full mRNA dataset</hands-on-title>
>
> 1. Import the files from Zenodo:
>
>    - Open the file {% icon galaxy-upload %} __upload__ menu
>    - Click of the **Paste/Fetch** button
>    - Copy the Zenodo links and press <kbd>Start</kbd>
>
>      ```
>      https://zenodo.org/record/4663389/files/mRNA_DESeq2_results_complete_dataset.tabular
>      ```
>
> 2. Rename each dataset according to the sample id (e.g. `mRNA_DESeq2_results_complete_dataset`)
> 3. Add all mRNA data analysis related tags: #mRNA #BR #control
{: .hands_on}

Now we continue with the DE genes analysis.

> <hands-on-title>Extract significantly DE genes</hands-on-title>
>
> 1. {% tool [Filter](Filter1) %} data on any column using simple expressions (Galaxy Version 1.1.1) with the following parameters:
>    - {% icon param-file %} *"Filter"*: `mRNA_DESeq2_results_complete_dataset`
>    - *"With following condition"*: `c7<0.05`
> 2. Rename the output as `Differentially expressed mRNAs`
>
> 3. {% tool [Filter](Filter1) %}data on any column using simple expressions (Galaxy Version 1.1.1) with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Differentially expressed mRNAs`
>    - *"With following condition"*: `c3>1`
> 4. Rename it as `Upregulated mRNAs`
>
> 5. {% tool [Filter](Filter1) %} data on any column using simple expressions (Galaxy Version 1.1.1) with the following parameters:
>    - {% icon param-file %} *"Filter"*: `Differentially expressed mRNAs`
>    - *"With following condition"*: `c3<-1`
> 6. Rename it as `Downregulated mRNAs`
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many genes show statistically significant differential expression?
> 2. How many of them are significantly upregulated (at least two-fold change)? And downregulated?
> 3. What is the most significantly DE downregulated gene and what is it biological function?
>
> > <solution-title></solution-title>
> >
> > 1. The total number of genes whose expression differential between the two experimental conditions is 4176.
> > 2. Of them, 328 are significantly downregulated by the BR treatment and 778 are upregulated.
> > 3. The most significantly DE gene is AT3G30180 (BR60X2). BR60X2 encodes a cytochrome p450 enzyme that catalyzes the last reaction in the production of brassinolide. It is capable of converting 6-deoxocastasterone into castasterone, a C-6 oxidation, as well as the further conversion of castasterone into brassinolide (source: [TAIR database](https://www.arabidopsis.org/servlets/TairObject?name=at3g30180&type=locus)).
> {: .solution}
>
{: .question}

# Identification of miRNA targets

To predict which miRNAs target which mRNAs, first we need their transcriptomic sequences in FASTA format. Now we will obtain the sequences of miRNAs whose expression is induced by brassinosteroids.

> <hands-on-title>Obtaining the sequences of upregulated miRNAs</hands-on-title>
> > <comment-title></comment-title>
> >
> > The following tools can be found in _Text Manipulation_ and _Filter and Sort_ sections.
> >
> {: .comment}
>
> 1. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-collection %} *"From"*: `Upregulated miRNAs`
> 2. Rename the output as `Upregulated miRNA ids`
> 3. {% tool [Filter FASTA](toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3) %} with the following parameters:
>    - {% icon param-file %} *"FASTA sequences"*: `mature_miRNA_AT.fasta`
>    - *"Criteria for filtering on the headers"*: `List of IDs`
>        - {% icon param-file %} *"List of IDs to extract sequences for"*: `Upregulated miRNA ids`
>        - *"Match IDs by"*: `Default: ID is expected at the beginning: >ID`
> 4. {% tool [Filter FASTA](toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3) %} with the following parameters:
>    - {% icon param-file %} *"FASTA sequences"*: `star_miRNA_seq.fasta`
>    - *"Criteria for filtering on the headers"*: `List of IDs`
>        - {% icon param-file %} *"List of IDs to extract sequences for"*: `Upregulated miRNA ids`
>        - *"Match IDs by"*: `Default: ID is expected at the beginning: >ID`
> 5. {% tool [Concatenate multiple datasets or collections](cat1) %} tail-to-head (Galaxy Version 1.0.0) with the following parameters:
>    - {% icon param-file %} *"Concatenate Dataset"*: output of **Filter FASTA** {% icon tool %} on `mature_miRNA_AT.fasta`
>    - + Insert Dataset
>        - {% icon param-file %} *"Select"*: output of **Filter FASTA** {% icon tool %} on `star_miRNA_seq.fasta`
> 6. Rename it as `Upregulated miRNA sequences`
> 7. Click on the {% icon galaxy-eye %} (eye) icon and inspect the `Upregulated miRNA sequences` file
{: .hands_on}

To identify putative targets of upregulated miRNAs, it is necessary to obtain the sequences of all downregulated mRNAs in FASTA format.

> <hands-on-title>Obtaining the gene sequences of downregulated mRNAs</hands-on-title>
> 1. {% tool [Cut columns from a table](Cut1) %} with the following parameters:
>    - *"Cut columns"*: `c1`
>    - *"Delimited by"*: `Tab`
>    - {% icon param-collection %} *"From"*: `Downregulated mRNAs`
> 2. Rename the output as `Downregulated mRNA ids`
> 3. {% tool [Filter FASTA](toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3) %} with the following parameters:
>    - {% icon param-file %} *"FASTA sequences"*: `transcriptome.fasta` (Input dataset)
>    - *"Criteria for filtering on the headers"*: `List of IDs`
>        - {% icon param-file %} *"List of IDs to extract sequences for: `Downregulated mRNA ids`
>        - *"Match IDs by"*: `Custom regex pattern`
>           - *"Regex search pattern for ID"*: `GENE=(AT.{7})`
> 4. Rename it as `Downregulated mRNA sequences`
> 5. Click on the {% icon galaxy-eye %} (eye) icon and inspect the `Downregulated mRNA sequences`  file
{: .hands_on}

## miRNA target prediction using **TargetFinder**

We are now ready to launch the search for miRNA target genes. For this we will use the __TargetFinder__ tool that is implemented and used for miRNA target prediction in plants ({% cite Srivastava2014 %}, {% cite Ma2017 %}).

> <hands-on-title>identification of potential miRNA targets</hands-on-title>
>
> 1. {% tool [TargetFinder](toolshed.g2.bx.psu.edu/repos/rnateam/targetfinder/targetfinder/1.7.0+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input small RNA sequences file"*: `Upreguled miRNA sequences`
>    - {% icon param-file %} *"Target sequence database file*": `Downregulated mRNA sequences`
>    - *"Prediction score cutoff value"*: `5.0`
>    - *"Output format"*: `Tab-delimited format`
> 2. Click on the {% icon galaxy-eye %} (eye) icon and inspect the output of **TargetFinder**.
{: .hands_on}

Congratulations! We have identified the following 5 potential genes involved in the brassinosteroid-miRNA regulatory network.

![fig15:TargetFinder results](../../images/transcriptomics_images/targetFinder_results.png "TargetFinder results.")

Finally, we can access all the information available on the genes identified in the TAIR database:

- [AT5G10180](https://www.arabidopsis.org/servlets/TairObject?type=locus&name=AT5G10180): ARABIDOPSIS SULFATE TRANSPORTER 68, AST6
- [AT3G09220](https://www.arabidopsis.org/servlets/TairObject?type=locus&name=AT3G09220): LAC7, LACCASE 7
- [AT2G46850](https://www.arabidopsis.org/servlets/TairObject?type=locus&name=AT2G46850): Protein kinase superfamily protein
- [AT5G64260](https://www.arabidopsis.org/servlets/TairObject?type=locus&name=AT5G64260): EXL2, EXORDIUM LIKE 2
- [AT3G63200](https://www.arabidopsis.org/servlets/TairObject?type=locus&name=AT3G63200): PATATIN-LIKE PROTEIN 9, PLA IIIB

Both AT4G14365 and AT1G26890 are not well characterized genes. In the case of AT5G50570, experimental research have demonstrated that this gene is involved in  flooding tolerance in *Medicago sativa*, through a signaling path mediated by miR156 ({% cite Feyissa2021 %}). On the other hand, according to {% cite Gao2017 %}, SPL13 is likely a negative regulator of plant vegetative growth. The interaction between miR156 and SPL transcription factors has been suggested for diverse Poaceae family members ({% cite Yue2021 %}).

> <comment-title>Hypothesis testing</comment-title>
> One of the hypotheses that we could propose from our results is: the inhibition of the AT2G46850 gene can result in plants with improved resistance to drought conditions. Is it possible to validate it? Yes! We propose this approach: to acquire [AT2G46850 mutant seeds](https://abrc.osu.edu/stocks/392113) and [wild type seeds](http://arabidopsis.info/StockInfo?NASC_id=N1093), grow them under two controlled conditions: watered and drought stress, and analyze plant weight after 33 days (Figure 13).
>
> ![fig16:Plant growth](../../images/transcriptomics_images/arabidopis_growth_protocol.png "Arabidopsis growth conditions protocol ({% cite deOllas2019 %}).")
>
{: .comment}

# Optional exercise

As additional activity, you can try to repeat the workflow by using the sequences stored in the NCBI GEO database with the accession number `GSE119382`. In that case, we will compare gene expression patterns of mutants overexpressing the brassinosteroid receptor BRL3 under two experimental conditions: control and drought-stress. The required datasets are available in the data library:

> <hands-on-title>Import data from the Data Libraries</hands-on-title>
> 1. Go into __Data__ (top panel) and click on __Data Libraries__
> 2. In the search box enter the following identifier: `4710649`
> 3. Select the following files:
> ```
> https://zenodo.org/record/4710649/files/SRR7779222_BRL3_mRNA_drought.fastqsanger.gz
> https://zenodo.org/record/4710649/files/SRR7779223_BRL3_mRNA_drought.fastqsanger.gz
> https://zenodo.org/record/4710649/files/SRR7779224_BRL3_mRNA_drought.fastqsanger.gz
> ```
> 4. Click on __Export to History__ and __as a Collection__
> 5. In the pop-up window press <kbd>Continue</kbd>
> 6. Provide it the name `BRL3 mRNA drought` and push <kbd>Create list</kbd>
> 7. Repeat the previous procedure with the remaining files:
> ```
> https://zenodo.org/record/4710649/files/SRR7779228_BRL3_mRNA_watered.fastqsanger.gz
> https://zenodo.org/record/4710649/files/SRR7779229_BRL3_mRNA_watered.fastqsanger.gz
> ```
> 9. Finally provide it the name `BRL3 mRNA control` and push <kbd>Create list</kbd>
{: .hands_on}

We will use the upregulated miRNAs obtained in the previous analysis in order to identify potential targets.

```
Upregulated miRNA   https://zenodo.org/record/4710649/files/upregulated_miRNA_BR_complete_dataset.fasta
```

# Conclusion


In this tutorial, we have analyzed RNA sequencing data to extract information about potential genes regulated by brassinosteroids. For this purpose, the approach chosen was the identification of genes complementary to miRNAs upregulated in response by brassinosteroids. The final result has been the identification of five potential miRNA targets.

tools_json:
["Sub-sample sequences files", "toolshed.g2.bx.psu.edu/repos/peterjc/sample_seqs/sample_seqs/0.2.6", "Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "Merge Collections", "__MERGE_COLLECTION__", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3", "Trim Galore!", "toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.6.10+galaxy0", "MiRDeep2 Mapper", "toolshed.g2.bx.psu.edu/repos/rnateam/mirdeep2_mapper/rbc_mirdeep2_mapper/2.0.0.8.1", "MiRDeep2 Quantifier", "toolshed.g2.bx.psu.edu/repos/rnateam/mirdeep2_quantifier/rbc_mirdeep2_quantifier/2.0.0", "Cut columns from a table", "Cut1", "DESeq2", "toolshed.g2.bx.psu.edu/repos/iuc/deseq2/deseq2/2.11.40.8+galaxy0", "Filter", "Filter1", "Sort", "sort1", "Salmon quant", "toolshed.g2.bx.psu.edu/repos/bgruening/salmon/salmon/1.10.1+galaxy2", "Filter FASTA", "toolshed.g2.bx.psu.edu/repos/galaxyp/filter_by_fasta_ids/filter_by_fasta_ids/2.3", "Concatenate multiple datasets or collections", "cat1", "TargetFinder", "toolshed.g2.bx.psu.edu/repos/rnateam/targetfinder/targetfinder/1.7.0+galaxy1"]


