# tutorials

topic: contributing
tutorial: rendering_gtn
title: Preview the GTN website as you edit your training material
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["Choose your own Adventure", "GitPod", "CodeSpaces", "Command-line"]
content_hash: b40893f6e2c530cfec14c456fb48e1f3
last_modified: 2026-05-27T09:01:47.929869
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/rendering_gtn/tutorial.html

content:
If you are working on training materials, you will likely want to preview your changes as you go! You have a few options on how to do this.

{% include _includes/cyoa-choices.html option1="Codespaces" option2="Gitpod" option3="Command-line" option4="Container-based" default="Codespaces"
       text="We recommend using Codespaces. We used to use Gitpod, however it loads a bit slower and the interface changed as of spring 2025, so the tutorial may be out of date. **CodeSpaces and GitPod are both online methods**; only the command-line option requires you to install things on your machine." %}

<div class="Codespaces" markdown="1">

If you are working on your own training materials and want preview them online without installing anything on your computer, you can do this using GitHub CodeSpaces! Everybody gets 60 free hours of CodeSpaces per month.



# Launching CodeSpaces


> <hands-on-title>Setting up Codespaces</hands-on-title>
>
> 1. **Create a fork** of the [GTN GitHub repository](https://github.com/galaxyproject/training-material)
>
>    {% snippet topics/contributing/faqs/github-fork-gtn.md %}
>
> 2. Already have a fork of the GTN? Make sure it is up to date.
>
>    {% snippet topics/contributing/faqs/github-fork-master-main.md %}
>
>    {% snippet topics/contributing/faqs/github-fork-sync.md %}
>
> 2. **Navigate to your fork** of the GTN
>
> 3. Click on the green **Code** button
>
> 4. At the top, switch to the **CodeSpaces** tab
>    ![the codespaces tab of the menu](images/codespaces-launch.png)
>
> 5. Click on **Create codespace on main**
>    - Note: if you switch to a specific branch in GitHub first, you can create a codespace for that branch
>
> 6. This will setup a [Visual Studio Code](https://code.visualstudio.com/) environment for you
>    - It may take a couple minutes to finish setting everything up
>    - In this environment you can also build the GTN website to preview your changes
>    - When everything is ready, you should see something like this:
>
>    ![screenshot of Codespaces just after startup](images/codespaces-home.png)
{: .hands_on}

# The VScode environment

Let's have a closer look at your CodeSpaces window:

- **Left:** Here you see all the files in the GTN repository
- **Top:** This is the main window where you can view and edit files
- **Bottom:** Terminal window. Here you can type commands (e.g. to build the website preview) and read output and error messages.

![VScode interface](images/codespaces-home.png)


# Build and preview the GTN website

Before we start making any changes, let's build the website and get a live preview.

> <hands-on-title>Setting up GitPod</hands-on-title>
>
> 1. In the terminal window (bottom), type the following command:
>    `make preview`
>
>    - This will take 2-3 minutes to complete
>
>    ![output in the terminal after issuing the make preview command](images/codespaces-make-preview.png)
>
> 3. When the build process is finished, a preview window will automatically open (at the top right)
>    - The preview will show the GTN 404 (codespace doesnt know what to show)
>    - Just click on **Return to homepage**.
>
>    > <tip-title>Window not opening? </tip-title>
>    > If the preview window doesn't open for you, or if you close it and want to reopen it, you can always do so as follows:
>    > 1. Go to the **Ports** tab of the bottom panel
>    >
>    >    ![screenshot of the tabs on the bottom panel](images/codespaces-ports-tab.png)
>    >
>    > 2. Hover over the link in the **Forwarded Address** column, 3 icons should appear
>    >
>    >    ![screenshot of the 3 icons](images/codespaces-ports-preview-buttons.png)
>    >
>    > 3. Click on:
>    >    - The **world/globe icon** to open the GTN preview in a new window, or
>    >    - Or, click on the **window icon** to the right of the globe icon to preview the GTN in a tab inside the codespaces environment
>    {: .tip}
>
> 4. If you opened the GTN preview inside the codespace, your window will now look something like this:
>
>    ![screenshot of the codespace with the preview editor opend to the GTN homepage](images/codespaces-preview-editor.png)
>
>    > <tip-title>Not opening?</tip-title>
>    > ![screenshot of firefox permissions dialog which shows blocked windows and an allow menu](../../images/gitpod_popup.png)
>    > Some browsers block popups by default, you may need to allow CodeSpaces to show popups in your browser.
>    {: .tip}
>
{: .hands_on}



# Editing Training Materials on CodeSpaces

Now that you have the codespace environment working and we have a live preview up, let's make some changes to the GTN materials and get an instant preview.


**Scenario:** You have spotted a typo in one of the tutorials, and would like to fix this and see the resulting GTN webpage.


> <hands-on-title>Make and view changes</hands-on-title>
>
> 1. In the preview of the GTN website, open the following tutorial:
>    - Topic: "Introduction to Galaxy Analyses" topic
>    - Tutorial: "A Short Introduction to Galaxy""
>    - We will edit this tutorial and watch the live preview window for the effects
>
>
> 2. On the file browser on the left, open the following file:
>
>    ```
>    topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>    ```
>
> 3. Change the title of the tutorial
>    - **From:** "A Short Introduction to Galaxy"
>    - **To:** "A Short and Cool Introduction to Galaxy"
>
>    ![we changed the title of the tutorial in the text editor window](images/codespaces-tutorial-edited.png)
>
> 4. Save the file
>    - **CTRL+S** to save the file
>    - You should immediately see a message in the terminal saying "regenerating". CodeSpaces has detected your changes and is rebuilding the website.
>
>    ![the terminal shows a message stating the website is being regenerated](images/codespaces-regenerating.png)
>
> 5. Move to the top right panel where the GTN is previewed and refresh the website
>    - {% icon galaxy-refresh %} Refresh button in front of the address bar of the preview panel
>    - You can also open the preview in it's own brower tab, using the {% icon galaxy_instance %} button at the top-right corner of the preview window. Then you can reload the page the regular way (e.g. <kbd>F5</kbd> or <kbd>ctrl + r</kbd> or the reload button in the browser)
>
>    > <tip-title> Reload not working? </tip-title>
>    > It is possible that this reload button gives you the 404 again, in that case there are 2 solutions
>    > 1. Right-click in the preview panel, and choose
>    >    - Chrome: "Reload Frame"
>    >    - Firefox: "This Frame -> Reload Frame"
>    > 2. Open the preview in it's own browser tab
>    >    - Click the {% icon galaxy_instance %} button at the top-right corner of the preview window
>    {: .tip}
>
> 6. You should see the change you made:
>
>    ![The updated preview with our changed tutorial title](images/codespaces-reloaded.png)
>
{: .hands_on}


In this way you can edit files in the text editor, and see the effects in the website preview.


# Saving your changes back to GitHub

When you have finished your changes, it all looks good in the preview, you want to save your changes back to GitHub so that you can either continue later, or make a Pull Request to the GTN to contribute your changes.


> <hands-on-title>Comitting changes</hands-on-title>
> Before you can commit your changes you have to create a branch. You have two options to preform this task:
>  1. **Option 1: via the terminal**
>     - Hit <kbd>ctrl+c</kbd> if your preview was still running to stop it
>     - Create a new branch, commit your changes, push changes:
>
>     ```bash
>     git checkout -b fix-title
>     git commit -m "update tutorial title" topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>     git push origin fix-title
>     ```
>
> 2. **Option 2: via the web interface**
>    - Create a new branch:
>      -  On the bottom-left, click on the branch logo (probably labelled "main")
>         ![button on the bottom of the page with the current branch](images/codespaces-branch-change1.png)
>      - Enter `fix-title` as the name for your new branch (at top of window)
>         ![Addint a new branch](images/codespaces-branch-change2.png)
>      - Choose "+ Create new branch..." from the dropdown
>    - Commit changes:
>      - On the left menu, click on the "changed files" tab
>        ![button on the left menu to show changed files](images/codespaces-commit-1.png)
>      - You should see your changed file (`tutorial.md`)
>      - Click on the "+" icon next to the file we edited to *stage changes*
>        ![stage changes button](images/codespaces-commit-plus.png)
>      - Enter a commit message (top of window)
>      - Hit the checkmark icon below the massage to commit the changes
>   - Publish changes
>     - Click the cloud button at bottom left to publish your changes
>       ![publish changes button](images/codespace-publlish.png)
>
{: .hands_on}

Next, we will see these changes to on your branch/fork. We will do this from outside of our codespace.

> <hands-on-title>Pushing changes to GitHub</hands-on-title>
>
> 1. In your browser (outside of codespaces), navigate to the [GTN GitHub page](https://github.com/galaxyproject/training-material)
> 2. GitHub will helpfully show you any recent branches you've pushed to your fork
>    - Click the **Compare & pull request** button to create a PR for your changes
>    ![compare and pull request button on the new branch](images/codespaces-compare-pr.png)
{: .hands_on}


# Closing your CodeSpace

Everybody gets 60 hours per month for free on CodeSpaces. Your codespace will automatically shut down after 30 minutes of inactivity,
but it is always a good idea to close your CodeSpace when you are finished with it, to conserve your quotum.


> <hands-on-title>Shutting down your CodeSpace</hands-on-title>
>
> 1. Return to the [GTN GitHub page](https://github.com/galaxyproject/training-material)
>
> 2. Click on the green **Code** button again
>
> 3. Under the Codespaces tab, you should see your running codespace
>
>    ![codespaces menu in github](images/codespaces-stop-1.png)
>
> 4. Click on the 3-dots menu to the right of your (randomly generated) codespace name
>
> 5. In this menu you can quit your codespace in two ways:
>    - **Stop codespace**: your changes will be kept and you can restart the codespace later
>    - **Delete** your codespace. Any changes you did not commit and push to GitHub are lost.
>      ![screenshot of the codespace options menu](images/codespaces-stop-2.png)
>
>    - In this menu you can also resume a stopped codespace by simply clicking **Open in Browswer**
>
{: .hands_on}


Congrats! You learned how to contribute to the GTN by using the CodeSpaces environment!

</div>

<div class="Gitpod" markdown="1">


If you are working on your own training materials and want preview them online without installing anything on your computer, you can do this using GitPod!



# Setting up Gitpod


> <hands-on-title>Setting up GitPod</hands-on-title>
>
> 1. **Create a fork** of the [GTN GitHub repository](https://github.com/galaxyproject/training-material)
>
>    {% snippet topics/contributing/faqs/github-fork-gtn.md %}
>
> 2. Already have a fork of the GTN? Make sure it is up to date.
>
>    {% snippet topics/contributing/faqs/github-fork-master-main.md %}
>
>    {% snippet topics/contributing/faqs/github-fork-sync.md %}
>
> 3. **Open** your browser and navigate to [gitpod.io/login](https://gitpod.io/login)
>    - Note: don't leave the `/login` part of the URL off, it will lead you to a different flavour of GitPod. We are using Gitpod classic
> 4. **Log in** with GitHub
>    ![GitPod sign-in menu with various options, including GitHub](../../images/gitpod/signin.png){: width="25%"}
> 5. Click on **Configure your own repository** under the Workspaces menu
>    ![GitPod homepage](../../images/gitpod/gitpod-home.png)
> 6. Under **Select a repository** choose your fork, e.g. `https://github.com/shiltemann/training-material`
>    ![screenshot of new workspace menu](../../images/gitpod/gitpod-workspace.png)
> 7. Click **continue**
>    - This will create an enviroment where you can make changes to the GTN and preview them
>    - **Note:** It can take quite some time to start up the first time (15-30 minutes)
>    - We can make future starts a lot faster using **prebuilds** (see tip box below), you can configure this now while you wait
>
>    ![progress indicator of GitPod first setup](../../images/gitpod/setup.png)
>
>    > <tip-title> Enabling Prebuilds for your GitPod workspace </tip-title>
>    > Prebuilds can speed up the launch of your GitPod workspace after the first time. When this is enabled,
>    > starting a workspace will check if there is a previous build to reuse. This should bring startup times
>    > down to 3-5 minutes.
>    >
>    > **Enable Prebuilds**
>    > 1. On the GitPod website, click on your name in the top left of the screen
>    > 2. Select **Repositories**
>    > 3. Add your fork as a repository
>    > 4. On the left, click **Prebuild settings**
>    > 5. Click the slider in front of **Prebuilds are disabled** to toggle the setting
>    > 6. Set **Branch Filter** to **Default Branch**
>    >
>    > ![screenshot of prebuild setting of our repository](../../images/gitpod/gitpod-prebuilds.png)
>    >
>    > **Run a prebuild now**
>    > 1. In GitPod, click on your name in the top left of the screen
>    > 2. Select **Prebuilds**
>    > 3. Click **Run a Prebuild**
>    > 4. **Select a Repository**: your fork
>    > 5. Click **Run Prebuild**
>    >    - This will also take a bit of time to complete
>    >    - In the future, it will automatically refresh this prebuild every 20 commits, so you only have to do this once
>    >
>    >
>    > **Note:** Prebuilds expire if you haven't used them in 7 days, so if you have taken a longer
>    > break from GTN you might have to wait longer again the first time. But otherwise it should
>    > bring the launch time down to around 3-5 minutes.
>    {: .tip}
>
> 8. Once the setup is finished, you should see a page something like this:
>    ![GitPod welcome screen](../../images/gitpod/welcomepage.png)
>
>    > <tip-title>Make this even easier with the GitPod browser extension!</tip-title>
>    >  1. To make this even easier, you can install the GitPod browser extension
>    >     - [Firefox](https://addons.mozilla.org/en-GB/firefox/addon/gitpod/)
>    >     - [Chrome or Edge](https://chrome.google.com/webstore/detail/gitpod-always-ready-to-co/dodmmooeoklaejobgleioelladacbeki)
>    >  2. Now it is just a single click to open any GitHub repository in GitPod
>    >     ![screenshot of the button the GitPod browser extension adds to GitHub repository ](../../images/gitpod/browser-extension.png)
>    {: .tip}
>
{: .hands_on}

# The GitPod Workspace

Let's have a closer look at the GitPod workspace:

- **Left:** Here you see all the files in the GTN repository
- **Top:** This is the main window where you can view and edit files
- **Bottom:** Terminal window. Here you can type commands (e.g. to build the website preview) and read output and error messages.

![GitPod welcome screen](../../images/gitpod/welcomepage.png)



# Build and preview the GTN website

Before we start making any changes, let's build the website and get a live preview.

> <hands-on-title>Setting up GitPod</hands-on-title>
>
> 1. In the terminal window (bottom), type the following command:
>    `make serve-gitpod`
>
>    ![output in the terminal after issuing the make serve-gitpod command](../../images/gitpod/command.png)
>
> 2. The first time you run this command, it will install some dependencies, so it may take a little bit of time (a few minutes)
>    - Next times will be a lot faster! (~30 seconds)
>
> 3. When the build process is finished, the terminal will show you a link, and a window should popup with a preview on the right.
>    - You may have to click the "Try Again" button if the preview was generated before the site building completed
>
>      ![gitpod with the GTN open in the right panel](../../images/gitpod/gitpod-launched.png)
>
>    - Click on the link in the terminal to see the GTN in full-screen: `Server address: http://127.0.0.1:4000/training-material/`
>
>    ![output in the terminal after the build process has completed ](../../images/gitpod/ready.png)
>
>    > <tip-title>Not opening?</tip-title>
>    > ![screenshot of firefox permissions dialog which shows blocked windows and an allow menu](../../images/gitpod_popup.png)
>    > Some browsers block popups by default, you may need to allow GitPod to show popups in order to click links.
>    {: .tip}
>
> 4. This will take you to a preview of the GTN website
>    - **Note:** It will take you the 404 page because GitPod doesn't know the location of the GTN homepage, but from here you can just click the homepage button.
>
{: .hands_on}


![screenshot of gitpod's code editor with a tutorial open, and the GTN preview on the right](../../images/gitpod.png)

# Editing Training Materials on GitPod

Now that you have the GitPod environment working and we have a live preview up, let's make some changes to the GTN materials and get an instant preview.


**Scenario:** You have spotted a typo in one of the tutorials, and would like to fix this and see the resulting GTN webpage.


> <hands-on-title>Make and view changes</hands-on-title>
>
> 1. In the preview of the GTN website, open the following tutorial:
>    - Topic: "Introduction to Galaxy Analyses" topic
>    - Tutorial: "A Short Introduction to Galaxy""
>    - We will edit this tutorial and watch the live preview window for the effects
>
>
> 2. On the file browser on the left, open the following file:
>
>    ```
>    topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>    ```
>
>    > <tip-title>Finding files faster?</tip-title>
>    > You can use the key combination <kbd>ctrl-p</kbd> to launch the file search dialog and simply enter the text `topics/introduction/tutorials/galaxy-intro-short/tutorial.md`
>    {: .tip}
>
> 3. Change the title of the tutorial
>    - **From:** "A Short Introduction to Galaxy"
>    - **To:** "A Short and Cool Introduction to Galaxy"
>
>    ![we changed the title of the tutorial in the text editor window](../../images/gitpod/changes.png)
>
>    - You should immediately see a message in the terminal saying "regenerating". GitPod has detected your changes and is rebuilding the website.
>
>    ![the terminal shows a message stating the website is being regenerated](../../images/gitpod/regenerating.png)
>
> 4. Move to the top right panel where the GTN is previewed and refresh the frame
>    - {% icon galaxy-refresh %} Refresh button in front of the address bar of the preview panel
>    - You can also open the preview in it's own brower tab, using the {% icon galaxy_instance %} button at the top-right corner of the preview window. Then you can reload the page the regular way (e.g. <kbd>F5</kbd> or <kbd>ctrl + r</kbd> or the reload button in the browser)
>
> 5. You should see the change you made:
>    ![The updated preview with our changed tutorial title](../../images/gitpod-changed.png)
>
{: .hands_on}


In this way you can edit files in the text editor, and see the effects in the website preview.


# Saving your changes back to GitHub

When you have finished your changes, it all looks good in the preview, you want to save your changes back to GitHub so that you can either continue later, or make a Pull Request to the GTN to contribute your changes.

> <hands-on-title>Save changes</hands-on-title>
>
> 1. **Option 1: via the terminal**
>    - Hit <kbd>ctrl+c</kbd> if your preview was still running to stop it
>    - Create a new branch, commit your changes, push changes:
>
>    ```bash
>    git checkout -b fix-title
>    git commit -m "update tutorial title" topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>    git push origin fix-title
>    ```
>
> 2. **Option 2: via the web interface**
>    - Create a new branch:
>      -  On the bottom-left, click on the branch logo (probably labelled "main")
>         ![button on the bottom of the page with the current branch](../../images/gitpod/branch-change.png)
>      - Enter `fix-title` as the name for your new branch (at top of window)
>      - Choose "+ Create new branch..." from the dropdown
>    - Commit changes:
>      - On the left menu, click on the "changed files" tab
>        ![button on the left menu to show changed files](../../images/gitpod/changedfiles.png)
>      - Click on the "+" icon next to the file we edited to *stage changes*
>        ![stage changes button](../../images/gitpod/stagechanges.png)
>      - Hit the checkmark icon at the top to commit the changes
>      - Enter a commit message (top of window)
>   - Publish changes
>     - Click the cloud button at bottom left to publish your changes
>       ![publish changes button](../../images/gitpod/publlish.png)
>
>
> 3. Your changes are now saved to your fork, and you can make a PR via the GitHub interface.
>    - If you used option 1 and want to do that right away, you can click on the link in the terminal:
>      ![link in the terminal to open a PR](../../images/gitpod/push.png)
>
> > <tip-title>GitPod Permissions</tip-title>
> > If this is your first time using GitPod, you may need to grant permissions to let GitPod push to your GitHub fork
> > ![prompt to changes permissions](../../images/gitpod/access.png)
> > This will take you through a process to authorize GitPod, and set permissions
> > ![permission settings for GitPod](../../images/gitpod/permissions.png)
> > If it doesn't take you to GitPod's permission page because your browser blocks popups, you can set them [on this page](https://gitpod.io/integrations).
> {: .tip}
{: .hands_on}


# Closing your GitPod Workspace

Everybody gets 50 hours per month for free on GitPod, so it is wise to close your GitPod workspace when you are finished with it, to conserve your quotum.

> <hands-on-title>Shutting down your GitPod workspace</hands-on-title>
>
> 1. Click on the menu button at the top left corner of the screen
>    - Choose "Gitpod: Stop Workspace"
>
>    ![stop workspace option in the main menu](../../images/gitpod/stopworkspace.png)
>
{: .hands_on}

</div>

<div class="Command-line" markdown="1">

If you want to run the entire GTN material website locally or test your new training material you can do this!

Currently, the website is generated from the metadata and the tutorials using Jekyll, a simple static site builder.
We can use Jekyll to run a server to check if the tutorial is correctly added and rendered.


# Installation of the requirements

The first step is to install the needed tools inside a conda environment. A conda environment is a directory that contains a specific collection of packages. For example here to run the website, we need ruby, pandas, requests, etc. By creating a conda environment and installing the needed tools there, we do not affect your main installation.

This step has to be done once.

We also need to make sure that a couple of other utilities and build requirements are present: **git**, **curl** & **make**. The easiest way to install these is with your package manager of choice - i.e. yum, apt, brew etc.

> <hands-on-title>Install the requirements</hands-on-title>
>
> 1. Open a Terminal
> 2. Use your package manager to install `git`, `curl` and `make`
>
>    - For Debian/Ubuntu: `sudo apt update && sudo apt install git curl make`
>    - For Fedora/CentOs/RedHat: `sudo yum install git curl make`
>
> 3. Clone the training material GitHub repository:
>    - `git clone --depth 1 --branch main https://github.com/galaxyproject/training-material.git`
>    - Note: The GTN repo is quite large, therefore we recommend to use the `--depth 1 --branch main` flags here. This will clone only the main branch, and only the most recent revision, rather than the full history.
> 4. Navigate to the the cloned repository
>    - `cd training-material`
> 5. Set up the conda environment (see tip box below for an method without conda)
>
>     It will install some needed tools (ruby, nodejs, etc) in a protected environment, without interfering with the existing tools or versions.
>
>     1. Install conda (if not already installed): `make install-conda`
>     2. (You may need to exit the terminal and re-open for conda to be recognised. Navigate back to the same place.)
>     3. Create the `galaxy_training_material` conda environment: `make create-env`
>
> 6. Install Jekyll and related modules into the conda environment: `make install`
>
> > <details-title>Troubleshooting `libxml2` errors</details-title>
> > If you encounter an error about libxml2 on Linux, please try to install `libxml2-dev` (executing `sudo apt install libxml2-dev`) if on Debian/Ubuntu or `libxml2-devel` (executing `sudo yum install libxml2-devel`) if on Fedora/RedHat/CentOS, and re-run `make install` .
> {: .details}
>
{: .hands_on}




> <tip-title>[Advanced] Building GTN without Conda (using Ruby's RVM)</tip-title>
>
> If you do not want to use conda, you can also run jekyll directly using Ruby and RVM:
> 1. Install RVM (Ruby Version Manager) on your system ([instructions](https://rvm.io/rvm/install))
>    - For Ubuntu:
>      ```
>      sudo apt-get install software-properties-common
>      sudo apt-add-repository -y ppa:rael-gc/rvm
>      sudo apt-get update
>      sudo apt-get install rvm
>      sudo usermod -a -G rvm $USER
>      ```
> 2. Install a Ruby version
>    ```
>    rvm install ruby-3.2.4
>    ```
> 3. Create ruby GTN environment
>    ```
>    rvm use 3.2.4@gtn --create
>    ```
> 4. Install dependencies
>    ```
>    bundle install
>    ```
> 5. Run the GTN
>    - Always make sure your ruby environment is activated `rvm use 3.2.4@gtn` and up-to-date `bundle install`
>    - Get a GTN preview
>      ```
>      bundle exec jekyll serve --incremental
>      ```
>    - Quicker build preview (but certain features are disabled)
>      ```
>      bundle exec jekyll serve --strict_front_matter -d _site/training-material --incremental --config _config.yml,_config-dev.yml
>      ```
> 6. Changes now showing?
>    - Stop the preview, run `make clean`, `bundle install`, and rebuild it
{: .tip}



# Checking the website generation

Once Jekyll and its modules are installed in our conda environment, we can check the generation of the website locally:

> <hands-on-title>Checking the website generation locally</hands-on-title>
>
> 1. Run a local Jekyll server with `make serve-quick` (if you use a Macbook, try `make serve-quick HOST=127.0.0.1` for forcing Jekyll to use the standard local loopback address.)
> 2. Visualize at [http://localhost:4000/training-material/ ](http://localhost:4000/training-material/)
> 3. Edit one of the tutorials:
>    - For example, open `topics/introduction/tutorials/galaxy-intro-peaks2genes/tutorial.md` in a text editor of your choice.
>    - Make some changes to the *Introduction* paragraph, and save the file.
>    - Refresh the tutorial page in your browser until you can see the changes you made.
>        - this may take a little bit of time; in the terminal you can monitor when the regeneration is complete
>
{: .hands_on}

With `make serve-quick`, a local Jekyll server will run in background. It will check the changes and regenerate the website accordingly. You may need to reload the page to see the changes (and sometimes to wait 1-2 minutes).


> <tip-title>make serve vs make serve-quick</tip-title>
>
> 1. Use `make serve` instead of `make serve-quick` to get all plugins, but also configure the post, host and pass additional flags. This however can be quite slow.
>
> 2. Need to speed up the cloning step? You coud fetch only the latest commit of the main branch:
>
>    ```
>    $ git clone {{ site.github_repository }}.git --depth 1 --branch main
>    ```
>
> 3. Running on a VM or remote machine?
>
>    If you are not running this on your local machine, but e.g. on a VM, you may need to configure a webserver to serve the website.
>
>    Below is an example NGINX configuration (e.g. in `/etc/nginx/sites-enabled/default`)
>
>    ```
>    location /training-material/ {
>      root /home/ubuntu/training-material/_site/;
>    }
>    ```
>
>    Change the `root` path above to wherever you cloned the training material folder
>
{: .tip}


# Stopping the server

Once you are done, you can stop the server and clean your repository.

> <hands-on-title>Stopping the server</hands-on-title>
>
> 1. Stop the server with <kbd>CTRL</kbd>-<kbd>C</kbd>
> 2. Clean the repository: `make clean`
>
{: .hands_on}

# Conclusion

</div>

<div class="container-based" markdown="1">

Using the container-based method, you can edit and preview GTN Training Material locally on you computer.

# Prerequisites

You will need the GTN Training Material source code and a container platform. If you wish to contribute your changes, you will also need git. The exact installation steps vary a little depending on your operating system.

For this tutorial, Docker will be used as container platform, as it is easy to use and available for all major operating systems.

Other container platforms should work as well. Beware with rootless container platform installations, you have to pay special attention to the user namespace mode and thus to the UID and GID configuration in order to preserve correct file ownership information.

## For Linux users

#### Docker

I recommend to install only Docker Engine (**not** Docker Desktop) for Linux users. Just follow the [official setup documentation](https://docs.docker.com/engine/install/) for your specific distribution.

An common practise is to assign your current user to the `docker` group. **Please do not!** This implies inherent security concerns, as it basically gives your current user unrestricted root access. Learn more: [https://docs.docker.com/engine/security/#docker-daemon-attack-surface](https://docs.docker.com/engine/security/#docker-daemon-attack-surface).

#### Git

Use the package manager of your specific distribution in order to install git. All major linux distributions do have git in their official package sources. Please consult your distros documentation for help with package installation.

###### Examples

```bash
# For Debian based distros, using apt:
sudo apt install git

# For Fedora based distros, using yum:
sudo yum install git

# For Fedora based distros, using dnf:
sudo dnf install git

# For Arch based distros, using pacman
sudo pacman -S git
```

## For Windows users

On windows you have multiple virtualization options in order to run linux containers. You should use the WSL2 method for various reasons. This tutorial is only tested for the WSL2 method only.

Additionally, to work with source code located within WSL2, I recommend installing VS Code.

#### WSL2

Please follow the [official setup documentation](https://learn.microsoft.com/en-us/windows/wsl/install) for installing WSL2.

###### TL;DR

Open PowerShell by right-clicking on the windows logo in taskbar and run:

```bash
wsl --install
```

#### Docker

Please follow the [official setup documentation](https://docs.docker.com/desktop/setup/install/windows-install/) for installing Docker Desktop on Windows.

#### Git

With the installation of Docker, an Ubuntu image has been installed on your system, which already includes git. No additional steps needed.

#### VS Code

Please follow the [official setup documentation](https://code.visualstudio.com/download) for installing VS Code. This is useful for windows users as it is one of the simplest methods to work with files located in the WSL2 (linux) file system.

## For MacOS users

#### Docker

Please follow the [official setup documentation](https://docs.docker.com/desktop/setup/install/mac-install/) for installing Docker Desktop on MacOS.

#### Git

Run a terminal and type a git command, e.g. `git --version`. If git is not installed on your system yet, MacOS will automatically prompt you to install it through Xcode. Just follow the on-screen instructions.

# Preparation

## Command line utilities

All following commands are meant to be entered in a shell. For Linux and MacOS, e.g. you can use a terminal application.

Windows users **must not** use the PowerShell, but the Ubuntu app that was automatically installed alongside WSL2 and Docker. If you are running it the first time, create a username and a password. Beware that files created in the Ubuntu app are located on a separate file system. If you need to access them from windows, you may open the explorer and choose the "Linux" Bookmark in the left sidebar. However, usually you should not need to access any of these files through windows.

## Git basics

In order to be able to contribute, you will need to have a Github account and you will need to create a fork of the GTN Training Material Repository. This can be simply done via Browser on the [GTN Training Material Github page](https://github.com/galaxyproject/training-material).

You will need to create a SSH key pair or provide other authentication vectors in order to push changes to your fork. Read more about this in the [official documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/about-ssh).

If this is your first time using git, please read the [Github contribution guide](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project).
## Clone the GTN Training Material Repository

For Linux and MacOS users, open your favorite terminal application. For Windows users, open the Ubuntu app which provides you with a terminal, as well. Run:

```bash
# You may want to replace the address with your forks address
# You may want to use SSH based clone if you want to contribute
git clone --depth 1 --branch main https://github.com/galaxyproject/training-material.git
```

## Configure environment

Switch into the cloned repository:

```bash
# "training-material" is the default name. If you have changed your forks name or if you provided a directory to clone into, please edit accordingly.
cd training-material
```

For Linux and MacOS users, you may open the directory in your favorite IDE.

For Windows users, if you have installed VS Code prior, run following command to open the directory in VS Code:

```bash
# For windows users
code .
```

Create a `.env` file. In this file, you may add several environment variables:

| Key | Description | Default |
| --- | --- | --- |
| COMPOSE_PROJECT_NAME | The project name to be used for composition, should be unique | gtn-dev |
| GTN_RUBY_TAG | The Ruby Tag to be used for the container | 3-bookworm |
| GTN_UID | The effective UID to use within the container, should match your current UID | 1000 |
| GTN_GID | The effective GID to use within the container, should match your current GID | 1000 |
| GTN_PORT | The Port on which GTN will be accessible, should not be already bound | 4000 |
| GTN_RELOAD_PORT | The Port on which the reload server will listen, should not be already bound | 4001 |

If you don't know your current UID and GID you may run:

```bash
# To find out your UID
id -u

# To find out your GID
id -g
```

Your `.env` should have a form like:

```bash
# KEY=VALUE
# e.g.:
COMPOSE_PROJECT_NAME=galaxytraining
```

# Using the composition

I recommend using the command line, but you can also do lifecycle operations via docker desktop after you have run the composition the first time.

Beware, the first time the composition is starting, it may take some time until everything is setup entirely.

When the composition has started, you may access the GTN Training Material on the port you provided, per default this is 4000: [http://localhost:4000/training-material/](http://localhost:4000/training-material/).

###### Starting

If you are done configuring, you may start the composition

```bash
# To start it with attached terminal
sudo docker compose up

# To start it with detached terminal
sudo docker compose up -d
```

If you have started the composition detached, you may retrieve the logs via

```bash
# -f will follow the logs, if you want to disconnect, press CTRL+C
sudo docker compose logs -f
```

###### Stopping

If you have started the composition with attached terminal, just press `CTRL+C`.

If you have started the composition detached, you may run:

```bash
sudo docker compose stop
```

###### Removing

If you want to remove associated containers and resources, you may run:

```bash
sudo docker compose down
```

If you have volumes involved, you may run this command with the `-v` flag in order to remove the volumes as well.

</div>


topic: contributing
tutorial: learning-principles
title: Principles of learning and how they apply to training and teaching
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["elixir", "train-the-trainers"]
content_hash: 35ae7eb25f2b0152474786d9bf16265e
last_modified: 2026-05-27T09:01:47.925030
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/learning-principles/tutorial.html

content:
In this tutorial we will discuss the principles of learning

> *Learning results from what the student does and thinks and only from what the student does and thinks. The teacher can advance learning only by influencing what the student does to learn*
>
> [H.A. Simon](https://en.wikipedia.org/wiki/Herbert_A._Simon) (one of the founders of the field of [Cognitive Science](https://en.wikipedia.org/wiki/Cognitive_science) and Nobel Laureate)
{: .quote}

This quotation from Herbert A. Simon clearly indicates that we cannot talk about teaching, teaching practices or effective teaching techniques if we don't understand first how people learn.

In this lesson, we want to share with you what we have collectively learnt about how learning works. Our knowledge in the field of cognition and learning comes from diving into pedagogical and cognitive research results, reading books such as ["How Learning Works"](https://www.amazon.it/How-Learning-Works-Research-Based-Principles/dp/0470484101) ({% cite ambrose2010learning %}) or ["Small Teaching"](https://www.wiley.com/en-gb/Small+Teaching%3A+Everyday+Lessons+from+the+Science+of+Learning%2C+2nd+Edition-p-9781119755555) ({% cite smallTeaching2021 %}) and many others, studying research articles published in the field of cognitive science and educational psychology, attending instructor training courses (like this one), and - last but not least - from our own experience as both instructors/teachers, learners, and passionate observers of learning processes and teaching practices.

Since this material cannot cover the broad literature on the subject of learning we strongly invite you to commit to read at least one book on how learning works ({% cite ambrose2010learning %}, {% cite smallTeaching2021 %}), have a look at the [Software Carpentry Instructor training materials](http://swcarpentry.github.io/instructor-training/) and explore the material available through the [ELIXIR Train-the-Trainer course](https://elixir-europe.org/platforms/training/train-the-trainer).

> <comment-title>Sources</comment-title>
>
> This tutorial is significantly based on {% cite via2020course %}, session one
> of ELIXIR-GOBLET Train the Trainer curriculum.
{: .comment}

<!--## Instruction design in 5 Phases

So how can we now use the knowledge we have gathered during this time in the course? Here we will go through the Instructional design in 5 phases and explain why learning outcomes (to articulate and specify these) is so important for the curricula and the course design process.

![A flow chart of the five phases of the instructional design. The 1st phase is to select the Learning Outcomes, followed by the selection of the corresponding Learning Experiences. The third phase is to select the content that is relevant to the LO. The fourth phase is the development of the necessary assessments for the learners, with the final phase being the evaluation of the effectiveness of the LE and the LO, at which point the flow connects back to the first phase for the refinement of the process.](images/PhasesInstructionalDesign.png "5 phases of instructional design described by Nicholls et al in Developing teaching and learning in higher education. London, UK: Routledge. pp 51-75")

1. **Writing LOs**: Important to build the training
2. **Selection learning experiences**: Focus on LOs
3. **Selection of content relevant to LOs**: we usually (traditionally) starting by the contentm, but here only 3rd step
4. **Develop assessment**

    Example: formative assessment as the previous challenge

5. **Evaluation** of your training

    Example
    * Final exams at university
    * Final feedback
-->

# How do people learn?

So let's start with "How do people learn and how does learning work"

## General definition of learning

(Relatively) permanent changes in behaviour, skills, knowledge, or attitudes resulting from identifiable psychological or social experiences.

* A key feature of learning: permanence
* The change has to last
* Changes do not count as learning if they are temporary
* Example: learning a phone number

Why is it important to learn about how people learn?

* It is common to confuse our effort (i.e. our teaching) with what learners get from our efforts (i.e. their learning)
    * We usually assume learners learnt
    * Nevertheless, the distinction between learning and teaching is especially important for teachers to remember and it is essential that teachers become aware of the fact that the way they think about learning affects the way they teach and that the very circumstances of teaching can influence teachers’ perceptions of learning, and therefore also influence how they teach.
    * For example, imagine that a cognitive scientist convinced you that people don’t retain anything of what you tell them after 20 minutes of passively listening to you. Would you keep lecturing for more than 20 minutes?
    * And how would you teach if, in contrast, you assumed that the mind is a vessel to be filled?
    * Moreover, most teachers tend to think that what is taught is equivalent to what is learned, namely that what they teach is the same as what students understand or retain of what they teach  (even though most of them know that assuming this is a mistake and that teaching and learning can be quite different).
* Learning how learning works helps clarify this distinction
* The way we think about learning affects the way we teach
* Understanding  how people learn makes you a more aware and therefore better teacher
* You  can use such knowledge to improve your day-to-day teaching

According the **Meaningful Learning Theory**, developed by the psychologist David Ausubel, meaningful learning generally arises from a process of learning to learn. He considers that if meaningful learning is achieved, it transcends the rote repetition of unconnected content and allows for the construction of meaning, making sense of what has been learned, and understanding its scope of application and relevance in academic and everyday situations.


## Learning metaphors, theories and models

Learning - similarly to thinking - is a complex process that requires multiple approaches spanning many different disciplines, ideas, theories and models to be understood.

Many learning theories, models, concept and ideas from from the **Science of Learning**, **Educational Psychology**, and the **Cognitive Sciences** are relevant to classrooms, in that they describe at least some of what usually happens there and offer guidance for assisting learning.

This session will provide you with some understanding of how learning works but it is far from being exhaustive. In particular, we will present and discuss learning models, concepts and perspectives that are important for you to know as they suggest things that you might do in your classroom to make learners’ learning more productive and your teaching more effective and mindful.

A metaphor is a word from one thematic domain that has been embedded in another one, thus entering a network of new linguistic relations. This is, indeed, what metaphors are all about: they are transplants from one discourse to another. They are a source of new ways of seeing things. "Metaphors of learning" are a source of new ways of seeing learning.

We can use metaphors to shape and reveal our way of thinking about learning and, therefore, shape our actions as teachers.

---

> A mind is a fire to be kindled, not a vessel to be filled
>
{: .quote author="Plutard"}

---

How the mind-as-a-vessel-to-be-filled metaphor may affect your way of teaching? You are likely to spend your time in the class at the blackboard, trying to 'transmit' to the students your own knowledge.

<!--{% include _includes/quiz.html id="learning_metaphors.yaml" %}-->


> <hands-on-title>Learning metaphors - {% icon time %} 5 min</hands-on-title>
>
> 1. Think about the following metaphors "A mind is a fire to be kindled". How this metaphors may affect your way of teaching?
> 2. Think about the following metaphors "Teaching is growing a garden". How this metaphors may affect your way of teaching?
>
{: .hands_on}


## Metaphors define the role of the teacher and the learners

Metaphor | Teacher | Learner
--- | --- | ---
Learning is knowledge transfer | Defines, develops and prepares contents | Receivers and memorizes
Learning metaphors | Prepares the environment, guides the exploration | Explores, works on information, attributes meanings
Learning is participation in social practices | Community that shares activities and social practices | As teacher

# Learning theories

Learning theories describes how students receive, process, and retain knowledge during learning. However, there is no single one theory (as in physics for example). There is a massive catalogue of learning theories out there and there is no universal theory of learning.

![The figure shows a dendrogram of the main learning theories that are currently available. The information is organized across four main levels; the Scientific Discipline (such as Organization, Education, Philosophy, etc), the Learning Theorists (such as Montessori and Bloom), the Learning Paradigms (such as Behaviourism, Constructivism), and the key concepts.](images/LearningTheories.png "This is a diagram representing the landscape of many known learning theories available.")

Philosphers and researchers from multiple disciplnes have proposed a large number of learning theories: a flavour of the diversity of approaches and philosophies can be gained, for example, from the [HoTEL](http://cmapspublic3.ihmc.us/rid=1LNV3H2J9-HWSVMQ-13LH/Learning%20Theory.cmap) (Holistic approach to Technology-Enhanced Learning) website. But which should we rely on? Which should we learn, teach and apply?

There is no universal theory of learning, but there are interesting elements for classroom teaching in many different theories There are elements of truth in different theories, and **evidence-based** research results that support learning principles recognized and applied today. In this sense there are interesting elements for a teacher in many different theories.

In the following sections we will explore the most important learning theories.

> <comment-title>Principal learning theories</comment-title>
>
> We will briefly describe the key ideas from each of these perspectives and explain how you can use them in your classroom to support learners' learning.
>
> #### Behaviourist Learning Theory
>
> **Behaviorism** is a psychological movement that posits behavior should be studied as a natural science akin to fields like chemistry or physics. It conceives learning as changes in overt behavior. In this context, learning is acquiring new behaviours by conditioning, reflex response to stimuli, reward/punishment.
>
> An important concept from this theory is **operant conditioning**, a variety of behaviourism that focuses on how the consequences of a behaviour affect the behaviour over time.
>
> This theory is based on the idea that certain consequences (reinforcement or inhibition of a behaviour) tend to make a behaviour happen more or less frequently. According to the behaviorist perspective on learning, the instructor assumes a dominant role within the classroom, exerting complete control over the learning process. Evaluation of learning is carried out by the teacher, who determines what is considered correct or incorrect.
>  
> #### Cognitivist Learning Theory
>
>
> Diverse authors agree that, in order to implement adequate online teaching practices, it is necessary to take into account the characteristics of mental processes ({% cite Wu1998 %}, {% cite Chandler2004 %}).
>
> **Cognitivism** emerged in the 1950s as a response to Behaviorism, which was deemed inadequate in explaining cognition. **Cognitive Learning Theory** aims to describe the links between cognitive structures -defined as the mental representations of objects or ideas- and the learning process ({% cite Qin2009 %}, {% cite vanderVeer1988 %}), with a special focus on the processing of information. Two important theories derive from this one: **Information Processing Theory** and **Cognitive load theory**:
>
> The **Information Processing Theory** describes learning as the result of sequential processing of information, which involve three types of memory: immediate memory, working memory and long term memory ({% cite Gentner2014 %}). According to this theory, adequate learning resources require to assume two key elements: firstly, the fact that working memory is limited; and secondly, that the interaction between working and long term memory plays an important role in the construction and transferability of knowledge ({% cite Chandler2004 %}, {% cite Herring1993 %}, {% cite Yilmaz2011 %}).
>
> On the other hand, **Cognitive Load Theory** seeks specifically to address the efficiency with which information is processed. It states that the verbal and visual information is processed by independent cognitive structures, both of them with a limited capacity ({% cite Fori2012 %}, {% cite Via2013 %}, {% cite Kats2010 %}, {% cite Wright2010 %}). An interesting concept derived from this theory is the cognitive overload, which refers to those situations in which the information flow exceeds the learner’s processing capacity, resulting in an inhibition of the learning process ({% cite Aidi2009 %}).
>
>
> #### Constructivist Learning Theory
>
> **Constructivism** holds that learning is essentially active; a person who learns something new incorporates it into their previous experiences and their own mental structures. Each new piece of information is assimilated and deposited into a network of previously existing knowledge and experiences  ({% cite Abbot1999 %}). All these ideas have been taken from different nuances, and two authors stand out: Jean Piaget with **psychological constructivism** and Lev Vigotsky with **social constructivism**.
>
>From the perspective of **psychological constructivism**, learning is fundamentally a personal matter. It maintains the idea that the individual, both through cognitive and social aspects of knowledge as well as affective ones, is not a mere product of the environment or a simple result of his or her internal dispositions, but a construction of his or her own that is produced day by day as a result of the interaction between these two factors. The driving force of this activity would be cognitive conflict; in any constructivist activity there must be a circumstance that shakes the previous structures of knowledge and forces a rearrangement of the old knowledge in order to assimilate the new.
>
>According to the **social approach of Lev Vigotsky**, learning is that process in which the individual assimilates certain historical-cultural experience at the same time as he/she appropriates it, which requires an active subject who gives meaning to this experience, transforming it into his/her subjectivity ({% cite RodrguezRuiz2005 %}). 
>
> The aim of Vigotsky's theory is to discover and stimulate the **zone of potential development** or zone of proximal development in each student, which is defined as the distance between the actual level of development, determined by the ability to independently solve a problem, and the level of development potential, determined through solving a problem in collaboration with a more able peer ({% cite Moreira1997 %}).
>
> For Vigotsky, cognitive development cannot be understood without reference to the social, historical and cultural context in which it occurs. According to this author, higher mental processes (thought, language,voluntary behaviour) have their origin in social processes; cognitive development would be the conversion of social relations into mental functions ({% cite Moreira1997 %}).
>
> #### Critical Learning Theory
>
> The **Critical Learning Theory**, also known as Transformative Learning Theory, aims to define the mechanisms involved in the development of critical thinking during the learning process. It considers that thought must be capable of questioning even the logic of reasoning itself, that is, it must be capable of problematising what has hitherto been treated as self-evident, of turning into an object of reflection what had previously been simply a tool. According to {% cite Marcuse1964 %}, critical thinking aims to define the irrational character of established rationality.
>
> According this theory, learning occurs when the learning process induces a shift in the student's frame of reference. Although this goal can be achieved in multiple ways, a successful strategy is to stimulate reflection on uncritically accepted assumptions. It is the educator responsibility to cultivate students’ skills related to autonomous thinking by designing activities for that purpose.
>
> Critical Learning Theory emphasises two elements: the importance of the subjective role of each learner in finding meaning for him/herself, and how social and economic conditions constrain and distort the social construction of meaning ({% cite Giroux2018 %}).
>
{: .comment}


> <hands-on-title>How do you go about learning something new? / How do you approach learning new things? - {% icon time %} 5 min</hands-on-title>
>
> - Order the following approaches (from the most to the less effective one) for you when you want to learn something new
>	* Read about it
>   * Attend a training session !
>   * Have a go ?
>   * Do, reflect, process, further understand?
> - Write if you recognise elements of one of more learning perspectives (constructivism / behaviourism) in your way of learning.
>
{: .hands_on}


# Principal learning theories: practical examples

In this section we will introduce some examples of practical applications from the different learning theories:

Learning theory | Example
--- | ---
Behaviorist | Dreyfuss model for skill acquisition.
Cognitivist | Human Memory and Learning from cognitive sciences
Constructivist | Bloom's Taxonomy
Critical | The hidden curriculum

## Behaviorist Learning theory: Dreyfuss model for skill acquisition

The Dreyfus model of skill acquisition presents a framework illustrating the process by which learners acquire skills through a combination of formal instruction and dedicated practice. It was presented by Hubert and Stuart Dreyfus in 1980 in a report on their research at the University of California, Berkeley, Operations Research Center for the United States Air Force Office of Scientific Research. Hubert Lederer Dreyfus (1929-2017) was a prominent American philosopher and a professor of philosophy. His areas of expertise encompassed phenomenology, existentialism, and the philosophy of psychology, literature, and artificial intelligence, exploring their profound philosophical implications. Stuart E. Dreyfus holds the position of professor emeritus in the Industrial Engineering and Operations Research Department at the University of California.

The Dreyfus model for skills acquisition by brothers Stuart and Huber Dreyfus uses a 5 stage from novice to expert.

![A representation of the Dreyfuss model, as a gradient bar. Starting from the left, is the Novice which relies primarily on rules to organize their skills. The Novice, with experience, transitions to Advanced Beginner and then competent, increasing the use of their intuition along with the rules. The next two stages are Proficient and Expert (at the right end of the bar), which know relies almost exclusively on their intuition.](./images/DreyfussModel.png "Dreyfuss model")

The stages follow a progression from rigid adherence (novice) to rules to an intuitive mode of reasoning based on tacit knowledge "silent knowledge" (expert).

The fundamental idea is that when teaching a concept, you have to tailor the style of teaching to where the learner is in their understanding and that progression follows a common pattern. Early stages of learning focus on concrete steps to imitate, the focus then shifts to understanding principles and finally into self-directed innovation.

The skill level is tied to a mental function transitioning from recollection (non-situational or situational), recognition (decomposed or holistic), decision (analytical or intuitive), and awareness (monitoring or absorbed)

### Example of Dreyfus model: the Carpentries

The Carpentries (a global organization building global capacity in essential data and computational skills for research) uses the Dreyfus model as its global [approach](https://carpentries.github.io/instructor-training/02-practice-learning/index.html#the-acquisition-of-skill) for skill acquisition:

![A simplified version of the Dreyfus model, as built by the Carpentries. There are three levels; the Novice, the Competent Practicioner and the Expert.](./images/DreyfussModelCarpentries.jpg "Carpentries build on the Dreyfus model. Source: The Carpentries Instructor training")

- **Novice**: someone who does not know what they do not know, i.e., they do not yet know what the key ideas in the domain are or how they relate. Novices may have difficulty formulating questions, or may ask questions that seem irrelevant or off-topic as they rely on prior knowledge, without knowing what is or is not related yet.

    Example: A novice learner in a Carpentries workshop might never have heard of the bash shell, and therefore may have no understanding of how it relates to their file system or other programs on their computer.

- **Competent practitioner**: someone who has enough understanding for everyday purposes. They will not know all the details of how something works and their understanding may not be entirely accurate, but it is sufficient for completing normal tasks with normal effort under normal circumstances.

    Example: A competent practitioner in a Carpentries workshop might have used the shell before and understand how to move around directories and use individual programs, but they might not understand how they can fit these programs together to build scripts and automate large tasks.

- **Expert**: someone who can easily handle situations that are out of the ordinary.

    Example: An expert in a Carpentries workshop may have experience writing and running shell scripts and, when presented with a problem, immediately sees how these skills can be used to solve the problem.

> <details-title>The Carpentries</details-title>
>
> [The Carpentries](https://carpentries.org/) was started by Greg Wilson and Brent Gorda and they coined the term "The Carpentries" for enabling researchers to do "computational thinking" and use tools of computation effectively.
>
> The Style of the Carpentries are that teachers are where the learners are e.g. by doing live-coding together.
{: .details}


## Cognitivist Learning Theory: Memory and learning

In a simple model from cognitive science, learning happens when information moves from **working memory** to **long-term memory**. This process can be impacted by **environment**

![A figure showning a simple representation of how memory works. The main interaction is between the two central boxes, representing the working memory and the long-term memory. The working memory maintain the active consiousness, when paying attention to the envnironment, and is extremely fast but rather limited in capacity (experiments have shown that it can keep track of 7+/-2 items). The long-term memory is much larger in capacity, but it's significantly slower to access - using a process called "remembering". At the same time, the process of depositing information from the working memory into the long-term memory, is the learning process.)](./images/HowMemoryWorks.png "Simple model (brain is much more complex)")

<!--1. Environment
2. Working memory
3. Long-term memory
4. Working memory: fast
	* Ex: phone number
5. Long term: slow
	* How much effort to remember a full poem-->

 
> <hands-on-title>How short is your short term memory? - {% icon time %} 10 min</hands-on-title>
>
> Based on {% cite Willingham2009 %}.
>
> 1. Read the following letter combination:
>  
>    > <solution-title>Hide</solution-title>
>    >
>    >    ```APH D BDN A CKG B DCI A```
>    >
>    {: .solution}
>
>   1. Hide them after 10 seconds
>   2. Take a piece of paper and write what you remember
>   3. Unhide them after 2 minutes
>   4. Check how many consecutive letters you can remember
>
> 2. Read the following letter combination
>
>    > <solution-title>Hide</solution-title>
>    >
>    >    ```A   PHD   B   DNA   C   KGB   D   CIA```
>    >
>    {: .solution}
>
>   1. Hide them after 10 seconds
>   2. Take a piece of paper and write what you remember
>   3. Unhide them after 2 minutes
>   4. Check how many consecutive letters you can remember
>
{: .hands_on}

<!--{% include _includes/quiz.html id="short_term_memory.yaml" %}-->

The amount of space in working memory does not depend on the number of letters. It depends on the **number of meaningful objects**. In the 2nd round, the meaning of the words (long-term memory) or connecting words with a story helps to remember. It make room in working memory.

To facilitate learning, we as instructors have to make room in working memory.

### What can we do to make room in working memory?

<!--{% include _includes/quiz.html id="short_term_memory.yaml" %}-->

> <hands-on-title>How can you avoid to overload learners working memory?  - *{% icon details %} 5 min - Groups of 2/3 persons*</hands-on-title>
>
> - Discuss one thing a teacher could do to avoid overloading their students' working memory
> - Write your proposal in the shared notes
{: .hands_on} 

What can we do to make room in working memory?
* Chunking

    Example: Short theoretical sessions alternated with practical sessions to allow the processing of the material

* Avoid extraneous **cognitive load**

    Example: Avoid too crowded slides

### Cognitive load

There are 3 cognitive loads recognized by cognitive science:

* **Intrinsic cognitive load**: effort associated with a specific topic.
* **Germane cognitive load**: the (desirable) mental effort required to create linkages between new information and old.
* **Extraneous cognitive load:** everything else that distracts or gets in the way. It refers to the way information or tasks are presented to a learner.

### Attention split effect

Split-attention occurs when learners are required to **split their attention** between **at least two sources of information** that have been separated either spatially or temporally

> <hands-on-title>Extraneous cognitive load - {% icon time %} 10 min - Groups of 2/3 persons</hands-on-title>
>
> * Pick one task you teach in your lessons/courses
> * Think and discuss within the room about the following questions
>     * What would be the extraneous load in performing this tak?
>     * How can you avoid it?
{: .hands_on}


### Six strategies for effective learning from cognitive psychology

**Six strategies for effective learning** (from "Understanding how we learn" {% cite weinstein2018understanding %}) are based on evidence from cognitive research.

![Six circles representing the six Strategies for Effective Learning.](./images/StrategiesEffectiveLearning.png "Six Strategies for Effective Learning {% cite weinstein2018understanding %} Six Strategies for Effective Learning by [Yana Weinstein, Megan Smith, & Oliver Caviglioli](http://www.learningscientists.org/downloadable-materials/) / [Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](http://creativecommons.org/licenses/by-nc-nd/4.0/), based on a work at[http://www.learningscientists.org](http://www.learningscientists.org/), [teachinghow2s.com/cogsci](http://teachinghow2s.com/cogsci).")

- **Spaced practice**: Creating a study schedule that spreads study activities out over time

    Spaced practice is the exact opposite of cramming. When you cram, you study for a long, intense period of time close to an exam. When you space your learning, you take that same amount of study time, and spread it out across a much longer period of time. Doing it this way, that same amount of study time will produce more _long-lasting learning._

- **Interleaving**: Switching between topics while studying

    You shouldn't study one idea, topic, or type of problem for too long. Instead, you should change it up often. Especially good for problem-solving topics like bioinformatics. Then you should jump from one topic to another and go back to the previous one, with the aim of developing the connections among topics.

- **Elaboration**: Asking and explaining why and how things work

    It can be used to mean a lot of different things. It involves explaining and describing ideas with many details. Elaboration also involves making connections among ideas you are trying to learn and connecting the material to your own experiences, memories, and day-to-day life.

- **Concrete examples**: When studying / explaining abstract concepts, illustrating them with specific examples

    The idea is to take an abstract concept and use real world examples to increase understanding and 2 examples are better than 1.

- **Dual coding**: Combining worlds with visuals

    Dual coding is combining words and visuals such as pictures, diagrams, graphic organizers, and so on. The idea is to provide two different representations of the information, both visual and verbal. Adding visuals to a verbal description can make the presented ideas more concrete, and provides two ways of understanding the presented ideas. The visuals should be meaningful.

- **Retrieval practice**: Bringing learned information to mind from long term memory

    Concept maps are examples of a retrieval practice when you bring out knowledge/memories from long-term memory

> <hands-on-title>How do you understand the 6 strategies for effective learning? - {% icon time %} 15 min - Groups of 2 persons</hands-on-title>
>
> - Pick up one strategy per group
> - Discuss about the strategy
>   - Did you understand it the same way?
>   - Do you have any questions?
> - Find out an example on how you would implement it as a teacher/instructor (what would you do as a teacher in order to facilitate learning based on the strategy at hand)
{: .hands_on}

We highly recommend to look at the materials in a book and a website and blog as they are intended to teach about principles of learning and to provide teachers and students with flexible guiding principles to guide learning and studying. In ["Training techniques to enhance learner participation and engagement" tutorial]({% link topics/teaching/tutorials/learner_participation_engagement/tutorial.md %}), a number of teaching practices are described. You should try to reconnect them to these learning strategies, i.e. try to understand how the proposed teaching practices facilitate learning based on these six strategies.

### Seven learning principles from cognitive sciences

From ["How Learning works"](https://www.academia.edu/6743627/How_Learning_Works_Seven_Research_Based_Principles_for_Smart_Teaching_Susan_A_Ambrose_Michael_W_Bridges_Michele_DiPietro_Marsha_C_Lovett_Marie_K_Norman_with_a_Foreword_by_Richard_E_Mayer) {% cite ambrose2010learning %}

* **Principle P1**: Students' **prior knowledge** can help or hinder learning.

    Hindrances include prior knowledge which may be accurate but insufficient to support learning new material. Inappropriate for the specific class, thus distorting comprehension of new material; or, completely inaccurate.

* **Principle P2**: How students **organise knowledge** influences how they learn and apply what they know.

    Generally, people organize knowledge "as a function of their experience, the nature of their knowledge, and the role that that knowledge plays in their lives"

* **Principle P3**: Students **motivation** determines, directs and sustains what they do learn.

    {% cite ambrose2010learning %} define motivation as "the personal investment that an individual has in reaching a desired state or outcome" and identify three core concepts for understanding motivation
    - the subjective value of a goal
    - the expectancies for successfully reaching the goal
    - ??

* **Principle P4**: To develop mastery, students must acquire **component skills**, practice **integrating** them, and  know when to **apply** what they have learned

    Mastery refers to "the attainment of a high degree of competence with a particular area". It could be either beneficial or detrimental.

* **Principle P5**: **Goal-directed** practice coupled with **targeted feedback** enhances the quality of students' learning

    Learning and performance are advanced when students employ a methodology that
    - focuses on a specific performance goal
    - pinpoints a suitable level of challenge relative to students’ current performance,
    - is in an amount and frequency sufficient to meet the performance criteria.

* **Principle P6**: Students' current level of development interacts with the **social, emotional, and intellectual climate** of the course to impact learning

    {% cite ambrose2010learning %} maintain that "student-centered teaching requires us to teach students, not content".

* **Principle P7**: To become self-directed learners, students must learn to **monitor and adjust** their approaches to learning.

    This principle relates to metacognition, or "the process of reflecting on and directing one's own thinking". You can find more information on the connection of metacognition to student learning and performance here {% cite Stanton2021 %}.

    Students develop metacognitive skills by learning to
    - assess the task at hand,
    - evaluate their own strengths and weaknesses,
    - plan appropriate approaches to complete the task,
    - apply strategies to monitor their performance, while making adjustments to the selected approach.

<!-- 
> <hands-on-title>How do you understand the 7 principles of learning?  - {% icon time %} 15 min - Groups of 2/3 person</hands-on-title>
>
> * Read and reflect together the principles
> * Pick one principle that is not (fully) clear to you
> * Ask one question about this principle
> * Write here any insights/thoughts/comments to share from your breakout room
{: .hands_on}-->

> <details-title>Resources to learn more about the Cognitive Learning Theory</details-title>
>
> There is a great deal of literature on how learning works.
>
> The reason why we mention these three books is that they show how to integrate effective, research based strategies for learning into classroom practice.
>
> Educational practice usually does not rely on research findings. People tend to rely on their intuitions about what’s best for learning. But relying on intuition may be a bad idea for teachers and learners alike.
>
> - ["How Learning Works"](https://www.amazon.it/How-Learning-Works-Research-Based-Principles/dp/0470484101) ({% cite ambrose2010learning %})
>
>    This book presents a set of evidence-based principles for how to help people learn that is grounded in cognitive theory (that is, the science of instruction). It tries to help teachers understand how research in the science of learning can improve their reaching.
>
>    The book is organised around seven learning principles. Each principle is based on research evidence from the science of learning and the science of instruction.
>
> - ["Small Teaching"](https://www.wiley.com/en-gb/Small+Teaching%3A+Everyday+Lessons+from+the+Science+of+Learning%2C+2nd+Edition-p-9781119755555) ({% cite smallTeaching2021 %})
>
>    Cognitive psychologists, neuroscientists, and biologists all have produced a revealing body of research over the past several decades on how human beings learn, but often translating these findings into the classroom is overwhelming for busy instructors.
>
>    "Small Teaching" bridges the gap between research and practice  by providing a fully developed strategy for making deliberate, structured, and incremental steps towards tuning into how students are hardwired to learn.
>
>    The book introduces small activities that require minimal preparation and grading. The models presented are specifically designed to be used as both one-time experiences to innovate a course session or unit plan as well as a menu of options that can be combined into an entirely new teaching approach.
>
> - "Understanding how we learn" {% cite weinstein2018understanding %}
>
>    By the "learning scientists" Weinstein and Sumeracki (and illustrated by Oliver Caviglioli), the books examines the cognitive psychology's application to education
>
>   It is designed to convey the concepts of research to the reality of an instructor's classroom. The book proposes 6 strategies for effective learning (based on evidence from cognitive research). It is structured in four parts:
>   - Evidence-based education and the science of learning
>   - Basics of human cognitive processes
>   - Strategies for effective learning
>   - Tips for students, teachers, and parents
>
> - [The Carpentries Instructor Training materials](https://carpentries.github.io/instructor-training/)
>
>   The Carpentries is a community of volunteers teaching computational and data management skills worldwide. The community also developed training materials to teach pedagogical skills. Much of the GTN TtT was inspired by the pedagogical materials from the Carpentries and the ELIXIR TtT.
>
{: .details}


<!-- 
> <hands-on-title>What is learning in your opinion/experience? How does learning occur? - {% icon time %} 5 min</hands-on-title>
>
> _Break-out_ room (2 ppl per room)
>
> We follow the first challenge with another one. Now IN PAIRS in breakout rooms (Zoom). Here the reflection is on LEARNING and what learning is in your experience. Write your thoughts, one or more sentences, in the shared Google Doc.
> 	* Discuss your reflections on learning and what learning is in your experience
>	* Write here any insights/thoughts/comments to share from your breakout room
{: .hands_on} -->


## Constructivist Learning Theory: The Bloom's Taxonomy

And one of the most used "models" or concepts is the one based on the Bloom’s six categories of cognitive skills. This tool is important to develop training, but not only.

**Bloom's taxonomy** is a set of six categories of cognitive skills, that are accompanied by actionable verb.

![The pyramid of Blooms Taxonomy of verbs. Starting from the bottom layer, the corresponding aspects are: remembering, understanding, applying, analyzing, evaluating and creating.](./images/bloomsTaxonomyPyramid.png "Source: https://tips.uark.edu/blooms-taxonomy-verb-chart/")

From bottom to top:
* **Remembering (original Knowledge):** involves recognizing or remembering facts, terms, basic concepts, or answers without necessarily understanding what they mean.
* **Understand (original comprehension):** involves demonstrating an understanding of facts and ideas by organizing, summarizing, translating, generalizing, giving descriptions, and stating the main ideas.
* **Apply/Application:** involves using acquired knowledge—solving problems in new situations by applying acquired knowledge, facts, techniques and rules. Learners should be able to use prior knowledge to solve problems, identify connections and relationships and how they apply in new situations.
* **Analyze:** involves examining and breaking information into component parts, determining how the parts relate to one another, identifying motives or causes, making inferences, and finding evidence to support generalizations.
* **Evaluate:** involves presenting and defending opinions by making judgments about information, the validity of ideas, or quality of work based on a set of criteria
* **Create (original synthesize):**involves building a structure or pattern from diverse elements; it also refers to the act of putting parts together to form a whole.

It is important to highlight that, under this framework, **critical thinking is considered a combination of analyzing, evaluating and creating**, i.e. the top three layers of the taxonomy. Blooms taxonomy levels are very useful to align instruction with learners' level of complexity of thinking as well as experience.

What are the uses of this tool?
- For learners
    - Help to structure their knowledge
- For instructors
    - Knowing at which level of cognitive complexity you want to teach
    - Write Learning Outcomes

        ![6 levels of Bloom's taxonomy (and their verbs) from bottom (lower-order thinking skills) to top (higher-order 'critical' thinking skills): Remember (recall or reiterate information) with memorize, recognise, identify, describe, reproduce, list, define, label, name, state, outline, order, arrange; Understand (demonstrate understanding of facts) with distinguish, classify, select, review, discuss, indicate, explain, estimate, locate, summarise; Apply (apply knowledge to real situations) with operate, manipulate, experiment, choose, modify, prepare, produce; Analyze (resolve ideas into simple parts, identify patterns) with calculate, examine, model, test, break down, infer, predict, solve; Synthesise (pull ideas into a coherent whole, create new ideas) with combine, formulate, illustrate, imagine, design, invent, compose; Evaluate (make & defend judgements, assess theories & outcomes) with critique, compare ideas, solve, recommend, rate](./images/bloomsTaxonomy.png "ACTION verbs express levels of cognitive complexity")


    - Design instruction and learning experiences i.e. how you are to teach/train, which infrastructure and how to present the materials etc
    - Assess learning (taking place during curricula or course and after)
    - Align instruction with learners' level of complexity of thinking as well as experience.


## Critical Learning Theory: The Hidden Curriculum Theory

The concept of hidden curriculum refers to the learning that is incorporated by students though these aspects do not appear explicitly in the official curriculum, relying on the organisational aspects of classroom life that are generally not perceived by either students or teachers ({% cite Giroux2018 %}). 

According the Hidden Curriculum Theory, what students learn from the formally sanctioned content of the curriculum is much less important than what they learn from the ideological assumptions embodied in the communicative systems. Through the hidden curriculum, students would internalise values that emphasise respect for authority, punctuality, cleanliness, docility and conformity ({% cite Giroux2018 %}). It considers that elements such as exposure to a core of common subjects would lead to a dissolution of individuality in the mass, in the collective, to a standardisation and homogenisation of psychologies ({% cite Olivo2005 %}).

According the Hidden Curriculum Theory, if there is really an interest in **fostering critical thinking**, the **discursive mode that should dominate in the classroom should be the interrogative one**. It considers that questioning is a way of thinking, and that is why thinking is not only deriving conclusions, but often consists of the opposite: starting from the accepted conclusions of a community and turning them upside down ({% cite Sztajnszrajber2013 %}).

According this theory, technological tools, due to their design, do not act just as neutral means for transmitting information, but they also transmit values and habits of thought ({% cite Anders1980 %}). As {% cite Giroux2018 %} notes, no matter how progressive an approach to critical thinking may be, it will squander its own potential if it operates on the basis of a web of social relations in the classroom that are authoritarianly hierarchical and promote passivity, docility and silence.

# Mental models

**Mental model** is a collection of concepts and facts, along with the relationships between those concepts, that a person has about a topic or field. It is how people organize their knowledge

You can think of mental models as a graph with nodes (facts, concepts) and edges (connections), as seen in the figure of the mental model of animals below:

![A dendrogram showing the various descriptors of an animal, and the relationships between them. For example, the box "animal" is connected to the entities "eats", "breathes" and "can move", as well as sub-entities, such as "bird" and "fish". Each one of these, in turn, is connected to even more specific sub-entities (such as "penguin" for "bird" and "salmon" for "fish").](./images/animalMentalModel.png "mental model of animals")

The mental model of an expert in any given subject will be far larger and more complex than that of a novice, including both more concepts and more detailed and numerous relationships. Linking with Dreyfus model:

- A novice has a minimal mental model of surface features of the domain. Inaccuracies based on limited prior knowledge may interfere with adding new information. Predictions are likely to borrow heavily from mental models of other domains which seem superficially similar.
- A competent practitioner has a mental model that is useful for everyday purposes. Most new information they are likely to encounter will fit well with their existing model. Even though many potential elements of their mental model may still be missing or wrong, predictions about their area of work are usually accurate.
- An expert has a densely populated and connected mental model that is especially good for problem solving. They quickly connect concepts that others may not see as being related. They may have difficulty explaining how they are thinking in ways that do not rely on other features unique to their own mental model.

## Misconceptions

* **Simple factual errors**: These are the easiest to correct.
* **Broken models**: We can address these by having learners reason through examples to see contradictions.
* **Fundamental beliefs**: These beliefs are deeply connected to the learner’s social identity and are the hardest to change.

# Teaching or training?

<!--{% include _includes/quiz.html id="teaching_or_training.yaml" %}-->

> <hands-on-title>Teaching or training?  - {% icon time %} 15 min</hands-on-title>
>
> * Based on your experience, what are the differences between teaching and training?
> * Write your reflexions in the shared notes
{: .hands_on}

![A cartoon showing one of the differences between teaching and training. One the left side, the teacher is showing the information on a blackboards - in training, the instructor is showing, in practice, how this can be applied.](./images/teachingVStraining.png "A cartoon showing one of the differences between teaching and training. Source: [Open Science Training Handbook](https://www.fosteropenscience.eu/content/open-science-training-handbook)")

Teaching is more theoretical and abstract, while training (when done well) is more hands-on and practical.

Teaching seeks to impart knowledge and provide information, while training is intended to develop abilities.

- **Teach**: to provide knowledge, instruction or information
- **Train**: to develop abilities through practice with instruction or supervision

For example, it is possible to teach someone about buoyancy, fluid dynamics, water displacement and coastline safety, but that knowledge will not make them a good swimmer. Specific, practical and applied training is necessary to use abstract knowledge to learn or master a skill. In many cases, teaching and training are complementary. Vocational training programs often combine teaching and training quite nicely.

# Conclusion

After going through the theory of learning with the 6 strategies and the Ambrose 7 principles as well as the differences between teaching and training, the next step is to understand how these elements can be applied in designing an effective curriculum developmental trajectory.


topic: contributing
tutorial: fair-gtn
title: FAIR Galaxy Training Material
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "gtn", "training"]
content_hash: 3985de93d085c4db31d7e31696a9b773
last_modified: 2026-05-15T22:28:13.150627
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/fair-gtn/tutorial.html

content:
Encouraging computational reproducibility in research, we will present a variety of data stewardship recommendations that we have found useful in the process of training development. As part of that process, we are exploring the application of the FAIR (Findable, Accessible, Interoperable, Reusable) guidelines to the Galaxy Training Network (GTN) materials, in order to improve their secondary use and adaptation.

This tutorial outlines how to set and use existing resources to make Galaxy training development and content creation FAIR.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Here, we refer to a set of good practices as described in *"Ten simple rules for making training materials FAIR"* {% cite Garcia2020 %}.

![Ten simple rules for making training materials FAIR.]({% link topics/fair/images/fair_gtn.png %} "Ten simple rules for making training materials FAIR. The primary guideline is to share; the Findability rules are description, identity, and registration; the latter two, along with access rules, correspond to accessibility; and the first rule is to share; With the exception of the format rule, which stands alone for interoperability, the remaining four criteria all relate to various facets of reusability")

Image credit: Luc Wiegers and Celia van Gelder {% cite wiegers_luc_2019_3593258 %}

## Plan to share your training materials

The Galaxy Training Network (GTN) provides researchers with online training materials, connects them with local trainers, and helps promoting open data analysis practices worldwide. It provides a record of training content development and ensure materials curation via  cross-domain repositories (i.g., [GitHub](https://github.com/galaxyproject/training-material), [Zenodo](https://zenodo.org/)). Keeping your materials in the right places from the beginning will make it possible for you to more effectively and widely distribute your work without any duplication in contribution. Instruction where to start creating a new tutorial can be found at [GTN contributing tutorial]({% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %}).

## Improve findability of your training materials by properly describing them

Describing your training materials with structured metadata is fundamental to making them FAIR resource. Creating metadata boost findability and preserve information and can be updated after publication. GTN  tutorial require several mandatory metadata information such as learning objectives following Bloom’s taxonomy {% cite chevron2014metacognitive %}, prerequisites, time estimate, or questions addressed by the tutorial. Content developers needs to know where to find all of the available metadata to reference it later. [Schema.org](https://schema.org/) is a collaborative project with a mission to create the addition of structured metadata to web pages. It describes data using shared vocabulary. It is in this spirit that the [BioSchemas](https://bioschemas.org/) community initiative was created to extend the Schema.org standard to life-science resources with training content specification. GTN offers schemas command line [tutorial]({% link topics/contributing/tutorials/schemas/tutorial.md %}) where metadata can be added or updated. This reduces the complexity of keywords and set the scene for controlled vocabularies environment. This increases the effectiveness of metadata filtering, which decreases ambiguity and makes it easier to find and retrieve information.

## Give your training materials a unique identity

To support teachers and trainers, the GTN tutorials rely on specific tags and identifiers. Adding persistent identifiers (PIDs) to training materials makes them easy to cite and aids citation counting in research metric systems {% cite mcmurry2017identifiers %}. Providing PIDs ensure that tag/identifier will continue to work even if webpage location changes. Data used for tutorials are required to be stored at [Zenodo](https://zenodo.org/) and associated with [Digital Object Identifiers](https://www.doi.org/) (DOI).

## Register your training materials online
Sharing and publishing with the GTN training helps minimize the amount of time and effort required for instructors to prepare for and run their training courses and workshops, by providing templates and a complete training infrastructure. Online registry makes your content more discoverable and accessible to wider community. GTN tutorials are automatically registered on [TeSS](https://tess.elixir-europe.org/) (ELIXIR Training e-support system).

## Define access rules for your training materials
The GTN materials are designed to be flexible in their use, open access and community-driven. It is important to follow the hosting website disclaimers and keep materials metadata in place.

## Use an interoperable format for your training materials
GTN Tutorials aim to follow best practices in course design, so that they can be used in different environments. Content of the tutorials and slides are written in [Markdown]({% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md %}) and supported by templates. Metadata are stored in YAML and workflows in JSON. Self-contained structure allow others to tune or reuse existing content.

## Make your training materials (re)usable for trainers
To help others determine whether the training materials are relevant and adaptable to their particular situations, metadata published alongside training materials should include context and sufficient description including: contributor details, license, description, learning outcomes, audience, requirements, tags/keywords, duration and last revision date Applying the proper licence and tagging training materials with metadata can make it simpler for others to (re)use and adapt them.

GTN provide strong technical support and set of [contributing self learning material]({% link topics/contributing/index.md %}).

## Make your training materials usable for trainees
Prerequisites and learning outcomes are particularly helpful metadata. To be effective, learning objectives must be written using active verbs that describe the expected trainee behaviours as well as the knowledge, skills, and expertise they will have received. Rich metadata requirements and SMART—Specific (Measurable, Attainable, Relevant, and Time-bound) learning outcomes following Bloom's taxonomy {% cite chevron2014metacognitive %} helps to clarify which trainees will benefit most from the training. Self-learning structure of the GTN materials supported by slides and video walkthroughs/tours adds another layer of usability.

## Make your training materials contribution friendly
GTN have clear [guidelines]({% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %}) for contribution and involvement. Community supported CONTRIBUTING files and chat offer the chance to exchange information such as contact details, expectations for contributions, and more. All contributors should be listed and thanked in the acknowledgements; how to cite the tutorial and give credit to contributors can be found at the end of each tutorial.

## Keep your training materials up-to-date
It is crucial to keep your training materials up-to-date and so you are aware of any new features, trends, or improvements in the topic (such as updated tools or databases). Transparent peer-review and curation process in collaborative and open set up ensure materials quality.

# Conclusion
There are many ways to improve the training content. It is crucial that we work together to make training materials FAIR so that everyone can benefit from them. These simple suggestions are intended to encourage dialogue and cooperation among Galaxy training community and help bring the latest developments to users.

The Galaxy Training Network is an example of a robust, effective Community of Practice.
For more information please look at this great article {% cite hiltemann2023galaxy %}, the corresponding FAIR guidelines {% cite fair-training-materials %} and follow [short introduction to FAIR data stewardship](http://fellowship.elixiruknode.org/).


topic: contributing
tutorial: fair-by-design
title: FAIR-by-Design methodology
difficulty: introductory
time_estimation: 20M
tools_json: []
requirements_json: []
tags_json: ["'FAIR-by-Design Learning Materials'", "\"FAIR Learning Objects\"", "FAIR-by-Design Methodology"]
content_hash: a6f71fd3007c18c714a6e3d99c3f09da
last_modified: 2026-05-15T22:28:10.821956
zenodo_link: https://zenodo.org/records/11548062
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/fair-by-design/tutorial.html

content:
Welcome to the FAIR-by-Design Methodology Microlearning GTN adapted tutorial.

> <agenda-title></agenda-title>
>
>Below are the main stages of the FAIR-by-Design Methodology {% cite filiposka_2023_8305540 %} that will help guide you on your journey of creating FAIR learning materials.
>
>![Visual representation of the FAIR-by-Design Methodology flow with its separate stages](../../images/methodology.png)
>
>The FAIR-by-Design Methodology is defined in [a number of stages](#main-stages) that help you incorporate the FAIR principles into the backward learning instructional design process so that the final output are FAIR learning materials from both learners and instructors perspective:
>
>1. [Prepare](#stage-1---prepare)
>	- Do you understand FAIR and its implications
>	- Define purpose, learning objectives, target audience
>2. [Discover](#stage-2---discover)
>	- Find existing resources
>	- Identify potential for reuse
>3. [Design](#stage-3---design)
>	- Define the syllabus and structure
>	- Identify granularity
>	- Define facilitation materials
>4. [Produce](#stage-4---produce)
>	- Develop content using common apps and file formats
>	- Define machine readable metadata
>	- Perform internal QA check
>5. [Publish](#stage-5---publish)
>	- Define license and other related info
>	- Release for instructors and learners
>	- Enable feedback gathering
>6. [Verify](#stage-6---verify)
>	- Final QA check
>	- Use gathered feedback for continuous improvement
>7. [Continuous Improvement](#stage-7---continuous-improvement)
>	- Create a list of potential improvements
>	- Choose a set of improvements to be implemented
>	- Start a new release cycle
>
>Review each stage and the essential steps that it includes.
>
>Essential information about the original microlearning unit that this adaptation is based on can be found in the [About](#about) section.
>
{: .agenda}

If you have any questions at any stage on your journey do not hesitate to [contact the FAIR-by-Design methodology team](mailto:sonja.filiposka@finki.ukim.mk).

May all your materials be FAIR!


# Main stages

## Stage 1 - Prepare

> Before anything else, preparation is the key to success!
{: .quote author="Alexander Graham Bell"}

### First things first: What is FAIR?

><question-title> What is FAIR? </question-title>
>
>> <solution-title>Findable</solution-title>
>> The editable learning material has a unique and persistent identifier (PID) and is described with sufficiently detailed metadata.
>{: .solution}
>
>
>> <solution-title>Accessible</solution-title>
>> The human and machine readable metadata and object are stored in a trusted repository with clear authentication and authorization procedures.
>{: .solution}
>
>
>> <solution-title>Interoperable</solution-title>
>> The metadata describing the learning material follows a the RDA minimum metadata schema combined with agreed-upon controlled vocabularies.
>> Formal, accessible, shared, and broadly applicable language(s) and format(s) are used to develop the material.
>{: .solution}
>
>
>> <solution-title>Reusable</solution-title>
>>The learning material has a clear usage license (CC-BY-4.0 recommended) and accurate information on provenance.
>{: .solution }
{: .question }

> <details-title>More details on the FAIR principles</details-title>
> [Follow the FAIR in a nutshell tutorial available on GTN]( {% link topics/fair/tutorials/fair-intro/tutorial.md %} )
>
> [Follow the FAIR Galaxy Training Material tutorial available on GTN]( {% link topics/fair/tutorials/fair-gtn/tutorial.md %} )
{: .details}

### Adopt a metadata schema

If you are not using a discipline specific metadata schema, then, to ensure that your learning materials are appropriately described using a common approach, you should adopt the:

  <a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects/#rda-minimal-metadata-for-learning-resources" class="btn btn-primary stretched-link">Go to the RDA Minimal Metadata Set for Learning Resources description details</a>

The following are the fields described in the RDA Minimal Metadata Set divided into three categories:

> <details-title>Descriptive info metadata fields</details-title>
> - **Title** =	The human readable name of the resource
> - **Abstract / Description** =	A brief synopsis about or description of the learning resource
> - **Author(s)** =	Name of entity(ies) authoring the resource
> - **Primary Language** =	Language in which the resource was originally published or made available
> - **Keyword(s)** =	Keywords or tags used to describe the resource
> - **Version Date** =	Version date for the most recently published or broadcast resource
{: .details}

> <details-title>Access info metadata fields</details-title>
> - **URL to Resource** =	URL that resolves to the learning resource or to a "landing page" for the resource that contains important contextual information including the direct resolvable link to the resource, if applicable.
> - **Resource URL Type**	 = Designation of the identifier scheme used for the resource URL, e.g., DOI, ARK, Handle
> - **License** =	A license document that applies to this content, typically indicated by URL
> - **Access Cost**	= Choice stating whether or not there is a fee for use of the resource (yes, no, maybe)
{: .details}

> <details-title>Educational info metadata fields</details-title>
> - **Target Group (Audience)** =	Principal users(s) for which the resource was designed
> - **Learning Resource Type** =	The predominant type or kind that characterizes the learning resource
> - **Learning Outcome** =	Descriptions of what knowledge, skills or abilities a learner should acquire on completion of the resource
> - **Expertise (Skill) Level** =	Target skill level in the topic being taught; example values include beginner, intermediate, advanced
{: .details}

More on the RDA minimal metadata schema here {% cite hoebelheinrich_2022 %}

> <tip-title>Using metadata in GTN tutorials</tip-title>
>
>- When developing materials in GTN, this information should be included in the Tutorial.md file header metadata.
>- Most of the fields are already defined in the GTN Tutorial metadata schema. The ones that are missing can be added using the "extra" field.
>
> > <details-title>More details on the GTN Metadata</details-title>
> > [Follow the GTN Metadata tutorial]( {% link topics/contributing/tutorials/schemas/tutorial.md %} )
> {: .details}
>
{: .tip}


><details-title>Learn more about FAIR learning objects</details-title>
>- [FAIR instructional design skills](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/01-FAIR%20skills%20%26%20principles/01-FAIR%20skills%20%26%20principles/#fair-instructional-design-skills)
 >- [FAIR guiding principles](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/01-FAIR%20skills%20%26%20principles/01-FAIR%20skills%20%26%20principles/#fair-guiding-principles)
 >- [FAIR learning objects](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/01-FAIR%20skills%20%26%20principles/01-FAIR%20skills%20%26%20principles/#fair-learning-objects)
{: .details}

### Start Ideating

<div class="row">
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-header"><i class="fa fa-fast-backward" aria-hidden="true"></i> Think backward</div>
      <div class="card-body">
        <h5 class="card-title">Step 1</h5>
        <p class="card-text">What are your desired effects, i.e. learning outcomes?</p>
      </div>
    </div>
  </div>

  <div class="col-sm-4">
    <div class="card bg-light mb-3" >
      <div class="card-header"><i class="fa fa-fast-backward" aria-hidden="true"></i> Think backward</div>
      <div class="card-body">
        <h5 class="card-title">Step 2</h5>
        <p class="card-text">How are you going to assess the learners achievement?</p>
      </div>
    </div>
  </div>

  <div class="col-sm-4">
    <div class="card bg-light mb-3" >
      <div class="card-header"><i class="fa fa-fast-backward" aria-hidden="true"></i> Think backward</div>
      <div class="card-body">
        <h5 class="card-title">Step 3</h5>
        <p class="card-text">How should you structure the material to provide effective learning experience?</p>
      </div>
    </div>
  </div>

</div>
><tip-title>Learn more about the backward learning process</tip-title>
> - [Steps of the backward learning process](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects_cont/#backward-instructional-design-process)
{: .tip }

### Define

> <question-title>Purpose</question-title>
> When and how the learning materials can be used and for what purposes?
{: .question }

> <question-title>Target Audience</question-title>
> Is there anything specific that needs to be taken into account, such as cultural context?
{: .question }

> <question-title>Prerequisites</question-title>
> What does the target audience need to know or understand before starting the learning process?
{: .question }

> <question-title>Scope</question-title>
> Is it going to be a single learning unit, or a group such as a course?
{: .question }

> <question-title>Learning Objectives</question-title>
> What competences will be gained after successful completing of the learning process?
>
> ><tip-title>Be SMART</tip-title>
> >Objectives should be **s**pecific, **m**easurable, **a**ttainable, **r**elevant and **t**ime-bound.
> {: .tip}
>
> ><tip-title>Use Blooms Taxonomy</tip-title>
> >Formulate the objectives as actionable verb + observable knowledge, skill, attitude, behavior or ability.
> {: .tip}
>
> [Read more](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects_cont/#defining-learning-objectives)
{: .question }

> <details-title>More details on ideating learning outcomes and related information</details-title>
> [Read about defining intended Learning Outcomes in the Design and plan session, course, materials tutorial available on GTN]( {% link topics/contributing/tutorials/design/tutorial.md#define-intended-learning-outcomes-los %} )
{: .details}

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/411-prepare/" class="btn btn-dark text-white stretched-link">Go to the full description of FAIR-by-Design Methodology: Prepare stage ...</a>

## Stage 2 - Discover

> Greater even than the greatest discovery is to keep open the way to future discovery.
{: .quote author="John Jacob Abel"}
### Get inspired

Reusable materials can be found anywhere. These are just some examples:

**GTN**

- [GTN learning resources](https://training.galaxyproject.org)
- [TeSS Catalogue by Elixir](https://tess.elixir-europe.org)

**OER**

- [DOAB](https://directory.doabooks.org/)
- [MERLOT](https://www.merlot.org/merlot/index.htm)
- [OASIS](https://oasis.geneseo.edu/index.php)
- [OER Commons](https://www.oercommons.org/)
- [OERTX CORA](https://www.projectcora.org/)
- [GALILEO](https://oer.galileo.usg.edu/)
- [FORRT](https://forrt.org/)

**EOSC**

- [EOSC Training catalogue on the EOSC Marketplace](https://search.marketplace.eosc-portal.eu/search/training?q=*)
- Most EOSC projects have their own training catalogues and/or platforms.

**General**

- [Creative Commons Search](https://search.creativecommons.org/) - content provided under a CC license
- [Zenodo](https://zenodo.org/) - a multi-disciplinary open repository
- [OSF](https://osf.io/) - a free, open research platform

### Potential for reuse

> <warning-title> Respect the licenses, to be respected!</warning-title>
> - Materials with non-permissible licenses can be used for inspiration only.
> - Materials with permissible licenses should be reused based on the license rules.
{: .warning}

### Don't forget the multimedia search

Different learners have different learning modalities (read/write, auditory, visual, kinesthetic). To elevate the learning experience you should use all types of multimedia in your learning materials.

[Go to the full description of FAIR-by-Design Methodology: Discover stage ...](https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/412-discover/){: .btn.btn-dark.text-white.stretched-link}

## Stage 3 - Design

> Design is intelligence made visible.
{: .quote author="Alina Wheeler"}

Now it's time to brainstorm.

### Concept map

Step 1: Build a concept map of your learning materials.

> <details-title>More details on concept maps</details-title>
> [Read about Concept Maps in the Design and plan session, course, materials tutorial available on GTN]( {% link topics/contributing/tutorials/design/tutorial.md#concept-maps %} )
{: .details}

Step 2: Make sure you align your concept map with the MVS profiles.

The aligned MVS profile can help you crystallize the learning objectives using the MVS taxonomy.

Each MVS profile defines a list of technical and soft skills required for the profile. Think on how to incorporate both aspects in your learning materials.

<a href="https://fair-by-design-methodology.github.io/MVS/latest/MVS%20Profiles/Civil%20Servant/civil_servant/" class="btn btn-primary stretched-link">Go to the MVS profiles catalogue</a>

### Structure is everything

><comment-title></comment-title>
>{%icon galaxy-gear %} Use an intuitive logical organisation of all learning materials.
{: .comment}

><comment-title></comment-title>
>{%icon tool-versions %} The goal is for other people to easily reuse a single item (plan, activity, unit, assessment, ...).
{: .comment}

><comment-title></comment-title>
>{%icon galaxy-dataset-map %} Take advantage of a hierarchical structure to combine learning units into larger compositions.
{: .comment}

><hands-on-title>How to organise the files</hands-on-title>
>
> GTN defines a specific hierarchical structure that needs to be followed when developing learning materials.
>
> > <details-title>Check out the following tutorials</details-title>
> > [How to create a skeleton for a new learning topic]( {% link topics/contributing/tutorials/create-new-topic/tutorial.md#creating-a-new-topic-with-its-own-materials %}  )
> >
> > [How to create the skeleton of a new tutorial]( {% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %}  )
> {: .details}
>
{: .hands-on}

><comment-title>Syllabus is ready</comment-title>
> You should by now have the first draft of your <a href='https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%203%20–%20Design/04-Conceptualisation/04-Conceptualisation/#syllabus/'>syllabus</a>. It contains all the fields from the RDA min metadata set plus the high level topics covered by the learning material.
{: .comment}

><tip-title>Available feedback form</tip-title>
> GTN provides a readily available feedback form that is automatically added at the end of each tutorial. The feedback form is used to gather quantitative and qualitative feedback.
>
> > <details-title> Learn more about feedback in training</details-title>
> > [Follow the Assessment and feedback in training and teachings tutorial available on GTN]( {% link topics/teaching/tutorials/assessment/tutorial.md#dealing-with-feedback %}  )
> {: .details}
>
{: .tip}

><tip-title>Facilitation guide kit</tip-title>
> A <a href='https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%203%20–%20Design/07-Facilitation/07-Facilitation/'> facilitation guide </a> should help prepare for the actual training.
>
> The facilitation guide kit includes documentation of the process of organising and running a training. What different people need to have, know and do so that everything runs smoothly.
>
> > <details-title> Learn more about facilitation</details-title>
> > [Follow the Organizing a workshop tutorial available on GTN]( {% link topics/teaching/tutorials/organize-workshop/tutorial.md %}  )
> >
> > [Follow the Running a workshop as instructor tutorial available on GTN]( {% link topics/teaching/tutorials/running-workshop/tutorial.md %}  )
> {: .details}
>
> Another option is to use something like the [TRIPLE project TRAINING TOOLKIT](https://project.gotriple.eu/project-deliverables/triple-training-toolkit/).
>
{: .tip}

><question-title>What about instructor notes?</question-title>
>  They need to be detailed enough so that anyone can reuse the learning content, especially slides properly.
>
>  Any specific information relevant for instructors that would like to organise a training based on your GTN tutorial content should be added in Details box.
>
>  This practice significantly increases the reuse potential of the material.
{: .question}

### How to design the learning content

<a href="https://www.csun.edu/sites/default/files/Holle-Lesson-Planning.pdf" class="btn btn-primary stretched-link">Read more about the Hunter Model.</a>

<div class="row">
  <div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon curriculum %} 1. Set Learning Objectives</h5>
        <p class="card-text">
         What is the goal?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon pref-identities %} 2. Identify Needs</h5>
        <p class="card-text">
        How to get there?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-panelview %} 3. Plan</h5>
        <p class="card-text">
        Share the agenda.
        </p>
      </div>
    </div>
  </div>
</div>

<div class="row">
  <div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon version %} 4. Hook</h5>
        <p class="card-text">
        Why is the content important?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-wf-edit %} 5. Instruct</h5>
        <p class="card-text">
        Watch how I do it.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-rulebuilder-history %} 6. Practise</h5>
        <p class="card-text">
        You help me do it, I'll watch you do it.
        </p>
      </div>
    </div>
  </div>
</div>

<div class="row">
  <div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-barchart %} 7. Wrap-Up</h5>
        <p class="card-text">
         Foster retention and reinforcement.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon license %} 8. Evaluate</h5>
        <p class="card-text">
        Monitor progress.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-history-answer %} 9. Reflect</h5>
        <p class="card-text">
        How did it go?
        </p>
      </div>
    </div>
  </div>
</div>

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/413-design/" class="btn btn-dark text-white stretched-link">Go to the full description of FAIR-by-Design Methodology: Design stage ...</a>

## Stage 4 - Produce

> To contrive is nothing! To construct is something! To produce is everything!
{: .quote author="Edward Rickenbacker"}

### Choose Tools & Formats

> <tip-title>Collaborative environment for team work</tip-title>
>  Choose an environment for producing the learning material that will enable multiple people to work on the same material at one.
>
>  - GitHub is one of the most popular options at the moment (find out more <a href='https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%204%20–%20Produce/08-Development%20Tools/08-Introduction%20to%20Markdown%20and%20Git/'>here</a>)
>  - GTN is also based on GitHub and defines [specific procedures on how one can contribute and collaborate](https://github.com/galaxyproject/training-material/blob/main/CONTRIBUTING.md) using GitHub
>
>  > <details-title> GTN: collaboration using GitHub</details-title>
> > [Follow the Contributing with GitHub via its interface tutorial available on GTN]( {% link topics/contributing/tutorials/github-interface-contribution/tutorial.md %}  )
> >
> > [Follow the Contributing with GitHub via command-line tutorial available on GTN]( {% link topics/contributing/tutorials/github-command-line-contribution/tutorial.md %}  )
> {: .details}
{: .tip}

> <tip-title>Granular versioning for easy rollback</tip-title>
>    - Versioning helps you maintain control over your changes.
>    - GitHub natively provides versioning and history retention that help easy roll back to an earlier stable state.
{: .tip}

> <tip-title>Open file formats to foster reuse</tip-title>
> - For other people to reuse the materials they should be made available using open file formats.
> - GTN promotes the use of the MD open file format for the main learning content empowered with open scientific notebooks and workflows.
> - If you use close file formats then you MUST clearly state the tools that have been used in more details.
{: .tip}

> <tip-title>Multimodal content to reach all audience</tip-title>
>  - Don't forget to include different types of multimedia to provide support for different learning modalities: read/write, auditory, visual, kinesthetic.
>  - In addition to hands-on exercises, GTN also provides support for audio and video modalities.
{: .tip}

> <tip-title>Two file sets: editable + final</tip-title>
> - The MD and supporting files are the main files used for development of the content. These files are what matters for you and other instructors.
> - Based on the editable files, GTN automatically generates the final (non-editable version) in HTML (and PDF). These  are shared with the learners.
> - GTN takes care of the revision numbering for you.
{: .tip}

> <tip-title>Don't forget to take advantage of co-creation</tip-title>
> Truly FAIR learning materials enable co-creation with external parties.
>  GitHub is a collaborative environment that supports co-creation in every step of the learning materials development and revision process.
>  The GTN fork-and-pull process of contributing to the learning material is a clear example of co-creation implementation.
{: .tip}


### Plan to reuse existing material?

#### Check the license

The existing materials you are reusing are available under a [CC license](https://creativecommons.org/share-your-work/cclicenses/), but it is different than the one you plan to use for your materials. Depending on how you want to reuse the material, you will need to consider the following aspects.

><details-title>I want to reuse it as a whole</details-title>
>- You can't use something that is licensed with ND (no derivatives).
>- In this case you must follow the rules on combining and adapting CC material.
>
>[Read more](https://creativecommons.org/faq/#combining-and-adapting-cc-material)
{: .details}

><details-title>I want to reuse a small part of it</details-title>
> No problem, you can reuse any existing CC licensed material in your learning materials as long as the reused portion is used as a showcase or to make a specific point and it is not the core of your work.
>
> Remember that if the work is licensed with ND, you can not modify it while reusing.
{: .details}

><tip-title>Learn more about IPR</tip-title>
>[How Intellectual Property Rights (IPR) protect the interests of the creators and owners by providing them with rights over their creation?](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects/#intellectual-property-rights-ipr)
{: .tip}

Still need help? [{% icon point-right %} Go to CC licensing FAQ](https://creativecommons.org/faq/#before-using-cc-licensed-material)


#### Attribute

All CC licenses require that you attribute the author, and this rule is recommended even if the license is public domain CC-0.

If the work you are reusing has a copyright notice ('© some text') you need to reproduce it while you credit the work.

You should also be able to remove attribution upon request.

##### How to attribute?

> <solution-title>Use the authors recommended attribution</solution-title>
> If the original author has provided a cite-as information, use it to attribute the work.
{: .solution}

> <solution-title>Use TASL</solution-title>
> Provide the Title, Author, Source and License of the work that you are reusing.
>
>- Source is the URL to the original work.
>- If there is a URL to the author personal pages, provide it together with the name.
>- Provide the name of the license and a URL to the license.
{: .solution}

> <details-title>Examples</details-title>
> Examples are taken from [Best Practices for Creative Commons attributions - how to attribute works you reuse under a Creative Commons license](https://www.newmediarights.org/guide/how_to/creative_commons/best_practices_creative_commons_attributions) submitted by [New Media Rights](https://www.newmediarights.org/) available under a [CC BY-NC 3.0 US DEED](https://creativecommons.org/licenses/by-nc/3.0/us/).
> - Webpage/Blog - Title (with link to original work), author (or username) (with link to author's website), and license (with link).
> 	- [Undercover Vampire Policeman](https://chriszabriskie.bandcamp.com/album/undercover-vampire-policeman) by [Chris Zabriskie](https://chriszabriskie.bandcamp.com/), available under a [Creative Commons Attribution 4.0 License](http://creativecommons.org/licenses/by/4.0/)
> - Book – Title, author, license written somewhere near the title and author if it’s a hard copy or if it’s an online book you should include a link to the licensed terms.
> 	- [From Dust to Digital: Ten Years of the Endangered Archives Programme](https://books.google.com/books?id=ImO3BgAAQBAJ&pg=PR4&dq=creative+commons+4.0&hl=en&sa=X&ved=0CEoQ6AEwCGoVChMIspCXhPPxxgIVSF0eCh27NA5X#v=onepage&q=creative%20commons%204.0&f=false) by Maja Kominko under a Creative Commons Attribution Non-commercial Non-Derivative 4.0 International license (CC BY-NC-ND 4.0)
> - Online Video - Title, author, license written into credits at end of video.  Ideally make the text clickable to the original work.  Put links to the original work and the license terms in the information section for the particular work (i.e. on the right in YouTube).
> 	- [http://www.youtube.com/watch?v=fDbbdeIXO0w#t=3m0s](http://www.youtube.com/watch?v=fDbbdeIXO0w#t=3m0s)
> - Podcast/Audio - Title, author, license read at the end of the entire work.
> 	- [“Je Suis Rick Springfield”](http://www.jonathancoulton.com/wiki/Je_Suis_Rick_Springfield) from the album [Artificial Heart](http://www.jonathancoulton.com/wiki/Artificial_Heart_(album)), by Jonathan Coulton, used under a [Creative Commons Attribution-Noncommercial 3.0 Unported License](http://creativecommons.org/licenses/by-nc/3.0/)
>- Photo/Drawing/Illustration – Title, author, license (with link online) or in close proximity to the tangible work (either in the border or directly on the work, if applicable).
> 	- [Comcast protest](http://www.flickr.com/photos/ari/8503459/in/set-214952/)” by Flikr user [Steve Rhodes](http://www.flickr.com/photos/ari/) used under [Creative Commons Attribution 2.0 license](http://creativecommons.org/licenses/by/2.0/deed.en)
{: .details}

> <details-title>Read more about how to attribute</details-title>
> [{% icon point-right %}  Attribution and Citing](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects/#attribution-and-citing)
>
> [{% icon point-right %} How to handle attribution?](https://courses.lumenlearning.com/suny-oerguide/chapter/how-to-handle-attribution/)
{: .details}


### Accessibility

The developed learning materials should cover the widest range of learner variability including the ones that use or do not use assistive technology.

![Universal access logo](../../images/universal-access-6602642_640.png)
<a href="https://pixabay.com/vectors/universal-access-human-icon-6602642/">"Universal Access Human"</a> by <a href="https://pixabay.com/users/inspire-studio-22128832/">J S</a> from <a href="https://pixabay.com/">Pixabay</a>


#### Standards

There are several standards that govern the rules on the level of accessibility.

Most commonly used is the <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">W3C Web Content Accessibility Guidelines (WCAG) standard version 2.1</a>. Three conformance levels exist, you should aim for AA which is the middle one.

PDF document accessibility is measured with a separate technical specification <a href="https://pdfua.foundation/en/why-pdf-ua/">PDF/UA (Universal Accessibility)</a>.


#### Accessibility guidelines

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%204%20%E2%80%93%20Produce/11-Accessibility/11-Checking_accessibility/#general-guidelines-for-development-of-accessible-materials" class="btn btn-primary stretched-link">Review some general guidelines for development of accessible material</a>

><tip-title>GTN Accessibility</tip-title>
> GTN has developed an <a href="https://training.galaxyproject.org/training-material/accessibility.html"> accessibility mission </a> aiming for WCAG 2.0 AA compliance by implementing a number of accessibility features.
>
{: .tip}


### Internal QA

<div class="row">
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon congratulations %} QA Self-assessment</h5>
        <p class="card-text">
         Check if everything is as it should be.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon pref-list %} Quantitative</h5>
        <p class="card-text">
        Are all required elements produced?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon solution %} Qualitative</h5>
        <p class="card-text">
        Do all learning units provide materials to reach the learning objectives with different modalities?
        </p>
      </div>
    </div>
  </div>
</div>

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/414-produce/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Produce stage ...</a>

## Stage 5 - Publish

> Publishing is the art of working on a creative idea and turning it into a masterpiece!
{: .quote author="Unknown"}

><warning-title>Publishing closed FAIR materials</warning-title>
> Having FAIR learning materials does not always mean that the materials are open to everyone and there are no costs or access rules attached. In this case the bundle that is going to be published in an open repository such as Zenodo should contain the following:
>  1. Syllabus, that contains all metadata that describe the materials. Metadata should always be open.
>  2. Accompanying information (optional) to augment the description of the materials and describe the details when it comes to accessing and using the materials from a trainer perspective.
>
> The complete learning materials package itself should be published in a closed repository where the corresponding access rules (and costs) can be implemented.
{: .warning}

### Final preparations

><tip-title>Time to provide the accompanying information</tip-title>
> Usually provided at (or near) the root of the learning materials, the purpose of the accompanying information is to further describe the content and define the rules of reuse.
{: .tip}

<div class="row">
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon text-document %} LICENSE</h5>
        <p class="card-text">
         The license defines the potential of reuse of your learning materials together with the rules regarding their attribution by others.<br>
         CC-BY-4.0 is the recommended license, which is default in GTN. <br>
         If you choose a different license, it must be supplied in the tutorial header as metadata.
         <br>
         <a href="https://creativecommons.org/licenses" > {% icon point-right %} Browse through the available CC license types
         </a>
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon sticky-note %} README</h5>
        <p class="card-text">
        A README is a text file that introduces and explains the contents of the learning topics and materials. It usually describes the context and written in a plain text format. <br> GTN provides a README file on the level of each topic, where context information about the topic and its tutorials can be provided.
        <br>
		<a href="https://www.makeareadme.com/">
        {% icon point-right %}  Make a README
        </a>
        </p>
      </div>
    </div>
  </div>
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon cofest %} CONTRIBUTORS</h5>
        <p class="card-text">
        GTN CONTRIBUTORS is a YAML file that lists the information about all contributors of learning materials on the platform.
        <br>You must have a GitHub user to be listed as a contributor. Additional information such as ORCID can also be provided.
        <br>
        <a href="{% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md#listing-contributors %}">
		{% icon point-right %} Add to CONTRIBUTORS.yaml
		</a>
        </p>
      </div>
    </div>
  </div>
</div>

<div class="row">
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon help %} FAQs</h5>
        <p class="card-text">
	        In addition to the learning content, the Frequently Asked Questions can further help both learners and instructors with specific information about the learning context or practicalities.
	        <br> Consider adding a tutorial-specific FAQ to the GTN tutorial, where you can answer questions about the challenges of working with the hands on activities, or provide other hints and guidelines to instructors and learners.
	        <br>
		<a href="{% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md#creating-new-faqssnippets %}">
        {% icon point-right %}  Create snippets
        </a>
        </p>
      </div>
    </div>
  </div>
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon event-cost %} FUNDERS</h5>
        <p class="card-text">
        If the creation of the learning materials is funded by an organisation or another body, they should be attributed accordingly. <br>
        In GTN this can be accomplished by adding the funding information into the special FUNDERS.yaml file.
         <br>
         <a href="https://github.com/galaxyproject/training-material/blob/main/FUNDERS.yaml">
         {% icon point-right %} Add to the Funders file
         </a>
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon new-history %} OTHER</h5>
        <p class="card-text">
        Additional miscellaneous information should be provided if possible.
        <br> For example, GTN also supports adding organisations information.
        <br>
        <a href="https://github.com/galaxyproject/training-material/blob/main/ORGANISATIONS.yaml">
        {% icon point-right %} Add org info
        </a>
		</p>
      </div>
    </div>
  </div>
</div>

<div class="row">
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon references %} CITATION</h5>
        <p class="card-text">
	        It is best practice to provide information on how you want others to cite your learning materials when they are referenced or reused. <br> GTN does this automatically by appending the "Citing this Tutorial" section at the end of each tutorial.
	        <br>
		<a href="https://training.galaxyproject.org/training-material/faqs/gtn/gtn_citing.html">
        {% icon point-right %}  See an example citation
        </a>
        </p>
      </div>
    </div>
  </div>
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon license %} CODE OF CONDUCT</h5>
        <p class="card-text">
        A code of conduct defines the rules for how to engage in a co-creation community.
        <br>It is based on the premise of an inclusive environment that respects all contributions.
         <br>
         <a href="https://galaxyproject.org/community/coc/">
         {% icon point-right %} Read the Galaxy Project Code of Conduct
         </a>
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon comment %} REVISIONS</h5>
        <p class="card-text">
        Information about new or updated versions of your learning materials helps others keep track of the changes more easily. <br> GTN automatically updates the Revision number of each tutorial after each release.
        <br>
        <a href="https://github.com/galaxyproject/training-material/releases">
        {% icon point-right %} Compare changes between GTN releases
        </a>
		</p>
      </div>
    </div>
  </div>
</div>

### Storing and indexing

><tip-title>Automated publishing</tip-title>
>If you are working on GTN following the provided rules and procedures, GTN will automatically publish the new tutorial once the full review and release process is completed.
>
>Just follow the [GTN guide to contributing via GitHub]( {% link topics/contributing/#st-contribute %}).
{: .tip}

><tip-title>To training catalogue</tip-title>
>You are at the point when you should also consider making a record in a relevant training catalogue.
>
>GTN automatically creates a new record in the [Elixir TeSS training catalogue](https://tess.elixir-europe.org) that is the most relevant catalogue for its community.
{: .tip}

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/415-publish/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Publish stage ...</a>

## Stage 6 - Verify

> You will find it a very good practice always to verify your references, sir!
{: .quote author="Martin Routh"}

### External QA

> <tip-title> A fresh set of eyes</tip-title>
> - Have someone who has not participated in the development of the learning materials review the final work. This will guarantee a review free of cognitive bias.
> - GTN implements this automatically via the fork-and-pull request process. Before publication your new contribution is being reviewed by GTN peers.
{: .tip}

> <tip-title> Go through the QA checklists</tip-title>
> In Skills4EOSC T2.4 has developed a number of QA checklists that you and your external reviewer need to go through so that you can ensure high-quality learning materials {% cite sanchez_2023_8305482 %}.
{: .tip}

### FAIR or not FAIR, that is the question

#### Measure FAIRness

Use the [FAIR-by-Design methodology QA checklist](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%206%20–%20Verify/19-Final%20QA%20check/19-finalQA/#fair-by-design-methodology-qa-checklist) to check if you have followed the most important aspects of the methodology and managed to produce FAIR learning materials.

The questions marked as essential achieve bare minimum FAIRness.

><details-title>Essential requirements</details-title>
> - **Findable**
>   - Is the complete learning resource (including instructors info) registered or indexed in at least one searchable repository? Is it in a FAIR repository?
>   - Is metadata for the resource provided in both human- and machine-readable format (e.g JSON, XMLor YAML)?
> - **Accessible**
>   - Has an accessibility checker tool been utilised to improve the accessibility of all learning resource files (PDF, HTML, video, etc.)?
>   - Are access rules (authentication & authorisation) implemented for the learning resource?
> - **Interoperable**
>   - Is the RDA minimal (or domain specific) metadata schema used for the learning material description?
>   - Is the resource available in open file formats which are tool agnostic and compatible with a wide variety of existing software?
> - **Reusable**
>   - Is there clear attribution for all reused resources with compatible licenses?
>   - Has the learning resource been made available for use by defining a permissable license or policy information that allows derivations?
{: .details}

><details-title>Optional requirements</details-title>
> - Did you follow the stages of the backward instructional design process while developing the learning resource?
> - Are controlled vocabularies (CVs) used for describing the resource characteristics aligned with the chosen metadata schema?
> - Does the learning resource represent a complete learning object defined around minimum one learning objective?
> - Does the resource incorporate an instructor kit that aids in facilitating the process of others reusing learning material by offering helpful how-to guides?
>    - facilitator guide
>    - activities description
>    - assessment activities and strategy to assess
>    - general learning content or instructor notes
>    - lesson unit plan
>    - syllabus
> - Have you employed a versioning system to track and control changes in your materials?
> - Are the resource access rules (how to access, e.g. registration procedure) explicitly communicated to learners?
> - Is the learning resource searchable in at least one relevant catalogue?
>    - Is it FAIR (can be searched based on metadata)?
> - Does the course include the possibility to provide feedback or comments from users and-or trainers/designers?
>    - If so, do you regularly gather and analyse that feedback?
> - Does the resource adopt an open community approach regarding its quality and reachability?
> - Has the learning resource been checked by a third party regarding its learning experience quality?
{: .details}

### Feedback QA

> <comment-title>Regularly gather feedback from learners and instructors</comment-title>
> Ensure that you actively and regularly gather feedback from both perspectives: the learners and the instructors.
{: .comment}

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/416-verify/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Verify stage ...</a>

## Stage 7 - Continuous Improvement
> To improve is to change; to be perfect is to change often
{: .quote author="Winston Churchill"}

### {% icon galaxy-download %} Gather

- Gather feedback from all available internal & external sources.
- Potential sources:
	- Feedback form
	- QA recommendations
	- Self-reflection after training
	- Git Issues
	- Matrix Chat
	- Direct mail contact
	- Other means of communication

### {% icon galaxy-barchart %} Analyse

- Analyse the gathered information in a structured way.
- Create a list of potential improvements with impact level (high, moderate, low).

### {% icon galaxy-wf-edit %} Improve

- Select items from the list that will be part of a new version.
- Choose items that make sense to be in the same new release.

### {% icon galaxy-history-refresh %} Repeat

- Start a new cycle of the FAIR-by-Design methodology that will implement the selected items.
- After the Verify stage, you will reenter continuous improvement with the newly gathered information.

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.2%20-%20Continuous%20Improvement/417-improvement/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Continuous Improvement ...</a>

# About

## FAIR-by-Design Microlearning

### Self-paced how to FAIR-by-Design guide

### Original Location

- online: [https://fair-by-design-methodology.github.io/microlearning/latest/](https://fair-by-design-methodology.github.io/microlearning/latest/)
- this tutorial is an adapted version of the original microlearning guide

### Training Description

Short how-to that guides you through the stages of the FAIR-by-Design methodology without any specific choice on tools and formats.

### Target audience

All interested parties who need to develop learning materials for any type of project-related training.

### Expertise Level / Skill Level: Beginner

### Primary Language: English

### Access Cost: No

### Prerequisites

No prior knowledge is required.

### Learning Objectives

- Design FAIR learning materials
- Structure FAIR learning materials
- Create and publish FAIR-by-Design learning materials
- Assess FAIR-ness of learning objects

### Keywords

FAIR, learning objects, methodology, practical implementation

### Certification Information

This training has no certification or any recognition mechanism included.

### Original Author(s)

- Sonja Filiposka, Dominique Green, Anastas Mishev, Vojdan Kjorveziroski, Andrea Corleto, Eleonora Napolitano, Gabriella Paolini, Sara di Giorgio, Joanna Janik, Luca Schirru, Arnaud Gingold, Christine Chardosek, Irakleitos Souyioultzoglou, Carolin Leister, Emma Lazzeri


### Contact information

- For more information please contact the T2.3 FAIR-by-Design Methodology Task Leader Sonja Filiposka using [sonja.filiposka@finki.ukim.mk](mailto:sonja.filiposka@finki.ukim.mk).

### Original License

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.

### Original microlearning unit DOI

[https://doi.org/10.5281/zenodo.11548062](https://doi.org/10.5281/zenodo.11548062)

#### Acknowledgement

These learning materials have been developed by following the [FAIR-by-Design Methodology](https://zenodo.org/records/8305540).


topic: introduction
tutorial: vsi_qc
title: Very Short Introductions: QC
difficulty: intermediate
time_estimation: 30M
requirements_json: []
tags_json: []
content_hash: f0e0da4039c224fd2941b9babc96de80
last_modified: 2026-05-15T22:28:13.732525
zenodo_link: https://zenodo.org/records/10870107
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/vsi_qc/tutorial.html

content:
In this tutorial we will look at assessing quality of data from two short read technologies: [Illumina](http://www.nature.com/doifinder/10.1038/nature07517) and [Element Biosciences](http://dx.doi.org/10.1038/s41587-023-01750-7). 

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## FASTQ format

[FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) is not a very well defined format. In the beginning various manufacturers of sequencing instruments were free to interpret FASTQ as they saw fit, resulting in a multitude of FASTQ flavors. This variation stemmed primarily from different ways of encoding quality values as described [on the Wikipedia article for FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) (below you will find an explanation of quality scores and their meaning). Today, the [FASTQ Sanger](https://www.ncbi.nlm.nih.gov/pubmed/20015970) version of the format is considered to be the standard form of FASTQ. Galaxy is using FASTQ Sanger as the only legitimate input for downstream processing tools and provides [a number of utilities for converting FASTQ files](https://www.ncbi.nlm.nih.gov/pubmed/20562416) into this form (see **FASTQ Quality Control** section of Galaxy tools).

The FASTQ format looks like this:


```

@M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCTGAACCAGCCAAGTAGCGTGCAG
+
ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE<,,CD,CEFC,@E9<FCFCF?9
@M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACTGATCCAGCCATGCCGCGTGCAG
+
ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF<,CEF,CFGGD8FFCFCFEGCF
@M02286:19:000000000-AA549:1:1101:11116:1322 1:N:0:23
CCTACGGGAGGCAGCAGTAGGGAATCTTCGGCAATGGACGGAAGTCTGACCGAGCAACGCCGCGTGAGT
+
AAC<CCF+@@>CC,C9,F9C9@9-CFFFE@7@:+CC8-C@:7,@EFE,6CF:+8F7EFEEF@EGGGEEE

```

Each sequencing read is represented by four lines:

1. `@` followed by read ID and optional information about sequencing run
2. sequenced bases
3. `+` (optionally followed by the read ID and some additional info)
4. quality scores for each base of the sequence encoded as [ASCII symbols](https://en.wikipedia.org/wiki/ASCII)

## Paired end data

It is common to prepare pair-end and mate-pair sequencing libraries. This is highly beneficial for a number of applications discussed in subsequent topics. For now let's just briefly discuss what these are and how they manifest themselves in FASTQ form.

![Paired-end and mate-pair reads](../../images/pe_mp.png "<b>Paired-end and mate-pair reads</b>. In paired end sequencing (left) the actual ends of rather short DNA molecules (less than 1kb) are determined, while for mate pair sequencing (right) the ends of long molecules are joined and prepared in special sequencing libraries. In these mate pair protocols, the ends of long, size-selected molecules are connected with an internal adapter sequence (i.e. linker, yellow) in a circularization reaction. The circular molecule is then processed using restriction enzymes or fragmentation. Fragments are enriched for the linker and outer library adapters are added around the two combined molecule ends. The internal adapter can then be used as a second priming site for an additional sequencing reaction in the same orientation or sequencing can be performed from the second adapter, from the reverse strand. (From <i>Understanding and improving high-throughput sequencing data production and analysis</i>, Ph.D. dissertation by <a href='https://ul.qucosa.de/api/qucosa%3A11231/attachment/ATT-0/'>Martin Kircher</a>)")


Thus in both cases (paired-end and mate-pair) a single physical piece of DNA (or RNA in the case of RNA-seq) is sequenced from two ends and so generates two reads. These can be represented as separate files (two FASTQ files with first and second reads) or a single file were reads for each end are interleaved. Here are examples:

### Two single files

File 1

```
 @M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
 CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCT
 +
 ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE
 @M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
 CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACT
 +
 ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF
```

File 2

```
@M02286:19:000000000-AA549:1:1101:12677:1273 2:N:0:23
CACTACCCGTGTATCTAATCCTGTTTGATACCCGCACCTTCGAGCTTA
+
--8A,CCE+,,;,<CC,,<CE@,CFD,,C,CFF+@+@CCEF,,,B+C,
@M02286:19:000000000-AA549:1:1101:15048:1299 2:N:0:23
CACTACCGGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATC
+
-6AC,EE@::CF7CFF<<FFGGDFFF,@FGGGG?F7FEGGGDEFF>FF
```

> <comment-title>Read order is important</comment-title>
> Note that read IDs are **identical** in two files and they are listed in **the same** order. In some cases read IDs in the first and second file may be appended with `/1` and `/2` tags, respectively.
{: .comment}

### Interleaved file

```
@1/1
AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
+
EGGEGGGDFGEEEAEECGDEGGFEEGEFGBEEDDECFEFDD@CDD<ED
@1/2
CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
+
GHHHDFDFGFGEGFBGEGGEGEGGGHGFGHFHFHHHHHHHEF?EFEFF
@2/1
AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
+
HHHHHHEGFHEEFEEHEEHHGGEGGGGEFGFGGGGHHHHFBEEEEEFG
@2/2
CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
+
HHHHHHHHHHHHHGHHHHHHGHHHHHHHHHHHFHHHFHHHHHHHHHHH
```

Here the first and the second reads are identified with `/1` and `/2` tags.

> <comment-title>FASTQ format is a loose standard</comment-title>
> FASTQ format is not strictly defined and its variations will always cause headache for you. See [this page](https://www.ncbi.nlm.nih.gov/sra/docs/submitformats/) for more information.
{: .comment}

## What are base qualities?

As we've seen above, FASTQ datasets contain two types of information:

- *sequence of the read*
- *base qualities* for each nucleotide in the read.

The base qualities allow us to judge how trustworthy each base in a sequencing read is. The following excerpt from an excellent [tutorial](http://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf) by Friederike Dündar, Luce Skrabanek, Paul Zumbo explains what base qualities are:

> <comment-title>From "Introduction to differential gene expression analysis using RNA-seq"</comment-title>
> Illumina sequencing is based on identifying the individual nucleotides by the fluorescence signal emitted upon their incorporation into the growing sequencing read. Once the fluorescence intensities are extracted and translated into the four letter code. The deduction of nucleotide sequences from the images acquired during sequencing is commonly referred to as base calling.
><br><br>
> Due to the imperfect nature of the sequencing process and limitations of the optical instruments, base calling will always have inherent uncertainty. This is the reason why FASTQ files store the DNA sequence of each read together with a position-specific quality score that represents the error probability, i.e., how likely it is that an individual base call may be incorrect. The score is called [Phred score](http://www.phrap.com/phred/), $$Q$$, which is proportional to the probability $$p$$ that a base call is incorrect, where $$Q = −10lg(p)$$. For example, a Phred score of 10 corresponds to one error in every ten base calls ($$Q = −10lg(0.1)$$), or 90% accuracy; a Phred score of 20 corresponds to one error in every 100 base calls, or 99% accuracy. A higher Phred score thus reflects higher confidence in the reported base.
><br><br>
> To assign each base a unique score identifier (instead of numbers of varying character length), Phred scores are typically represented as ASCII characters. At http://ascii-code.com/ you can see which characters are assigned to what number.
><br><br>
> For raw reads, the range of scores will depend on the sequencing technology and the base caller used (Illumina, for example, used a tool called Bustard, or, more recently, RTA). Unfortunately, Illumina has been anything but consistent in how they calculated and ASCII-encoded the Phred score (see below)! In addition, Illumina now allows Phred scores for base calls with as high as 45, while 41 used to be the maximum score until the HiSeq X. This may cause issues with downstream sapplications that expect an upper limit of 41.
{: .comment}

![Illumina quality score](../../images/illumina_qs.png "Comparison between different encodings")

Base call quality scores are represented with the Phred range. Different Illumina (formerly Solexa) versions
used different scores and ASCII offsets. Starting with Illumina format 1.8, the score now represents the standard
Sanger/Phred format that is also used by other sequencing platforms and the sequencing archives.

![FASTQ quality score](../../images/fastq_qs.png "The ASCII interpretation and ranges of the different Phred score notations used by Illumina and the original Sanger interpretation. Although the Sanger format allows a theoretical score of 93, raw sequencing reads typically do not exceed a Phred score of 60. In fact, most Illumina-based sequencing will result in maximum scores of 41 to 45 (image from <a href='https://en.wikipedia.org/wiki/FASTQ_format'>Wikipedia</a>).")


# Assessing quality

This tutorial provides two sample datasets: one generated with Illumina and the other with ElemBio (AVITI) machine. Pick any. We will compare qualities of the two platforms at the and of this tutorial. 

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    **Element** (AVITI). This datasets contains forward (r1) and reverse (r2) reads for two samples---`s1` and `s2`:
>
>    ```
>    https://zenodo.org/records/10870107/files/elem_s1_r1.fq.gz
>    https://zenodo.org/records/10870107/files/elem_s1_r2.fq.gz
>    https://zenodo.org/records/10870107/files/elem_s2_r1.fq.gz
>    https://zenodo.org/records/10870107/files/elem_s2_r2.fq.gz
>    ```
>
>    **Illumina**. This datasets contains forward (_1) and reverse (_2) reads for two samples---`bl` and `ch`:
>
>    ```
>    https://zenodo.org/record/5119008/files/M117-bl_1.fq.gz
>    https://zenodo.org/record/5119008/files/M117-bl_2.fq.gz
>    https://zenodo.org/record/5119008/files/M117-ch_1.fq.gz
>    https://zenodo.org/record/5119008/files/M117-ch_2.fq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the datatype is set to `fastqsanger.gz` (Galaxy usually automatically assigns the right datatype, but check anyway).
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Create a paired collection from your data (relevant part beging at 0:57 and and the case of this tutorial we have 4 datasets): 
>
>    {% include _includes/youtube.html id="uZUt9XIHUQo" title="Create a paired collection tutorial" start=57 %}
>
{: .hands_on}

## Assess quality and clean the reads with `fastp`

[`fastp`](https://github.com/OpenGene/fastp) is fast utility for computing read qualities and performing initial clean up of the data such as [adapter removal](https://github.com/OpenGene/fastp?tab=readme-ov-file#adapters).

> <hands-on-title> Running `fastp` </hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Paired Collection`
>        - {% icon param-collection %} *"Select paired collection(s)"*: `output` (Input dataset collection)
>    - In *"Output Options"*:
>        - *"Output JSON report"*: `Yes`
>
{: .hands_on}

## Sub-step with `MultiQC`[]

[`multiqc`](https://multiqc.info/) is a report generation tool. It can process outputs on multiple tools including `fastp`:

> <hands-on-title> Generating quality report </hands-on-title>
>
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `fastp`
>                - {% icon param-file %} *"Output of fastp"*: `report_json` (output of **fastp** {% icon tool %})
>
{: .hands_on}

# Interpreting the results

The comparison between Element and Illumina QC plot is shown below:

![QC scores for Element and Illumina](../../images/qc_elem_ill.svg "Element (top) versus Illumina (bottom). Element base qualities are consistently over 40 for most bases within each read. Blue and black lines represent samples <tt>s1</tt> and <tt>s2</tt> in case of Element and <tt>bl</tt> and <tt>ch</tt> in case of Illumina.")

Both datasets are exceptionally good. However, the elemnt data has even higher quality.

# Conclusion

Now you can easily evaluate quality of your own data in Galaxy!

tools_json:
["fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1"]


topic: introduction
tutorial: dummy
title: Upload data to Galaxy
difficulty: introductory
time_estimation: 40m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 97eff559278b0171dd4784621704c3aa
last_modified: 2026-05-15T22:28:13.720084
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/dummy/tutorial.html

content:
There are several ways to get data into Galaxy:
- Uploading a file by pasting in a URL
- Uploading a file from the computer
- Uploading using a FTP server
- Import an entire history
- Import data from a shared library
- ...

> <agenda-title></agenda-title>
> 1. TOC
> {:toc}
>
{: .agenda}

# Upload a file via an URL

> <hands-on-title>Data upload from URL</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Copy the following URL
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    ````
>
> 3. Import it in the history
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

# Import from a shared library

> <hands-on-title>Data upload from URL</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Copy the following URL
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    ````
>
> 3. Import it in the history
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}


topic: introduction
tutorial: data-manipulation-olympics
title: Data Manipulation Olympics
difficulty: introductory
time_estimation: 1h
requirements_json: []
tags_json: ["cyoa"]
content_hash: 8912bee40567881eba6ddded731cfa42
last_modified: 2026-05-27T09:01:49.212310
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/data-manipulation-olympics/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->


<!-- set up some variables to easily update tool versions throughout tutorial, since most tools are used many times %} -->
{% assign version_cat="cat1" %}
{% assign version_compute="toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.0" %}
{% assign version_count="Count1" %}
{% assign version_cut_columns="Cut1" %}
{% assign version_cut_advanced="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0" %}
{% assign version_datamash="toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0+galaxy2" %}
{% assign version_filter="Filter1" %}
{% assign version_join="join1" %}

{% assign version_remove_beginning="Remove beginning1" %}
{% assign version_remove_columns_by_header="toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/0.0.1" %}
{% assign version_replace_text_column="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1" %}
{% assign version_replace_text_line="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1" %}
{% assign version_paste="Paste1" %}

{% assign version_select_first="Show beginning1" %}
{% assign version_sort="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1" %}
{% assign version_split="toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.6" %}
{% assign version_unique="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0" %}
{% assign version_wc="wc_gnu" %}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

If you've opened this tutorial via the {% icon level %} icon in Galaxy (top menu bar), you can click on the tool names in the last column to quickly open them in Galaxy and start using them on your own!

<!-- GTN:IGNORE:006 -->

| Operation              | Description                        | Galaxy Tool    |
|------------------------|------------------------------------|----------------|
| Convert format         | Change the file format             | {% icon galaxy-pencil%} Edit attributes |
| Word count             | Count the number of lines, words and characters in a file | {% tool [Line/Word/Character count]({{version_wc}}) %} |
| Sort on a column       | Change the order of the rows based on values in one or more columns | {% tool [Sort]({{version_sort}}) %} |
| Filter                 | Remove rows based on values in one or more columns | {% tool [Filter]({{version_filter}}) %}|
| Counting               | Count occurrences of values in a column   | {% tool [Count]({{version_count}}) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Group on a column      | And perform simple operations (count, mean, min, max etc) | {% tool [Group](Grouping1) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Compute on rows        | to derive new column values from existing ones | {% tool [Compute]({{version_compute}}) %} |
| Find and Replace       | in a specific column               | {% tool [Column Regex Find and Replace]({{version_replace_text_column}}) %}|
| Find and Replace       | on every line                      | {% tool [Regex Find and Replace]({{version_replace_text_line}}) %}|
| Join two Datasets      | side by side on a specified field  | {% tool [Join two Datasets]({{version_join}}) %} |
| Concatenate datasets   | one after the other                | {% tool [Concatenate multiple datasets or collections]({{version_cat}}) %} |
| Remove Beginning       | Good for removing header lines     | {% tool [Remove beginning of a file]({{version_remove_beginning}}) %} |
| Select First lines     | Good for finding top 10s or saving header lines | {% tool [Select first lines]({{version_select_first}}) %} |
| Cut Columns            | By header name                     | {% tool [Remove columns by heading]({{version_remove_columns_by_header}}) %}|
| Cut Columns            | By column number                   | {% tool [Cut columns from a table]({{version_cut_columns}}) %}, {% tool [Advanced Cut columns from a table]({{version_cut_advanced}}) %}|
| Paste                  | Two files side by side             | {% tool [Paste]({{version_paste}}) %} |
| Split file             | Based on values of a column        | {% tool [Split]({{version_split}}) %} |
| Unique                 | Remove duplicate rows              | {% tool [Unique]({{version_unique}}) %} |

**TIP: Adding tools to your Favourites:** If you find yourself frequently using the same tool often but struggle to find it back in the long list of tools, you can **star** your favourite tools in Galaxy!

{% snippet faqs/galaxy/tools_favorite.md %}


In this tutorial, these tools are explained in more detail, and we provide some exercises for you to practice.


# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"](./images/cover.jpg)


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!


# Upload Data

Before we can do any manipulation, we will need some data. Let's upload our table with Olympics results now.

> <hands-on-title>Get data</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `olympics.tsv` via link
>
>    ```
>    {{page.zenodo_link}}/files/olympics.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. **Expand** on the item in your history to see some metadata and a short preview of the contents.
>
> 3. **View** {% icon galaxy-eye %} the dataset by clicking on the eye icon.
>
>    > <question-title></question-title>
>    >
>    > 1. What is the format of the file?
>    > 2. What does each row represent?
>    > 3. How many lines are in the file? (Hint: {% tool [Line/Word/Character count](wc_gnu) %})
>    > 4. How many columns?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. When you expand the `olympics.tsv` dataset in your history (see also screenshot below), you will see `format: tabular`, this is another term for a tab-separated (`tsv`) file.
>    > > 2. Each row represents an athlete's participation in an event. If an athlete competes in multiple events, there is a line for each event.
>    > > 3. 234,523. Look at the expanded view in your history, this tells us there are ~250,000 rows in the dataset. We can get the exact number using the  {% tool [Line/Word/Character count](wc_gnu) %} tool.
>    > > 4. There are 17 columns in this file. There are multiple ways to find this answer:
>    > >    - Count the columns manually (only doable for small files)
>    > >    - In the expanded view, scroll sideways on the dataset preview, at the top the columns are numbered
>    > >    - Click on the {% icon galaxy-info %} i icon on the dataset; here you will find more detailed information about the file and the job that created it.
>    > >      At the bottom is also a preview (peek) of the dataset and numbered columns
>    > >
>    > > ![a screenshot of the expanded view of the dataset in the history, it shows the datatype, number of lines in the file, and a preview
>    > > of the dataset with numbered columns](./images/columns-number.png)
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.tsv` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.tsv` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.




# File Format Conversion

The file we uploaded is a `.tsv` file. This stands for *tab-separated values*. This means that this is a file containing rows and columns, where a TAB character is used to signify a column ends and a new one begins. Galaxy is great at understanding tab-separated files, and most of the data manipulation tools are designed to work with such files.

A similar format you may come across a lot in data science, is the `.csv` file, or *comma-separated values* file. This is the same as `.tsv`, but uses comma (`,`) characters to indicate new columns, instead of TAB (`\t`) characters.

Galaxy can convert these two formats into each other.



> <hands-on-title>Convert Tabular to CSV</hands-on-title>
>
> 1. Convert the `olympics.tsv` file into a CSV file
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="csv (using 'Convert tabular to CSV')" %}
>
> 2. {% icon galaxy-eye %} **View the converted dataset**
>
>    > <question-title></question-title>
>    >
>    > 1. What do you notice?
>    > 2. Why are some values in quotes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The display in the center panel should be indistinguishable from before the conversion; however, if you look at the "quick view" of the data in the history step, you will see the commas separating columns.
>    > > 2. If the data in a column contains a comma (e.g. in this file we have events such as `swimming 5,000 meters`), we put the value in quotes to signify that that comma is part of the data, not a column delimiter.
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


## Converting vs Changing the datatype

The **file format conversion** step changed the dataset itself (by changing TABs to commas), and therefore created another dataset in your history. It is also possible to **change the datatype**; this does not change the underlying file/data, but just tells Galaxy what type of file it is. Galaxy uses this information in a variety of ways, such as:
  - How to display the data when you click the {% icon galaxy-eye %} **View** button
  - Which visualization options to offer
  - Filtering the list of possible inputs for tools (to only allow those of the correct datatype and protect you from mistakes)
  - Which file format conversions to make available
  - etc..


When you upload a file to Galaxy, by default it will attempt to auto-detect the file format. This is very often correct, but some times you may need to manually set the datatype to the correct value.


> <hands-on-title>Change datatype from Tabular to TXT</hands-on-title>
>
> 1. **Change** the datatype of the `olympics.tsv` file to `txt`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="txt" %}
>
> 2. {% icon galaxy-eye %} **View the dataset** again
>
>    > <question-title></question-title>
>    >
>    > 1. What do you see?
>    > 2. Why didn't this step create a new item in your history?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Since Galaxy now no longer know that this is a tabular file with rows and columns,
>    > > it displays the data as-is, no longer arranging the view into neat columns.
>    > >
>    > > 2. The file itself did not change, only the metadata. We simply told Galaxy to treat this file as a plain text file, instead of a tabular file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Change** the file back from `txt` to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}



# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the year?
>    > 2. Do we want ascending or descending order if we want the oldest games at the top?
>    > 3. What should we do with the very first row (the one with the header names?)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 12
>    > > 2. The file should be sorted in ascending (increasing) order
>    > > 3. The header line should always stay on top, so we want to ignore it when we sort the file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} tool
>    - Read the help text at the bottom of the tool form
>    - Which settings do you think we need to use?
>
> 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
>    - {% icon param-text %} *"Number of header lines"*: `1``
>    - Column selections
>      - {% icon param-select %} *"on column"*: `Column 12`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
>
> 4. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top of the file now?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful (e.g. `Olympics in chronological order`)
>    - It can be easy to lose track of what steps you have performed and what each output contains, especially if you are doing many
>      data manipulation steps. Do future-you a favour and give datasets good names!
>
{: .hands_on}

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, and then within each year, we sort again alphabetically by name. The sort tool can do this!


> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains athlete names?
>    > 2. Do we want ascending or descending order if we want to sort alphabetically with A first?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 2
>    > > 2. The file should be sorted in ascending (increasing) order
>    > >
>    > {: .solution}
>    {: .question}
>
>
> 2. {% icon galaxy-refresh %} **Rerun** the sort tool with the following parameters:
>    - All parameter from the first step should already be set for you, and should remain the same
>    - {% icon param-repeat %} Insert Column selections
>      - {% icon param-select %} *"on column"*: `Column 2`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Alphabetical sort`
>
> 3. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful
>
{: .hands_on}

## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> > <br>
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Height is listed in column 7. Since this is a simple number (no scientific notation), we can use *fast numerical sort*,
> >    and because we want the tallest on top, we will need to sort in *descending* (decreasing) order.
> > 2. Rerun the tool for step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Here we don't get a clear answer. That is because not all athletes have height data available, and blank fields (`NA`) are being sorted to the top. We can filter out rows with missing data to get our answer (see [Filter](#filtering) section to learn how to do this). For now we cannot answer this question with just the sort tool. Some times multiple tools are required to perform such tasks. The [exercise section at the end of this tutorial](#exercises-putting-it-all-together) has many exercises that require a combination of different tools.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> > 1. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Descending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 2. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Ascending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 12` (year column)
> >     - {% icon param-select %} *"in"*: `Descending order` (most recent on top)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >   - {% icon param-repeat %} Insert Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7` (height column)
> >     - {% icon param-select %} *"in"*: `Descending order` (tallest first)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).

## Filtering based on column contents

> <hands-on-title>Filter table based on a column</hands-on-title>
>
> We will filter the file to show only winter Olympics
>
> 1. Look at the `olympics.tsv` file and answer the following questions
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains this information?
>    > 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 13, the column with the *season* header
>    > > 2. The values can be `Summer` or `Winter`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the tool {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %}
>    - read the help text at the bottom of the tool carefully
>    - get familiar with how to indicate columns and write expressions for filtering
>
>    > <question-title></question-title>
>    >
>    > 1. How would you write the expressions for the following conditions:
>    >    1. column 6 must be 'Yes'
>    >    2. column 13 must be smaller than 75
>    >    3. column 7 cannot be 'NA'
>    >    4. column 2 cannot be empty
>    >
>    > 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    >    How would you write expressions for the following filtering conditions:
>    >    1. column 5 is larger than 2 or smaller than -2
>    >    2. column 5 is larger than 2 and smaller than 10
>    >    3. the sum of columns 4 and 6 is greater than or equal to 25,000
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The answers are:
>    > >    1. `c6=='Yes'`
>    > >    2. `c13<75`
>    > >    3. `c7!='NA'`
>    > >    4. `c2!=''`
>    > >
>    > > 2. The answers are:
>    > >    1. `c5>2 or c5<-2`
>    > >    2. `c5>2 and c5<10`
>    > >    3. `c4+c6 >= 25000`
>    > >
>    > {: .solution}
>    {: .question}
>
>    Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics:
>
> 3. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c13=='Winter'`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 4. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > How many lines are in this file? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 44,681 (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Repeat** the step for the Summer Olympics
>
>    > <question-title></question-title>
>    >
>    > 1. How many lines do you expect in this file?
>    > 2. How many lines are in this file? Were you right?
>    >
>    > > <solution-title noprefix>Hints</solution-title>
>    > >
>    > > 1. Use the {% tool [Line/Word/Character count]({{version_wc}}) %} to find the number of lines in the `olympics.tsv` file and subtract the number of rows in the Winter Olympics file
>    > > 2. Be careful to consider whether these counts include the header line of the file or not
>    > >
>    > {: .solution}
>    >
>    > > <solution-title noprefix>Answers</solution-title>
>    > >
>    > > 1. The original file has 234,523 lines, and the Winter Olympics had 44,681 lines. So we would expect 234,523 - 44,681 = 189,842 rows of data. Since we have subtracted the header line in this equation as well, we expect the Summer Olympics file to have 1 more line that this, so 189,843 total lines.
>    > > 2. 189,843. If you were off by one or two lines, it may have been that you counted the header lines double.
>    > > <br>
>    > > It is always useful to take a moment to think about the expected outcome; this makes it easier to spot mistakes and will save you time in the long run.
>    > >
>    > {: .solution}
>    {: .question}
>
> 6. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive (e.g. `Filter: Winter Olympics only` etc)
>    - This is primarily to get you into the habit. If this is a file you might want to view or use again later, it will be hard to find back again unless you name it well.
>
{: .hands_on}

## Advanced Filtering with Python Syntax

In addition to basic filtering, the Galaxy filtering tool allows for more advanced filtering using Python syntax. This can be particularly useful when you need to apply more complex conditions or manipulate the data in specific ways.

> <comment-title> Understanding Python Syntax for Filtering </comment-title>
>
>    Before we dive into specific examples, let's briefly discuss some basic Python concepts that will help you understand how to write these filtering expressions:
>
> - **Methods**: In Python, methods are functions that are associated with an object. For example, in the expression `c2.startswith('Liu')`, `startswith` is a method that checks if the value in column 2 (`c2`) starts with the substring 'Liu'.
>
> - **Boolean Expressions**: These are expressions that evaluate to either `True` or `False`. For example, `c2.startswith('Liu')` will return `True` if the value in column 2 starts with 'Liu', and `False` otherwise.
>
> - **Comparison Operators**: These include `==` (equal to), `!=` (not equal to), `<` (less than), `>` (greater than), `<=` (less than or equal to), and `>=` (greater than or equal to). These operators are used to compare values and are commonly used in filtering.
>
> - **String Methods**: Strings in Python have several built-in methods that can be used for filtering. Some commonly used ones include:
>   - `startswith(substring)`: Checks if a string starts with a specified substring.
>   - `endswith(substring)`: Checks if a string ends with a specified substring.
>   - `find(substring)`: Returns the lowest index of the substring if it is found in the string; otherwise, it returns `-1`.
>   - `count(substring)`: Returns the number of non-overlapping occurrences of the substring in the string.
>
{: .comment}

These concepts will be used in the filtering expressions we write in Galaxy. Let's now explore some specific scenarios where you might use these expressions.


### Filtering Based on Substring Occurrence

We want to filter rows where a particular column contains a specific substring using the `cX.find()` method, where `X` is the column number.

For example, while you can easily filter Summer or Winter Olympics using `c13=="Summer"` or `c13=="Winter"` if the data is stored in a dedicated column, sometimes the data may not be that straightforward. The relevant information might be embedded in another column, like in `c11` with entries such as "1992 Summer Olympics".

In such cases, you can use substring filtering:

> <question-title></question-title>
>
> 1. How would you filter the rows where the 11th column contains "Summer"?
> 2. How would you filter the rows where the 11th column does **not** contain "Summer"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c11.find('Summer') != -1`
> > 2. `c11.find('Summer') == -1`
> >
> {: .solution}
{: .question}

> <hands-on-title>Filtering Based on Substring Occurrence</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file for rows where column 11 contains "Summer":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c11.find('Summer') != -1`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string "Summer" in column 11? (Hint: use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `189843`(this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on }


### Filtering Rows that Start or End with a Specific Value

Filter rows based on whether a column starts or ends with a particular value using `cX.startswith()` or `cX.endswith()`.

For example, if we want to:
 - Find all athletes whose names start with "Liu".
 - Find all teams whose names end with "China".


> <question-title></question-title>
>
> 1. How would you filter rows where the 2nd column starts with "Liu"?
> 2. How would you filter rows where the 9th column ends with "China"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c2.startswith('Liu')`
> > 2. `c9.endswith('China')`
> >
> {: .solution}
{: .question}


> <hands-on-title>Filtering Rows that Start or End with a Specific Value</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file to find all athletes whose names start with "Liu":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c2.startswith('Liu')`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string start with "Liu" in column 2? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `324` (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on}


### Filtering Based on the Count of Substring Occurrences

Sometimes, you may need to filter rows based on **how many times** a certain substring appears in a column. This can be done using the `cX.count()` method.

For example, if we want to find athletes with three-part names by counting spaces in the name column.

> <hands-on-title>Filtering Based on the Count of Substring Occurrences</hands-on-title>
>
> We want to filter the dataset to find athletes with three-part names.
>
> 1. Have a look at the `olympics.tsv` file
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the names of athletes?
>    > 2. For athletes with 3-part names, how many spaces do we expect in that column?
>    > 3. What expression would you use to find these cases?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Column 2 contains the names
>    > > 2. For 3-part names we expect 2 space characters
>    > > 3. `c2.count(' ') == 2`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>     - {% icon param-file %} *"Filter"*: `olympics.tsv`
>     - {% icon param-text %} *"With the following condition"*: `c2.count(' ') == 2`
>     - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 3. {% icon galaxy-eye %} **View** the filtered file.
>
>     > <question-title></question-title>
>     >
>     > 1. How many rows contain three-part names (i.e., two spaces) in column 2?
>     >
>     > > <solution-title>Answers</solution-title>
>     > >
>     > > 1. `11259` (this is including the header line)
>     > >
>     > {: .solution}
>     {: .question}
>
> 4. **Exercises.** Now let's take it a step further and filter athletes with even longer names.
>
>    > <question-title></question-title>
>    >
>    > 1. How would you filter rows where the 2nd column contains **three spaces** (i.e., four-part names)?
>    > 2. How many medals were won by athletes with five-part names **or more**?
>    > 3. How many parts does the longest name have? Give an example of such a name.
>    > 4. Bonus: **How many** athletes have 6-part names won a medal?
>    >
>    > > <solution-title>Hints</solution-title>
>    > >
>    > > 1. None ;)
>    > > 2. You can also use greater than or less than symbols in your expression. You can filter on two lines using  logical operators like `and`.
>    > > 3. Keep increasing the count until no results are returned
>    > > 4. The number of lines returned indicates the number of medals won by long-named athletes, to
>    > >    find out how many different athletes have such a name, we can use the {% tool [Count - occurrences of each record](Count1) %}
>    > >    tool on Column 2.
>    > {: .solution}
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `c2.count(' ') == 3`
>    > > 2. `c17 != 'NA' and c2.count(' ') >= 4` returns 27 lines, meaning 26 medals won accounting for the header line excluding the header line. Appart from the name you need to additionally filter out athletes that did not won a medal.
>    > > 3. `c2.count(' ') == 5` is the maximum value to still return results.
>    > >    Five spaces means a 6-part name. For example Patricia Galvin de la Tour d'Auvergne
>    > > 4. `c17 != 'NA' and c2.count(' ') == 5` returns 5 records. So 5 medals were won by these long-named athletes.
>    > >     Using the {% tool [Count - occurrences of each record](Count1) %} to count occurrences in column 2, we
>    > >     discover that there are **3 different athletes with 6-part names**.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Exercises

Ok, time to train! let's see if you can use the {% tool [Filter]({{version_filter}}) %} tool to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm?
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Expand the output or use the tool {% tool [Line/Word/Character count]({{version_wc}}) %} to see the number of lines in the file
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> >
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `c17=='Gold'`)
> >  2. 24,633  (Expression: `c17=='Gold' or c17=='Silver' or c17=='Bronze'`, or `c17!='NA'`)
> >  3. 131     (Expression: `c17=='Gold' and c12==2018` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `c17!='NA' and c7>=170 and c7<=180`)
> >  5. 812     (Expression: `c17=='Gold' and (c7<160 or c7>190)` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count. The tool {% tool [**Count** occurrences of each record]({{version_count}}) %} does exactly this.


> <hands-on-title>Count occurrences of values in columns</hands-on-title>
>
> Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)
>
> 1. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11`
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How many different Olympic games are in our file?
>    > 2. Which Olympic games had the most participations? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 52; there are 53 lines in the resulting file, with one line containing the value of the column header (`games`).
>    > >
>    > >    The resulting file looks something like:
>    > >
>    > >     ```
>    > >     615	1896 Summer Olympics
>    > >     2503	1900 Summer Olympics
>    > >     2643	1904 Summer Olympics
>    > >     3213	1908 Summer Olympics
>    > >     4610	1912 Summer Olympics
>    > >     3448	1920 Summer Olympics
>    > >     5242	1924 Summer Olympics
>    > >     358	1924 Winter Olympics
>    > >     4493	1928 Summer Olympics
>    > >     ...
>    > >     ```
>    > >
>    > > 2. 1996 Summer Olympics. (10501 participations)
>    > >
>    > {: .solution}
>    {: .question}
>
>    You may have noticed that we could have selected multiple columns in this tool. This lets us count on combinations of columns.
>    Let's try counting the number of men and women in each olympic games.
>
> 3. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11, Column 3`
>
> 4. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>    >
>    > 1. How many women were in the first Olympic games?
>    >
>    > 2. Which Olympic games had the most women participants? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice).
>    > >    The file looks something like this:
>    > >    ```
>    > >    2	F	1896 Summer Olympics
>    > >    43	F	1900 Summer Olympics
>    > >    17	F	1904 Summer Olympics
>    > >    55	F	1908 Summer Olympics
>    > >    97	F	1912 Summer Olympics
>    > >    132	F	1920 Summer Olympics
>    > >    269	F	1924 Summer Olympics
>    > >    ```
>    > >
>    > > 2. 2020 Summer Olympics (4652)
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting tool above, we would get a results file for each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

To answer these types of questions we can use a slightly more advanced tool, called {% tool [Datamash]({{version_datamash}}) %}. This tool can do a lot more than counting, but here we will show you how to use it to count in this way.


> <hands-on-title>Counting number of different sports per Olympic Games</hands-on-title>
>
> We will now determine how many different sport there were in each of the different Olympics
>
> 1. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
>    - {% icon param-toggle %} *"Input file has a header line"*: `Yes`
>    - {% icon param-toggle %} *"Sort Input"*: `Yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Count Unique values`
>      - {% icon param-select %} *"On Column"*: `Column: 15` (the `sport` column)
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. Why did we tell the tool to sort the file?
>    > 2. How many sport were in the first Olympics? How many in the latest?
>    > 3. Which Olympics had the most different sports?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. This is for technical reasons; the tool assumes that your file is sorted (on the column that you supplied to the "Group by fields" parameter).
>    > >    This assumption can greatly speed up the calculation, but will give incorrect results if the file was not actually sorted.
>    > >    If you know for sure that your data is sorted in the right way, you can skip this step. If you are unsure, enable this parameter to be safe, and the
>    > >    tool will perform the sorting step before its calculation.
>    > > 2. 10 and 38.
>    > > 3. The 2020 Summer Olympics had the most different sports (38)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} the output to something descriptive
>
{: .hands_on}


Note: Datamash can do a lot more than counting,  and we will showcase some of these other operatins in the [Grouping](#grouping) section.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting participations (rows), we can use the simple {% tool [Count]({{version_count}})%} tool
> > 2. Since we are counting a bit more complex question, we need the {% tool [Datamash]({{version_datamash}}) %} tool
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations
> >  2. 15 and 250.
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> >  1. {% tool [Count]({{version_count}}) %} with the following parameters:
> >     - {% icon param-file %} *"from dataset"*: `olympics.tsv`
> >     - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 9` (the `team` column)
> >     - {% icon param-select %} *"How should the results be sorted?"*: `With the most common values first`
> >
> >     This gives an output like:
> >     ```
> >     17286	United States
> >     11700	France
> >     10230	Great Britain
> >     8898	Italy
> >     7988	Canada
> >     ```
> >
> >  2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
> >   - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
> >   - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
> >   - {% icon param-toggle %} *"Input file has a header line"*: `yes`
> >   - {% icon param-toggle %} *"Sort Input"*: `yes`
> >   - *"Operation to perform on each group"*:
> >       - {% icon param-select %} *"Type"*: `Count Unique values`
> >       - {% icon param-select %} *"On Column"*: `Column: 9` (the `team` column)
> >
> >   This gives an output like:
> >   ```
> >   1896 Summer Olympics	15
> >   1900 Summer Olympics	29
> >   1904 Summer Olympics	13
> >   ...
> >   2016 Summer Olympics	280
> >   2018 Winter Olympics	108
> >   2020 Summer Olympics	250
> >   ```
> >
> {: .solution}
{: .question}








# Grouping

Often we may want to group rows based on a value in a column, and perform some operation on the resulting rows. For example we would like to group the olympics data by one value (e.g. year, country, sport), and determine some value for each group (e.g. number of medals won, average age of athletes, etc).

In the [counting](#counting) section of this tutorial we show how to get answers that require a count (e.g. number of medals won), but sometimes we want to do something more complex, like calculating the average height of athletes in a group, say per country or per sport. This section will show some example of these types of questions.

We can use the {% tool [Datamash]({{version_datamash}}) %} tool for this purpose.

> <hands-on-title>Tallest athlete per sport</hands-on-title>
>
> We would like to answer the following question: *How tall was the tallest athlete of each sport?*
>
> 1. Open the {% tool [Datamash]({{version_datamash}}) %} tool and read the help section at the bottom
>
>    > <question-title></question-title>
>    >
>    > Which settings do you think we need to provide to answer our question?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - *"Group by fields"*: We want to group by sport (Column 15).
>    > > - *"Sort"*: `Yes`. This may not be obvious, but because our file is currently not sorted by our chosen group (sport), we need to tell the tool to do this.
>    > > - *"Skip NA or NaN values"*: since we do have NA values for athletes for whom height data is unknown, we should set this to `Yes`
>    > > - Our file has a header line, so we should indicate this as well
>    > > - *"Operation"*: we want to determine the **Maximum** height (Column 7)
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `15`
>    - {% icon param-toggle %} *"Sort input"*: `yes`
>    - {% icon param-toggle %} *"Input file has a header line"*: `yes`
>    - {% icon param-toggle %} *"Skip NA or NaN values"*: `yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Maximum`
>      - {% icon param-select %} *"On Column"*: `Column: 7`
>
> 3. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How tall was the tallest athlete in basketball? And what about karate?
>    > 2. Why do some sports have a value of `inf`?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Basketball's tallest athlete was 192cm. For Karate it is 163.
>    > > 2. Our dataset had quite a number of `NA` (unknown) values in the height column, especially for the earlier Olympics. For sports that had only NA values, there is no maximum so the tool outputs `inf` instead.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Grouping on multiple columns

You may have noticed that we could also provide multiple columns to group on. If we do this, we can compute values for combinations of groups, such as sex and sport, to find e.g. the tallest woman in basketball or the shortest man per Olympics. There are also many more options for the computation we perform, so perhaps we are more interested not in the tallest athlete, but the average height. Let's perform some 

tools_json:
["Line/Word/Character count", "{{version_wc}}", "Sort", "{{version_sort}}", "Filter", "{{version_filter}}", "Count", "{{version_count}}", "Datamash", "{{version_datamash}}", "Group", "Grouping1", "Compute", "{{version_compute}}", "Column Regex Find and Replace", "{{version_replace_text_column}}", "Regex Find and Replace", "{{version_replace_text_line}}", "Join two Datasets", "{{version_join}}", "Concatenate multiple datasets or collections", "{{version_cat}}", "Remove beginning of a file", "{{version_remove_beginning}}", "Select first lines", "{{version_select_first}}", "Remove columns by heading", "{{version_remove_columns_by_header}}", "Cut columns from a table", "{{version_cut_columns}}", "Advanced Cut columns from a table", "{{version_cut_advanced}}", "Paste", "{{version_paste}}", "Split", "{{version_split}}", "Unique", "{{version_unique}}", "Import", "upload1", "wc_gnu", "Sort - data in ascending or descending order", "**Filter** data on any column using simple expressions", "Count - occurrences of each record", "Count1", "**Count** occurrences of each record", "Compute on rows", "Column Regex Finda and Replace", "Unique occurrences of each record", "Unique - occurrences of each record", "Join two Datasets side by side on a specified field", "Split by group", "Filter - data on any column using simple expressions", "**Count**"]


topic: introduction
tutorial: galaxy-intro-peaks2genes
title: From peaks to genes
difficulty: introductory
time_estimation: 3H
requirements_json: []
tags_json: ["deutsch", "espa\u00f1ol", "italiano"]
content_hash: bab6f01a4f6602f5c7a8842aec5df4d1
last_modified: 2026-05-27T09:01:49.214398
zenodo_link: https://doi.org/10.5281/zenodo.1025586
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-peaks2genes/tutorial.html

content:
We stumbled upon a paper ({% cite Li2012 %}) called *"The histone acetyltransferase MOF is a key regulator of the embryonic stem cell core transcriptional network"*. The paper contains the analysis of possible target genes of an interesting protein called Mof. The targets were obtained by ChIP-seq in mice and the raw data is available through [GEO](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE37268).
However, the list of genes is neither in the supplement of the paper, nor part of the GEO submission.
The closest thing we could find is a file in GEO containing a list of the regions where the signal is significantly enriched (so called *peaks*):

1 | 3660676 | 3661050 | 375 | 210 | 62.0876250438913 | -2.00329386666667
1 | 3661326 | 3661500 | 175 | 102 | 28.2950833625942 | -0.695557142857143
1 | 3661976 | 3662325 | 350 | 275 | 48.3062708406486 | -1.29391285714286
1 | 3984926 | 3985075 | 150 | 93 | 34.1879823073944 | -0.816992
1 | 4424801 | 4424900 | 100 | 70 | 26.8023246007435 | -0.66282

**Table 1** Subsample of the available file

The goal of this exercise is to **turn this list of genomic regions into a list of possible target genes**.

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

> <hands-on-title>Open Galaxy</hands-on-title>
>
> 1. Browse to a Galaxy instance: the one recommended by your instructor or one in the list **Galaxy instance** on the head of this page
> 2. Log in or register (top panel)
>
>    ![Login or Register on the top panel](../../images/login_register.png)
{: .hands_on}

The Galaxy interface consist of three main parts. The available tools are listed on the left, your analysis history is recorded on the right, and the central panel will show the tools and datasets.

![Galaxy interface screenshot showing history panel on the right, tools panel on the left, and main panel at the center](../../images/galaxy_interface.png "The Galaxy interface")

Let's start with a fresh history.


> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you have an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to make it easy to recognize
>
>    > <tip-title>Rename a history</tip-title>
>    >
>    > * Click on the title of the history (by default the title is `Unnamed history`)
>    >
>    >   ![Renaming history]({% link shared/images/rename_history.png %})
>    >
>    > * Type `Galaxy Introduction` as the name
>    > * Press <kbd>Enter</kbd>
>    >
>    {: .tip}
>
{: .hands_on}

## Data upload

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Download the list of peak regions (the file [GSE37268_mof3.out.hpeak.txt.gz](https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSE37268&format=file&file=GSE37268%5Fmof3%2Eout%2Ehpeak%2Etxt%2Egz)) from [GEO](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE37268) to your computer
> 2. Click on the upload button in the upper left of the interface
>
>    ![Upload data icon](../../../galaxy-interface/images/upload_button.png)
>
> 3. Press **Choose local files** and search for your file on your computer
> 4. Select `interval` as **Type**
> 5. Press **Start**
> 6. Press **Close**
> 7. Wait for the upload to finish. Galaxy will automatically unpack the file.
>
> 8. After this you will see your first history item in Galaxy’s right pane. It will go through
> the gray (preparing/queued) and yellow (running) states to become green (success):
>
>    ![History section](../../images/intro_01.png)
>
>    Directly uploading files is not the only way to get data into Galaxy
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="interval" %}
>
>    > <tip-title>Importing data to Galaxy</tip-title>
>    > There are [more options]({% link topics/galaxy-interface/tutorials/get-data/slides.html %}) for advanced users.
>    {: .tip}
>
{: .hands_on}


> <comment-title>Interval file format</comment-title>
> **Interval** format is a Galaxy format for representing genomic intervals. It is tab-separated, but has the added requirement that three of the columns must be:
> - chromosome ID
> - start position (0-based)
> - end position (end-exclusive)
>
> An optional strand column can also be specified, and an initial header row can be used to label the columns, which do not have to be in any special order. Unlike BED format (see below) arbitrary additional columns can also be present.
>
> You can find more information about formats that can be used in Galaxy at the [Galaxy Data Formats page](https://usegalaxy.org/static/formatHelp.html).
{: .comment}


> <hands-on-title>Inspect and edit attributes of a file</hands-on-title>
>
> 1. Click on the file in the history panel
>
>    Some meta-information (e.g. format, reference database) about the file and the header of the file are then displayed, along with the number of lines in the file (48,647):
>
>    ![Expanded file in history](../../images/input_expanded_file.png)
>
> 2. Click on the {% icon galaxy-eye %} (eye) icon (**View data**) in your dataset in the history
>
>    The content of the file is displayed in the central panel
>
> 3. Click on the {% icon galaxy-pencil %} (pencil) icon (**Edit attributes**) in your dataset in the history
>
>    A form to edit dataset attributes is displayed in the central panel
>
> 4. Search for `mm9` in **Database/Build** attribute and select `Mouse July 2007 (NCBI37/mm9)` (the paper tells us the peaks are from `mm9`)
>
>    ![The database/build version can be selected from a dropdown menu. Users can start typing the database name to filter the list](../../images/Search-for-mm9.PNG)
>
> 5. Click on **Save** at the bottom
> 6. Add a tag called `#peaks` to the dataset to make it easier to track in the history
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
>    The dataset should now look like below in the history
>
>    ![Peaks file](../../images/input_tagged_file.png){: width="250px" height="300px"}
>
{: .hands_on}

In order to find the related genes to these peak regions,
we also need a list of genes in mice, which we can obtain from UCSC.

> <hands-on-title>Data upload from UCSC</hands-on-title>
>
> 1. Search for `UCSC Main` in the tool search bar (top left)
>
>     ![UCSC Main tool in tools section](../../images/101_01.png)
>
> 2. Click on `UCSC Main` {% icon tool %}
>
>     You will be taken to the **UCSC table browser**, which looks something like this:
>
>     ![UCSC table browser interface](../../images/intro_02.png)
>
> 3. Set the following options:
>     - *"Clade"*: `Mammal`
>     - *"Genome"*: Search for `mm9` and select `Mouse (mm9)`
>     - *"Assembly"*: `July 2007 (NCBI37/mm9)`
>     - *"Group"*: `Genes and Gene Predictions`
>     - *"Track"*: `RefSeq Genes`
>     - *"Table"*: `refGene`
>     - *"Region"*: `Genome`
>     - *"Output format"*: `BED - browser extensible data`
>     - *"Send output to"*: `Galaxy` (only)
>
> 4. Click on the **Get output** button
>
>    You will see the next screen:
>
>    ![Output settings](../../images/intro_03.png)
>
> 5. Make sure that *"Create one BED record per"* is set to `Whole Gene`
> 6. Click on the **Send query to Galaxy** button
> 7. Wait for the upload to finish
> 8. Rename our dataset to something more recognizable like `Genes`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Genes" %}
>
> 9. Add a tag called `#genes` to the dataset to make it easier to track in the history
>
{: .hands_on}

> <comment-title>BED file format</comment-title>
> The **BED - Browser Extensible Data** format provides a flexible way to encode gene regions. BED lines have three required fields:
> - chromosome ID
> - start position (0-based)
> - end position (end-exclusive)
>
> There can be up to and nine additional optional fields, but the number of fields per line must be consistent throughout any single set of data.
>
> You can find more information about it at [UCSC](https://genome.ucsc.edu/FAQ/FAQformat#format1) including a description of the optional fields.
{: .comment}

Now we have collected all the data we need to start our analysis.

# Part 1: Naive approach

We will first use a "naive" approach to try to identify the genes that the peak regions are associated with. We will identify genes that overlap at least 1bp with the peak regions.

## File preparation

Let's have a look at our files to see what we actually have here.

> <hands-on-title>View file content</hands-on-title>
>
> 1. Click on the {% icon galaxy-eye %} (eye) icon (**View data**) of the peak file to view the content of it
>
>    It should look like this:
>
>    ![Contents of the peak file](../../images/intro_04.png)
>
> 2. View the content of the regions of the genes from UCSC
>
>    ![Contents of UCSC file](../../images/intro_05.png)
>
{: .hands_on}

To view both the files side by side for comparision, enable the window manager.

{% snippet faqs/galaxy/features_scratchbook.md %}

> <question-title></question-title>
>
> While the file from UCSC has labels for the columns, the peak file does not. Can you guess what the columns stand for?
>
>
> > <solution-title></solution-title>
> >
> > This peak file is not in any standard format and just by looking at it, we cannot find out what the numbers in the different columns mean. In the paper the authors mention that they used the peak caller [HPeak](https://www.ncbi.nlm.nih.gov/pubmed/20598134).
> >
> > By looking at the HPeak manual we can find out that the columns contain the following information:
> >
> >  - chromosome name by number
> >  - start coordinate
> >  - end coordinate
> >  - length
> >  - location within the peak that has the highest hypothetical DNA fragment coverage (summit)
> >  - not relevant
> >  - not relevant
> >
> {: .solution}
{: .question}

In order to compare the two files, we have to make sure that the chromosome names follow the same format.
As we can see, the peak file lacks `chr` before any chromosome number. But what happens with chromosome 20 and 21? Will it be X and Y instead? Let's check:

> <hands-on-title>View end of file</hands-on-title>
>
> 1. Search for {% tool [Select last lines from a dataset (tail)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.5+galaxy2) %} tool and run with the following settings:
>     - *"Text file"*: our peak file `GSE37268_mof3.out.hpeak.txt.gz`
>     - *"Operation"*: `Keep last lines`
>     - *"Number of lines"*: Choose a value, e.g. `100`
> 2. Click **Run Tool**
> 3. Wait for the job to finish
> 4. Inspect the file through the {% icon galaxy-eye %} (eye) icon (**View data**)
>
>    > <question-title></question-title>
>    >
>    > 1. How are the chromosomes named?
>    > 2. How are the chromosomes X and Y named?
>    >
>    > > <solution-title></solution-title>
>    > > 1. The chromosomes are just given by their number. In the gene file from UCSC, they started with `chr`
>    > > 2. The chromosomes X and Y are named 20 and 21
>    > {: .solution }
>    {: .question}
{: .hands_on}

In order to convert the chromosome names we have therefore two things to do:

1. add `chr`
2. change 20 and 21 to X and Y

> <hands-on-title>Adjust chromosome names</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2) %} in a specific column with the following settings:
>     - *"File to process"*: our peak file `GSE37268_mof3.out.hpeak.txt.gz` ({% icon warning %} careful! select the initial peaks file)
>     - *"in column"*: `Column: 1`
>     - *"Find pattern"*: `[0-9]+`
>
>         This will look for numerical digits
>
>     - *"Replace with"*: `chr&`
>
>         `&` is a placeholder for the find result of the pattern search
>
> 2. Rename your output file `chr prefix added`.
>
> 3. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2) %}  : Let's rerun the tool with two more replacements
>    - *"File to process"*: the output from the last run, `chr prefix added`
>    - {% icon param-repeat %} Replacement
>      - *"in column"*: `Column: 1`
>      - *"Find pattern"*: `chr20`
>      - *"Replace with"*: `chrX`
>    - {% icon param-repeat %} Insert Replacement
>      - *"in column"*: `Column: 1`
>      - *"Find pattern"*: `chr21`
>      - *"Replace with"*: `chrY`
>
>    > <tip-title>Rerunning a tool</tip-title>
>    >
>    > * Expand the dataset information
>    > * Press the {% icon galaxy-refresh %} icon (**Run this job again**)
>    {: .tip}
>
> 4. Inspect the latest file through the {% icon galaxy-eye %} (eye) icon. Have we been successful?
>
>    We have quite a few files now and need to take care to select the correct ones at each step.
>
>    > <question-title></question-title>
>    >
>    > How many regions are in our output file? You can click the name of the output to expand it and see the number.
>    >
>    > > <solution-title></solution-title>
>    > > It should be equal to the number of regions in your first file, `GSE37268_mof3.out.hpeak.txt.gz`: 48,647
>    > > If yours says 100 regions, then you have run it on the `Tail` file and need to re-run the steps.
>    > {: .solution }
>    {: .question}
>
> 5. Rename the file to something more recognizable, e.g. `Peak regions`
{: .hands_on}

## Analysis

Our goal is to compare the 2 region files (the genes file and the peak file from the publication)
to know which peaks are related to which genes. If you only want to know which peaks are located **inside** genes (within the gene body) you
can skip the next step. Otherwise, it might be reasonable to include the **promoter** region of the genes into the comparison, e.g. because
you want to include transcriptions factors in ChIP-seq experiments. There is no strict definition for promoter region but 2kb upstream of the TSS (start of region) is commonly used. We'll use the **Get Flanks** tool to get regions 2kb bases upstream of the start of the gene to 10kb bases downstream of the start (12kb in length). To do this we tell the Get Flanks tool we want regions upstream of the start, with an offset of 10kb, that are 12kb in length, as shown in the diagram below.

![Get Flanks](../../images/intro_get_flanks.png)

> <hands-on-title>Add promoter region to gene records</hands-on-title>
>
> 1. {% tool [Get Flanks](toolshed.g2.bx.psu.edu/repos/devteam/get_flanks/get_flanks1/1.0.0) %} returns flanking region/s for every gene, with the following settings:
>     - *"Select data"*: `Genes` file from UCSC
>     - *"Region"*: `Around Start`
>     - *"Location of the flanking region/s"*: `Upstream`
>     - *"Offset"*: `10000`
>     - *"Length of the flanking region(s)"*: `12000`
>
>     This tool returns flanking regions for every gene
>
> 2. Compare the rows of the resulting BED file with the input to find out how the start and end positions changed
>
>    > <tip-title>Inspecting several files using the scratchbook</tip-title>
>    >
>    > * Click **Enable/Disable Scratchbook** on the top panel
>    >
>    >    ![Enable/Disable Scratchbook](../../images/intro_scratchbook_enable.png)
>    >
>    > * Click on the {% icon galaxy-eye %} (eye) icon of the files to inspect
>    > * Click on **Show/Hide Scratchbook**
>    >
>    >    ![Show/Hide Scratchbook](../../images/intro_scratchbook_show_hide.png)
>    {: .tip}
>
> 3. Rename your dataset to reflect your findings (`Promoter regions`)
{: .hands_on}

The output is regions that start from 2kb upstream of the TSS and include 10kb downstream. For input regions on the positive strand e.g. `chr1 134212701 134230065` this gives `chr1 134210701 134222701`. For regions on the negative strand e.g. `chr1 8349819 9289958` this gives `chr1  9279958 9291958`.

You might have noticed that the UCSC file is in `BED` format and has a database associated to it. That's what we want for our peak file as well. The **Intersect** tool we will use can automatically convert interval files to BED format but we'll convert our interval file explicitly here to show how this can be achieved with Galaxy.

> <hands-on-title>Change format and database</hands-on-title>
>
> 1. Click on the {% icon galaxy-pencil %} (pencil) icon in the history entry of our peak region file
> 2. Switch to the **Datatypes** tab
> 3. In section **Convert to Datatype** under *"Target datatype"* select: `bed (using 'Convert Genomic Interval To Bed')`
> 4. Press **Create Dataset**
> 5. Check that the "Database/Build" is `mm9` (the database build for mice used in the paper)
> 6. Again rename the file to something more recognizable, e.g. `Peak regions BED`
{: .hands_on}

It's time to find the overlapping intervals (finally!). To do that, we want to extract the genes which overlap/intersect with our peaks.

> <hands-on-title>Find Overlaps</hands-on-title>
>
> 1. {% tool [Intersect](toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0) %} the intervals of two datasets, with the following settings:
>     - *"Return"*: `Overlapping Intervals`
>     - *"of"*: the UCSC file with promoter regions (`Promoter regions`)
>     - *"that intersect"*: our peak region file from **Replace** (`Peak regions BED`)
>     - *"for at least"*: `1`
>
>    > <comment-title></comment-title>
>    > The order of the inputs is important! We want to end up with a list of **genes**, so the corresponding dataset with the gene information needs to be the first input (`Promoter regions`).
>    {: .comment}
>    ![Genes overlapping peaks](../../images/intro_overlapping_genes.png)
{: .hands_on}

We now have the list of genes (column 4) overlapping with the peak regions, similar to shown above.

To get a better overview of the genes we obtained, we want to look at their distribution across the different chromosomes.
We will group the table by chromosome and count the number of genes with peaks on each chromosome

> <hands-on-title>Count genes on different chromosomes</hands-on-title>
>
> 1. {% tool [Group](Grouping1) %} data by a column and perform aggregate operation on other columns, with the following settings:
>     - *"Select data"*: the result of the intersection
>     - *"Group by column"*: `Column: 1`
>     - Press **Insert Operation** and choose:
>         - *"Type"*: `Count`
>         - *"On column"*: `Column: 1`
>         - *"Round result to nearest integer?"*: `No`
>
>    > <question-title></question-title>
>    >
>    > Which chromosome contained the highest number of target genes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The result varies with different settings, for example, the annotation may change due to updates at UCSC. If you followed step by step, with the same annotation, it should be chromosome 11 with 2164 genes. Note that for reproducibility, you should keep all input data used within the analysis. Rerunning the analysis with the same set of parameters, stored Galaxy, can lead to a different result if the inputs changed e.g. the annotation from UCSC.
>    > {: .solution }
>    {: .question}
>
{: .hands_on}

## Visualization

We have some nice aggregated data, so why not draw a barchart of it?

Before we do that we should polish our grouped data a bit more though.

You may have noticed that the mouse chromosomes are not listed in their correct
order in that dataset (the **Group** tool tried to sort them, but did so
alphabetically).

We can fix this by running a dedicated tool for sorting on our data.

> <hands-on-title>Fix sort order of gene counts table</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} data in ascending or descending order, with the following settings:
>     - *"Sort Query"*: result of running the Group tool
>     - in {% icon param-repeat %} *"Column selections"*
>       - *"on column"*: `Column: 1`
>       - *"in"*: `Ascending order`
>       - *"Flavor"*: `Natural/Version sort (-V)`
>
>    {% snippet faqs/galaxy/tools_similar.md toolname="Sort data in ascending or descending order" toolversion="1.1.1" %}
>
{: .hands_on}

Great, we are ready to plot things!

> <hands-on-title>Draw barchart</hands-on-title>
>
> 1. Click on {% icon galaxy-barchart %} (visualize) icon on the output from the **Sort** tool
> 2. Select `Bar, Line and Scatter` from the main panel
> 3. Click on the **<<** in the upper right corner
> 4. Switch to the **Settings** tab and change the axis labels
> 5. Choose a title below **Title**, e.g. `Gene counts per chromosome`
> 6. Play around with the settings in the **Tracks** tab
> 7. When you are happy, click the {% icon cloud-upload %} icon to **Save** visualization in the top right of the *main panel*
>
>    This will store it to your saved visualisations. Later you can view,
>    download, or share it with others by clicking again on **Visualization** on the left menu bar. In the opened menu bar click on the blue **Saved Visualizations** button on top. In the main panel you can find your saved plot.
>
{: .hands_on}

## Extracting workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step.
Wouldn't it be nice to just convert this history into a workflow that we'll be able to execute again and again?

Galaxy makes this very simple with the `Extract Workflow` option. This means that any time you want to build a workflow, you can just perform it manually once, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis. It also allows you to easily share or publish your analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking on the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click on {% icon galaxy-history-options %} (**History options**) at the top of your history panel and select **Extract Workflow**.
>
>    ![`Extract Workflow` entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
> 3. Replace the **Workflow name** to something more descriptive, for example: `From peaks to genes`
>
> 4. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes.
>
>    Since we did some steps which where specific to our custom peak file, we might want to exclude:
>    - **Select last** {% icon tool %}
>    - all **Replace Text** {% icon tool %} steps
>    - **Convert Genomic Intervals to BED**
>    - **Get flanks** {% icon tool %}
>
> 5. Click on the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created. But where did it go?
>
> 6. Click on **Workflows** in the left menu of Galaxy
>
>    Here you have a list of all your workflows
>
> 7. Select the newly generated workflow and click on **Edit**
>
>    You should see something similar to this:
>
>    ![Editing workflow interface](../../images/intro_06.png)
>
>    > <comment-title>The workflow editor</comment-title>
>    > We can examine the workflow in Galaxy's workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.
>    {: .comment}
>
>     Although we have our two inputs in the workflow they are missing their connection to the first tool (**Intersect** {% icon tool %}), because we didn't carry over some of the intermediate steps.
>
> 8. Connect each input dataset to the **Intersect** {% icon tool %} tool by dragging the arrow pointing outwards on the right of its box (which denotes an output) to an arrow on the left of the **Intersect** box pointing inwards (which denotes an input)
> 9. Rename the input datasets to `Reference regions` and `Peak regions`
> 10. Press **Auto Re-layout** to clean up our view
>     ![Auto re-layouting](../../images/intro_07.png)
> 11. Click on the {% icon galaxy-save %} **Save** icon (top) to save your changes
>     ![Save workflow button]({% link topics/contributing/images/save_workflow.png %}){: width="50%"}
>
>    > <tip-title>Hiding intermediate steps</tip-title>
>    > When a workflow is executed, the user is usually primarily interested in the final product and not in all intermediate steps. By default all the outputs of a workflow will be shown, but we can explicitly tell Galaxy which output to show and which to hide for a given workflow. This behaviour is controlled by the little asterisk next to every output dataset:
>    >
>    > ![Workflow editor mark output](../../../../shared/images/workflow_editor_mark_output.png)
>    >
>    > If you click on this asterisk for any of the output datasets, then *only* files with an asterisk will be shown, and all outputs without an asterisk will be hidden (Note that clicking *all* outputs has the same effect as clicking *none* of the outputs, in both cases all the datasets will be shown).
>    {: .tip}
> 12. Return to the Galaxy home page by clicking the Galaxy icon in the top-left corner of the page.
> 
{: .hands_on}

Now it's time to reuse our workflow for a more sophisticated approach.

# Part 2: More sophisticated approach

In part 1 we used an overlap definition of 1 bp (default setting) to identify genes associated with the peak regions. However, the peaks could be broad, so instead, in order to get a more meaningful definition, we could identify the genes that overlap where most of the reads are concentrated, the **peak summit**. We will use the information on the position of the peak summit contained in the original peak file and check for overlap of the summits with genes.

## Preparation

We again need our peak file, but we'd like to work in a clean history. Instead of uploading it twice, we can copy it to a new history.

> <hands-on-title>Copy history items</hands-on-title>
>
> 1. Create a new history and give it a new name like `Galaxy Introduction Part 2`
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Click on the **History options** {% icon galaxy-history-options %} at the top right of your history. Click on the **Show Histories Side-by-Side**
>
>       You should see both of your histories side-by-side now
>
> 3. Drag and drop the edited peak file (`Peak regions`, after the replace steps), which contains the summit information, to your new history.
> 4. Click on the Galaxy name in the top menu bar (top left) to go back to your analysis window
>
{: .hands_on}

## Create peak summit file

We need to generate a new BED file from the original peak file that contains the positions of the peak summits. The start of the summit is the start of the peak (column 2) plus the location within the peak that has the highest hypothetical DNA fragment coverage (column 5, rounded down to the next smallest integer because some peak summits fall in between to bases). As the end of the peak region, we will simply define `start + 1`.

> <hands-on-title>Create peak summit file</hands-on-title>
>
> 1. {% tool [Compute on rows](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters:
>   - *"Input file"*: our peak file `Peak regions` (the interval format file)
>   - *"Input has a header line with column names?": `No`
>   - In *"Expressions"*:
>       - {% icon param-repeat %} *"Expressions"*
>           - *"Add expression"*: `c2 + int(c5)`
>           - *"Mode of the operation"*: Append
>       - {% icon param-repeat %} *"Expressions"*
>           - *"Add expression"*: `c8 + 1`
>           - *"Mode of the operation"*: Append
>
>   This will create an 8th and a 9th column in our table, which we will use in our next step:
>
> 2. Rename the output `Peak summit regions`
>
{: .hands_on}

Now we cut out just the chromosome plus the start and end of the summit:

> <hands-on-title>Cut out columns</hands-on-title>
> 1. {% tool [Cut](Cut1) %} columns from a table with the following settings:
>   - *"Cut columns"*: `c1,c8,c9`
>   - *"Delimited by Tab"*: `Tab`
>   - *"From"*: `Peak summit regions`
>
>    The output from **Cut** will be in `tabular` format.
>
> 2. Change the format to `interval` (use the {% icon galaxy-pencil %}) since that's what the tool **Intersect** expects.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="interval" %}
>
>    The output should look like below:
>
>    ![Peak summits](../../images/intro_summits.png){: width="200px"}
{: .hands_on}

## Get gene names

The RefSeq genes we downloaded from UCSC did only contain the RefSeq identifiers, but not the gene names. To get a list of gene names in the end, we use another BED file from the Data Libraries.

> <comment-title></comment-title>
> There are several ways to get the gene names in, if you need to do it yourself. One way is to retrieve a mapping through Biomart and then join the two files (**Join two Datasets side by side on a specified field** {% icon tool %}). Another is to get the full RefSeq table from UCSC and manually convert it to BED format.
{: .comment}

> <hands-on-title>Data upload</hands-on-title>
>
> 1. {% tool [Import](upload1) %} `mm9.RefSeq_genes_from_UCSC.bed` from [Zenodo](https://zenodo.org/record/1025586) or from the data library:
>
>    ```
>    https://zenodo.org/record/1025586/files/mm9.RefSeq_genes_from_UCSC.bed
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md genome="mm9" %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md path='Click on "GTN - Material", "Introduction to Galaxy Analyses", "From peaks to genes", and then "DOI: 10.5281/zenodo.1025586"' %}
>
>    As default, Galaxy takes the link as name, so rename them.
>
> 2. Inspect the file content to check if it contains gene names.
>    It should look similar to below:
>    ![Gene names](../../images/intro_gene_names.png)
>
> 3. Rename it `mm9.RefSeq_genes`
> 4. Apply the tag `#genes`
>
{: .hands_on}

## Repeat workflow

It's time to reuse the workflow we created earlier.

> <hands-on-title>Run a workflow</hands-on-title>
> 1. Open the **Workflows** menu (left menu bar)
> 2. Find the workflow you made in the previous section, and select the option **Run** {% icon workflow-run %}
> 3. Choose as inputs our `mm9.RefSeq_genes` (`#genes`) BED file and the result of the **Cut** tool (`#peaks`)
> 4. Click **Run workflow**
>
>    The outputs should appear in the history but it might take some time until they are finished.
>
{: .hands_on}

We used our workflow to rerun our analysis with the peak summits. The **Group** tool again produced a list containing the number of genes found in each chromosome.
But wouldn't it be more interesting to know the number of peaks in each unique gene? Let's rerun the workflow with different settings!

> <hands-on-title>Run a workflow with changed settings</hands-on-title>
> 1. Open the workflow menu (left menu bar)
> 2. Find the workflow you made in the previous section, and select the option **Run**
> 3. Click on **Workflow run settings** {% icon galaxy-wf-options %} and then on **Expanded workflow form**
> 4. Choose as inputs our `mm9.RefSeq_genes` (`#genes`) BED file and the result of the **Cut** tool (`#peaks`)
> 5. Click on the title of the {% icon tool %} **Group** tool to expand the options
> 6. Change the following settings by clicking on the {% icon galaxy-pencil %} (pencil) icon on the left:
>     - *"Group by column"*: `7`
>     - In *"Operation"*:
>       - *"On column"*: `7`
> 7. Click **Run workflow**
{: .hands_on}

Congratulations! You should have a file with all the unique gene names and a count on how many peaks they contained.

> <question-title></question-title>
>
> Which gene has the most number of peaks? 
>
> > <solution-title></solution-title>
> > *Kcnma1*
> > 
> > The list of unique genes is not sorted or you workflow included a sorting step but for the gene names. Try to sort it on your own!
> > You can use the tool "{% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2) %} data in ascending or descending order" or rerun to sort tool of your workflow. Sort on colum `Column: 2` in `Descending order` with `Fast numeric sort (-n)` flavor.
> {: .solution }
{: .question}


# Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

To share a history, click on the {%  icon galaxy-history-options %} history options and select `Share & Manage Access`. On this tab (**Share or Publish**) you can **Make History accessible** or **Share History with Individual Users**:


1. **Make History accessible**

    This generates a link that you can give out to others. Anybody with this link will be able to view your history.

2. **Make History publicly available**

    This will not only create a link, but will also publish your history. This means your history will be listed under `Histories → Published Histories` in the left menu and other users can access the history there.

3. **Share with Individual Users**

    This will share the history only with specific users on the Galaxy instance.

![The menu for sharing histories includes buttons for making the history accessible, publishing it on this Galaxy server, and displays a sharable link to the history. At the bottom is a button for sharing the history with individual users.](../../images/publish.png)

> <hands-on-title>Share history and workflow</hands-on-title>
>
> 1. Share one of your histories with your neighbour
> 2. See if you can do the same with your workflow!
> 3. Find the history and/or workflow shared by your neighbour
>
>    Histories shared with specific users can be accessed by those users under `Histories → Shared with Me`.
>
{: .hands_on}

# Conclusion

{% icon trophy %} You have just performed your first analysis in Galaxy. You also created a workflow from your analysis so you can easily repeat the exact same analysis on other datasets. Additionally you shared your results and methods with others.

tools_json:
["Select last lines from a dataset (tail)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_tail_tool/9.5+galaxy2", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_column/9.5+galaxy2", "Get Flanks", "toolshed.g2.bx.psu.edu/repos/devteam/get_flanks/get_flanks1/1.0.0", "Intersect", "toolshed.g2.bx.psu.edu/repos/devteam/intersect/gops_intersect_1/1.0.0", "Group", "Grouping1", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/9.5+galaxy2", "Compute on rows", "toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1", "Cut", "Cut1", "Import", "upload1"]


topic: introduction
tutorial: galaxy-intro-rdm
title: Introduction to Galaxy as an RDM platform
difficulty: introductory
time_estimation: 3H
requirements_json: []
tags_json: []
content_hash: 2627aa3168aefb79c0412a4aa44a7b17
last_modified: 2026-05-27T09:01:49.215261
zenodo_link: https://zenodo.org/records/18803585/
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-rdm/tutorial.html

content:
This tutorial aims to familiarize you with the Galaxy user interface, with a special focus on highlighting Galaxy's many RDM (Research Data Management) features.

Galaxy has over 10,000 available tools in its [Tool Shed](https://toolshed.g2.bx.psu.edu/), covering a wide variety of scientific domains, ranging from life sciences to astronomy and digital humanities, and covering techniques from simple text manipulation to advanced machine learning and other complex algorithms.

To keep this tutorial accessible for people with different backgrounds, we perform a toy analysis on a tabular dataset, namely a table of all athletes competing in the Olympics. The question we ask ourselves is ***"What is the age distribution of Olympic athletes?"***. In addition, we want to make sure our analysis is reproducible, so that it can be easily repeated on different datasets and shared with others.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Overview

## The Research Data Life Cycle

The research life cycle refers to the series of stages through which a research project or study progresses from inception to completion. Although the specifics of the research process vary across disciplines, they share several key phases that help ensure systematic, rigorous research and reliable results. It ranges from **planning** and designing your study, to **collecting**, **processing**, and **analysing** your data, evaluating results, and finally **preserving** and **sharing** your data and findings for **reuse** by others. As this is an iterative process, it is often referred to as the Research data life cycle.

![RDM life cycle]({% link topics/introduction/images/galaxy-intro-rdm/rdm-overview.png %}){: style="width:40%"}

> <tip-title> Why RDM? Play the Scarytales game of RDM gone wrong! </tip-title>
>
> Good RDM practices are critical to scientific research, to illustrate this in a fun way, have a look at the [RDM Scary Tales](https://forschungsdaten-thueringen.de/rdm-scarytales/articles/overview.html) game (based on the game [Black Stories](https://boardgamegeek.com/boardgame/18803/black-stories))
>
{: .tip}

## Galaxy as part of the RDM Life Cycle

Galaxy supports you in your research throughout the different stages of the life cycle, covering the steps from data collection to data reuse.

![The RDM lifecycle with Galaxy features listed for each stage]({% link topics/introduction/images/galaxy-intro-rdm/rdm-all.png %})

For  more information, see also the [RDMKit Galaxy page](https://rdmkit.elixir-europe.org/galaxy_assembly).


## Watch

Below is a 5-minute video introducing Galaxy as a cross-domain RDM platform.

{% include _includes/youtube.html id="2w8okrORVtM"  title="Introduction to Galaxy: Image analysis in Life Sciences, Climate, Earth Science and Astronomy" nofigure="true" %}


## Scope
In this tutorial, we will take you through all the stages of the Research data life cycle and provide a hands-on introduction to the Galaxy platform at each stage.


# The Galaxy Web Interface

Before we go into the stages of the RDM life cycle, let's start with the basics and log into Galaxy and explore the graphical user interface.

## Create an account on a Galaxy instance/server
If you already have an account, skip to the next section!

In Galaxy, *server* and *instance* are often used interchangeably. These terms basically mean that different regions have different Galaxy servers/instances, with slightly different tool installations and appearances. If you don't have a specific server/instance in mind, we recommend registering at one of the main public servers/instances, detailed below.

{% snippet faqs/galaxy/account_create.md %}

Depending on your Galaxy server, you may also be able to log in with your institutional or social account.

{% snippet faqs/galaxy/account_single_sign_on.md %}


## What does Galaxy look like?

> <hands-on-title>Log in to Galaxy</hands-on-title>
> 1. Open your favourite browser (Chrome, Safari, Edge or Firefox as your browser, not Internet Explorer!)
> 2. Browse to your Galaxy instance
> 3. Log in or register
>
>    {% snippet faqs/galaxy/account_create.md %}
>
> ![Screenshot of Galaxy Australia with the register or login button highlighted](../../images/galaxy-login.png)
>
>   > <comment-title>Different Galaxy servers</comment-title>
>   >  This is an image of Galaxy Australia, located at [usegalaxy.org.au](https://usegalaxy.org.au/)
>   >
>   > The particular Galaxy server that you are using may look slightly different and have a different web address.
>   >
>   > You can also find more possible Galaxy servers at the top of this tutorial in **Available on these Galaxies**
>   {: .comment}
{: .hands_on}

The Galaxy homepage is divided into four sections (panels):
- The **Activity Bar** on the left: _This is where you will navigate to the resources in Galaxy (Tools, Workflows, Histories, etc.)_
- Currently active **"Activity Panel"** on the left: _By default, the {% icon tool %} **Tools** activity will be active and its panel will be expanded_
- **Viewing panel** in the middle: _The main area for context for your analysis_
- **History** of analysis and files on the right: _Shows your "current" history; i.e.: Where any new files for your analysis will be stored_

![Screenshot of the Galaxy interface with aforementioned structure](../../images/galaxy_interface.png)

Now that you are logged in to Galaxy, let's start!


# Collect: Data import

![The RDM lifecycle with the collect stage highlighted]({% link topics/introduction/images/galaxy-intro-rdm/rdm-collect.png %}){: style="width:75%"}

The ***collect*** stage in Galaxy usually consists of importing data into what we call your analysis ***history***, this is your Galaxy working environment. Data can be uploaded from from your own machine, from a URL, or imported directly from various general-purpose or domain-specific databases that have been integrated into Galaxy. Before we start our analysis, let's familiarize ourselves with the Galaxy history system.

## The Galaxy History

Your "History" is in the panel at the right. This is where all the files you import or create will be shown. It is also a record of the actions you have taken. Galaxy tracks the provenance of all datasets: which tools, versions, and parameter settings were used to create them. Everything you need to write the methods section of your journal publication.
Before we begin, let's name our history. It is recommended to create a new history for each analysis that you perform, and giving your histories good names will help keep your analyses organized.


### Name your current history

> <hands-on-title>Name history</hands-on-title>
> 1. Go to the **History** panel (on the right)
> 2. Click {% icon galaxy-pencil %} (**Edit**) next to the history name (which by default is "Unnamed history")
>
>    ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value. An input box is below it.]({% link shared/images/rename_history.png %}){:width="250px"}
>
> 3. Type in a new name. Make sure it is a descriptive name.
>    - for example, `Olympics Data Analysis`
> 4. Click **Save**
>
> > <comment-title>Renaming not an option?</comment-title>
> > If renaming does not work, it is possible you aren't logged in, so try logging in to Galaxy first. Anonymous users are only permitted to have one history, and they cannot rename it.
> {: .comment}
>
{: .hands_on}


## Upload a dataset


> <comment-title> Galaxy Data Import Options </comment-title>
>
> There are various ways to get data into Galaxy
> - Uploading from **your computer**
> - Import from **URL**
> - Import directly from **data repositories**, e.g.
>   - SRA/NCBI/EBI/Uniprot (Biological Sequence Data)
>   - OMERO (Image database)
>   - Copernicus (Climate Data)
>   - CERN Open Data (Particle Physics)
>   - Zenodo (general purpose publishing platform)
>   - many more (See "Get Data" section of the Tool panel in Galaxy)
> - **Bring-your-own-data** (e.g. Dropbox, Google Drive, OneData, eLabFTW)
>
>   {% snippet faqs/galaxy/manage_your_repositories.md %}
>
> - Connections to your **LIMS** (Laboratory Information Management System)
>
>   {% snippet faqs/galaxy/importing-data-from-sierra-lims.md %}
>
{: .comment}

For this tutorial, we will import datasets from the general-purpose FAIR data repository [Zenodo](https://zenodo.org).


> <hands-on-title>Upload a file from URL</hands-on-title>
>
> 1. At the top of the **Activity Bar**, click the {% icon galaxy-upload %} **Upload** activity
>
>    ![upload data button shown in the galaxy interface](../../images/upload-data.png)
>
>    This brings up a box:
>
>    ![the complicated galaxy upload dialogue, the 'regular' tab is active with a large textarea to paste subsequent URL](../../images/upload-box.png)
>
> 3. Click **Paste/Fetch data**
> 4. Paste in the address of a file:
>
>    ```
>    https://zenodo.org/records/18803585/files/olympics-2010-winter.tsv
>    ```
>
> 5. Click **Start**
> 6. Click **Close**
>
{: .hands_on}

Your uploaded file is now in your current history.
When the file has been uploaded to Galaxy, it will turn green.

> <comment-title></comment-title>
> After this, you will see your first history item (called a "dataset") in Galaxy's right panel. It will go through
> the grey (preparing/queued) and yellow (running) states to become green (success).
>
{: .comment}

What is this file?

> <hands-on-title>View the dataset content</hands-on-title>
> 1. Click the {% icon galaxy-eye %} (eye) icon next to the dataset name, to look at the file content
>
>    ![galaxy history view showing a single dataset olympics-2010-winter.tsv. Display link is being hovered.]({% link topics/introduction/images/galaxy-intro-rdm/eye-icon.png %}){:width="25%"}
{: .hands_on}

The contents of the file will be displayed in the central Galaxy panel. If the dataset is large, you will see a warning message which explains that only the first megabyte is shown.


This file contains a table listing all athletes who competed in the 2010 Winter Olympics in Oslo.

![galaxy center panel view showing a single dataset olympics-2010-winter.tsv. ]({% link topics/introduction/images/galaxy-intro-rdm/file-preview.png %} "Preview of the dataset in Galaxy. Each row corresponds to an athlete, and each column provides further information about this athlete, including birth year, weight, and medals.")


> <question-title> Explore the dataset </question-title>
>
> 1. How many athletes participated in these Olympics?
> 2. What was the location of these Olympic Games?
>
> > <solution-title></solution-title>
> > 1. 4402 athletes. Each row signifies an athlete; there are 4403 rows, one of which is the header row.
> > 2. Vancouver. This information is given in column 12.
> {: .solution}
{: .question}


## Dataset attributes (metadata)

Let's have a look at the metadata that Galaxy tracks for your datasets.

> <hands-on-title> Explore metadata </hands-on-title>
>
> 1. **Expand** the item in your history by clicking on its name
>    - Here you will see a peek of the contents, some basic file attributes such as the format, number of lines, and number of columns
>
>    ![the history item expanded, showing dataset preview, number of links, and a series of buttons]({% link topics/introduction/images/galaxy-intro-rdm/file-expanded.png %})
>
> 2. **Click** on the **"Dataset Details"** {% icon details %} button
>    - Here you can see further metadata such as file size, creation date, hash, format, original URL, and more
>    - Scrolling down, you will also see details of the upload job that performed the import. We will look more closely at this later.
>
>    ![dataset details]({% link topics/introduction/images/galaxy-intro-rdm/dataset-details.png %} "Screenshot of the dataset details")
>
> 3. **Rename** the file to include the city of the Olympics. You can do this by **editing the dataset attributes**
>    - This can be done by clicking on the **Edit** tab at the top of your screen, or the pencil icon {% icon galaxy-pencil %} on the expanded dataset.
>    - For example, rename it to `2010 Winter Olympics Vancouver`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
>    ![the renamed dataset]({% link topics/introduction/images/galaxy-intro-rdm/dataset-renamed.png %})
>
{: .hands_on}


# Process: Data preparation and QC

![The RDM lifecycle with the process stage highlighted]({% link topics/introduction/images/galaxy-intro-rdm/rdm-process.png %}){: style="width:75%"}


The ***process*** phase of the research life cycle involves preparing your data for analysis. This includes steps such as data cleaning, data transformation, and quality control.
Galaxy offers many tools that can help prepare your data for analysis, such as format conversions and data manipulation tools.

## Use a tool

Recall that our research question in this tutorial is **"What is the age distribution of Olympic athletes?"**
Looking at the dataset, you will see that we do not have an "age" column in our table. We do, however, have a column with the birth year
of each athlete, and a column containing the year of the Olympics. Let's prepare our data for analysis by calculating a new age column
based on these two existing columns.


> <hands-on-title>Find a tool</hands-on-title>
>
> 1. **Search** for the tool {% tool [Compute - on rows](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %}
>    - Click on **Tools** {% icon tool %} in the **Activity bar** on the left
>    - Enter "Compute" in the search bar
>
> 2. **Open** the tool by clicking on it
>    - You will see the tool form in the center panel of Galaxy
>
> 3. **Scroll** down to the **Help** section and read about the tool
>    - Here you will always find **usage information** about the tool, including **citations** and links to **tutorials** describing the tool.
>    - How could we use this tool to add an age column to our dataset?
>
>    ![tool help section]({% link topics/introduction/images/galaxy-intro-rdm/tool-help.png %} "Help section of the Compute tool")
>
{: .hands_on}

We can use this tool to compute an age column for our dataset, but first, we must ask ourselves some questions:

> <question-title> Explore the dataset </question-title>
>
> 1. Which column contains the birth year information
> 2. Which column contains the year of the Olympics?
> 3. How can we compute the age of the athlete from these columns?
>
> > <solution-title></solution-title>
> > 1. column 4 (c4)
> > 2. column 10 (c10)
> > 3. we subtract the columns, `c10-c4`
> {: .solution}
{: .question}


We now have what we need to add an age column to our dataset, let's do it:

> <hands-on-title>Use a tool</hands-on-title>
>
> 1. {% tool [Compute - on a row](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters
>    - {% icon param-file %} *"Input file"*: `2010 Winter Olympics Vancouver`
>    - *"Input has a header line with column names?"*: `Yes`
>    - *"Expressions"*
>      - {% icon plus %} Insert Expressions
>        - *"Add expression"*: `c10-c4`
>        - *"Mode of the operation"*: `Append`
>        - *"The new column name"*: `age`
> 2. **Run** {% icon workflow-run %} the tool
>
{: .hands_on}

This tool will run, and a new output dataset will appear at the top of your history panel.

> <hands-on-title> Check the results </hands-on-title>
>
> 1. **View** {% icon galaxy-eye %} the resulting file
>    - make sure the new column was successfully added, and the column header is "age"
>
> > <question-title> </question-title>
> >
> > 1. What column number is our new "age" column?
> > 2. What age is the first Olympian in our file, *Muhammad Abbas*?
> >
> > > <solution-title></solution-title>
> > > 1. The column was added at the end, column 16.
> > > 2. Age 23.
> > {: .solution}
> {: .question}
>
>
{: .hands_on}


{% snippet faqs/galaxy/tutorial_mode.md %}


## Tool provenance

We already examined the attributes for the file we uploaded. For datasets that result from running tools, Galaxy tracks even more provenance.
Let's look at this now.

> <hands-on-title> Explore metadata </hands-on-title>
>
> 1. **Expand** the item in your history by clicking on its name
> 2. **Click** on the **"Dataset Details"** {% icon details %} button
>
{: .hands_on}

Here you will see all the metadata that Galaxy keeps track of. It has all the same basic information as we saw with the uploaded file.
In addition, it shows which tool produced this output, complete with exact parameter settings and tool version.

![tool parameters of our tool run]({% link topics/introduction/images/galaxy-intro-rdm/tool-provenance.png %} "Parameters of the job (tool run) that produced this dataset")

![job information of our tool run]({% link topics/introduction/images/galaxy-intro-rdm/job-information.png %} "Job information of our tool run. ")



> <question-title> Examine the Job metadata </question-title>
>
> 1. What was the version of the tool that produced your dataset?
> 2. What was the command that was run behind the scenes?
>
> > <solution-title></solution-title>
> > 1. Version 2.1. This can be found under *"Job Information -> Galaxy Tool ID"*, where the last part is the version. E.g. `toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1`. Note that this may be different for you if a newer version has been released since writing this tutorial.
> > 2. The command that is run can be found under *"Job Information -> Command Line"*. It will be something like:
> >    ```
> >    python '/opt/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/column_maker/aff5135563c6/column_maker/column_maker.py'
> >    --column-types int,str,str,int,float,float,str,str,str,int,str,str,str,str,str  --header --file '/data/jwd07/main/097/599/97599988/configs/
> >    tmp1vp1f4gh' --fail-on-non-existent-columns --fail-on-non-computable '/data/dnb12/galaxy_db/files/7/a/6/dataset_7a6bad76-3181-45e2-a460-
> >    31cbe2a6e4a3.dat' '/data/jwd07/main/097/599/97599988/outputs/dataset_5ce07003-fe0c-4836-8f16-b64f25dc9219.dat'
> >    ```
> {: .solution}
{: .question}


## Visualise a dataset


> <hands-on-title> Visualise a dataset </hands-on-title>
>
> 1. Expand the output from **Compute** {% icon tool %}
> 2. Click on the **visualise** {% icon galaxy-visualise %} icon
>
>    ![visualisation button]({% link topics/introduction/images/galaxy-intro-rdm/visualise-button.png %})
>
> 3. Select the **Boxplot** option
>
>    ![visualisation options]({% link topics/introduction/images/galaxy-intro-rdm/visualise-options.png %})
>
> 4. In the "Tracks" tab, change **Column of y-axis values** to `Column: 16` (our new age column)
>
>    ![track configuration]({% link topics/introduction/images/galaxy-intro-rdm/visualise-track.png %})
>
{: .hands_on}

This is a quick way to get a feeling for our data.

![histogram visualisation]({% link topics/introduction/images/galaxy-intro-rdm/visualisation-boxplot-labels.png %} "screenshot of the resulting boxplot. Hovering your mouse over the plot shows you the labels")

> <question-title></question-title>
>
> 1. What age range were our athletes?
>
> > <solution-title></solution-title>
> > 1. Based on the box plot, it looks like our youngest athlete was 15, and our oldest was 51. The mean age was 25.
> {: .solution}
{: .question}


> <tip-title> Save your visualisation </tip-title>
>
> 1. **Name** your visualisation in the "Settings" tab. You can also set the axis labels here
>    - *"Title"*: `Olympics Age Distribution`
>    - *"X-Axis Label"*: `2010 Winter Olympics`
>    - *"Y-Axis Label"*: `Age in years`
>
>    ![plot settings]({% link topics/introduction/images/galaxy-intro-rdm/visualise-settings.png %})
>
> 2. Click on the **Save** {% icon cloud-upload %} icon at the top-right
>
>    ![visualisation save button]({% link topics/introduction/images/galaxy-intro-rdm/viz-save.png %})
>
>    Your visualisation has now been saved to your account. To find it back:
>
> 3. In the **Activity bar**, click on **Visualization** {% icon galaxy-visualise %}
> 4. Click on **Saved Visualizations** at the top of the panel
>
>    ![saved visualisations]({% link topics/introduction/images/galaxy-intro-rdm/visualisations-saved.png %})
>
> 5. Here you will find your saved visualisations
>    - Here you can view, adjust, and rename your previously saved visualisaions
>
>    ![saved visualisations list]({% link topics/introduction/images/galaxy-intro-rdm/visualisations-list.png %})
>
{: .tip}


## Re-run a tool

Our file only contained information for a single Olympics, let's have a look at a second Olympics as well.

We will import another file from Zenodo, but in a slightly different way. Instead of providing the download URL for the dataset, we can also browse Zenodo repositories (and many other data repositories) directly from the Galaxy upload menu.

> <hands-on-title> Upload a second dataset </hands-on-title>
>
> 1. **Option 1:** Choose from Repository
>    - Open the **Upload** window
>    - At the bottom, click on  "Choose from Repository"
>
>      ![choose from repositories button]({% link topics/introduction/images/galaxy-intro-rdm/upload-from-repositories.png %})
>
>    - Search for *"Zenodo"* at the top
>      - If you do not find anything, this is not supported on your Galaxy yet, please skip to option 2 below
>    - Search for the repository with the same name as this tutorial *"Introduction to Galaxy as an RDM platform"*
>
>      ![choose from repositories button]({% link topics/introduction/images/galaxy-intro-rdm/zenodo-search.png %})
>
>    - Check the file `olympics-2008-summer.tsv`
>
>      ![choose from repositories button]({% link topics/introduction/images/galaxy-intro-rdm/zenodo-repo-import.png %})
>
>    - Click **Select**
>    - Click **Start**
>    - Close the upload window
>
> 2. **Option 2:** From URL (same method as before)
>
>    ```
>    https://zenodo.org/records/18803585/files/olympics-2008-summer.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> > <question-title> Examine the file </question-title>
> >
> > 1. Which Olympics is this file for? Which city was it held in?
> >
> > > <solution-title></solution-title>
> > > 1. This file is from the 2008 Summer Olympics in Beijing.
> > >
> > {: .solution}
> {: .question}
{: .hands_on}


Now that we have a second dataset, we want to run the same **Compute** {% icon tool %} tool on this data so that we get an age column.
We could open the tool again and reconfigure all the settings, but there is an easier way to repeat what we did before.

> <hands-on-title> Re-run a tool </hands-on-title>
>
> 1. Click the **Re-run** {% icon dataset-rerun %} button on the output from the **Compute tool**
>    - You will see the tool form with the exact same settings we used before
>    - Because Galaxy tracks all the parameter settings, it is easy to repeat a tool on new data, without having to choose all the parameters again.
>
>    ![rerun button]({% link topics/introduction/images/galaxy-intro-rdm/rerun-tool.png %})
>
> 2. **Change the input dataset** {% icon param-file %} to the summer olympics file we just uploaded
>    - **Run** {% icon workflow-run %} the tool
>
> > <question-title> How did it go? </question-title>
> >
> > 1. What do you see?
> >
> > > <solution-title></solution-title>
> > > 1. If all went well, something went wrong in this step. That is, your dataset turned red instead of green. Not to worry, we will show you how to troubleshoot errors next.
> > >
> > >    ![a red failed dataset in history]({% link topics/introduction/images/galaxy-intro-rdm/dataset-failed.png %})
> > >
> > {: .solution}
> {: .question}
>
>
{: .hands_on}

Oh no! The dataset turned red! This means something went wrong. In the next section, we will show you how you can troubleshoot errors in Galaxy.


## Troubleshooting errors

So something went wrong with one of your tools. This will happen now and then, and can have different causes. It might be something you can fix yourself (e.g. a problem with the input dataset), or it might be something that needs to be fixed in Galaxy (e.g. a bug in the tool). Next, we will see how you can find more information about the error and submit a bug report if you think it might be a problem with the tool.


> <hands-on-title> Troubleshoot a failed tool </hands-on-title>
>
> 1. Click on the **bug icon** {% icon galaxy-bug %} on the failed (red) dataset
>
>    ![the bug icon on a historty item]({% link topics/introduction/images/galaxy-intro-rdm/tool-bug.png %})
>
{: .hands_on}

You will now see details about the error in the center panel:

![Error tab of dataset details]({% link topics/introduction/images/galaxy-intro-rdm/galaxy-bugreport.png %} "the error tab of the dataset details page. This page shows us the error message (stderr) and other tool logs (stdout). It also has a form to submit a bug report at the bottom.")

The error messages can sometimes be a bit cryptic, but the more you use the tools, the easier it will get. If you do not know how to fix the error yourself, you can submit a bug report at the bottom of this page. This will be sent to the administrators of the Galaxy you are using.


> <question-title> Examine the Error message </question-title>
>
> 1. Can you guess what went wrong based on the error message?
> 2. Is this something we can fix? How?
>
> > <solution-title></solution-title>
> > 1. The error message says `could not convert string to float: 'NA'`. This suggests there is a line in the input file that contains an unexpected value (`NA`). This is a common way to denote a missing value, but if we assume this column to be a number and use it in our calculation
> >    things can go wrong.
> >
> >    ```
> >    Failed to convert some of the columns in line #1859 to their expected types.
> >    The error was: "could not convert string to float: 'NA'" for the line:
> >    "19504	Cha Yong-Hwa	F	NA	145	39	North Korea	PRK	2008 Summer	2008
> >     Summer	Beijing	Gymnastics	Gymnastics Women's Individual All-Around	NA"
> >    ```
> >
> >    Apparently, no birth year was known for this athlete from North Korea
> >
> > 2. Yes, since the problem is with our input file, this is something we can fix ourselves.
> >    - One solution could be to remove all lines that contain `NA` in the birth year column.
> >    - Another would be to replace all `NA` values with `nan` (not a number), which is the appropriate way to indicate missing values in numeric columns
> >    - In our case, there is an easier option: we can tell the **Compute** {% icon tool %} tool how to deal with such cases.
> >
> {: .solution}
{: .question}

The error is caused because Galaxy is trying to interpret the birthyear column as a number, but it cannot do this for columns
containing an "NA" (Not Available) value.

So now that we know what caused the error, let's fix it by re-running our tool once more, with different error-handling settings.
We can tell the **Compute** {% icon tool %} tool to stop autodetecting the column type, and instruct it what
to do with "NA" values.

> <hands-on-title> Re-run the tool with error handling parameters </hands-on-title>
>
> 1. **Re-run** {% icon dataset-rerun %} the failed (red) dataset
>    - Expand the **Error Handling** section at the bottom of the tool form
>      - {% icon param-toggle %} *"Autodetect column types"*: `No`
>      - *"If an expression cannot be computed for a row"*: `Skip the row`
>    - Change the *"Expression"* parameter to: `int(c10)-int(c4)`
>      - the `int()` part tells the tool to turn the value into an integer (whole number). Since we told the tool not to autodetect anymore, we need to tell it how to interpret the values in the column.
>
>    ![expression of the tool form]({% link topics/introduction/images/galaxy-intro-rdm/compute-error-handling-expression.png %})
>
>    ![error handling section of the tool form]({% link topics/introduction/images/galaxy-intro-rdm/compute-error-handling.png %})
>
>
> > <question-title> </question-title>
> >
> > 1. What age is the first Olympian in this file, *Ragnhild Margrethe Aamodt*?
> >
> > > <solution-title></solution-title>
> > > 1. Age 27. The age column is the last one.
> > {: .solution}
> {: .question}
{: .hands_on}

If this solution seemed a bit cryptic, don't worry too much; there are always multiple ways to solve the problem. The important thing is that you ran into a problem, looked at the error, and then solved it.

If you get an error message that you don't understand or don't know how to solve, you can always ask for help in one of our **support channels**.

{% snippet faqs/galaxy/support.md %}


### Starring your favourite tools

Since Galaxy has so many tools to choose from, once you find one that is useful for you, you will likely want to use it more often.
To make it easier to find your favourite tools, you can star them.

> <hands-on-title> Star/Favourite a tool </hands-on-title>
>
> 1. **Star** {% icon galaxy-star %} the **Compute** {% icon tool %} tool
>
>    {% snippet faqs/galaxy/tools_favorite.md %}
>
> 2. You can access your favourite tools by clicking on the {% icon galaxy-star %} icon at the top of the **Tool Panel**
>    - this will filter the tool panel for the tools you have starred and your most-used tools
>
{: .hands_on}



## Keeping your history clean

If you have failed items in your history, you might want to delete them. This helps keep your history organized.


> <hands-on-title> Delete failed dataset </hands-on-title>
>
> 1. Click on the **trashcan icon** {% icon galaxy-delete %} on the failed (red) dataset
>
> > <tip-title> Deleted by accident? </tip-title>
> >
> > Did you accidentally delete a dataset you didn't mean to delete? Not to worry, your data is not gone yet.
> > You can show these deleted datasets in your history, and **undelete** them.
> >
> > 1. Click on  **include deleted** {% icon galaxy-delete %} at the top of your history
> >    - so not on the dataset, but at the top of the history panel
> >    - you will see the deleted dataset appear in your history again
> >    - if you expand the deleted dataset, the delete icon has turned into an undelete icon {% icon galaxy-undelete %}
> >
> >    ![history option to include deleted datasets]({% link topics/introduction/images/galaxy-intro-rdm/history-include-deleted.png %})
> >
> {: .tip}
{: .hands_on}

Your dataset is now gone from your history. But deleting it does not remove it completely yet. So if you delete something by accident, you can still view it and undelete it.

You can also delete datasets in bulk:

{% snippet faqs/galaxy/datasets_deleting.md %}

### Storage Quota

Sometimes you really want to permanently delete a dataset, for example, to free up your storage quota. By default, you get **250 GB storage** (exact number may depend on your Galaxy instance), and more can usually be requested on a temporary basis. If you are running out of storage space, you can *purge* (permanently delete) datasets as well. This cannot be undone.

{% snippet faqs/galaxy/datasets_purging_datasets.md %}

{% snippet faqs/galaxy/account_reduce_quota_usage.md %}

We recommend always keeping your history clean and deleting any failed steps.

### Further learning about data preprocessing in Galaxy

Galaxy offers a wide range of basic file manipulation tools that are very helpful for data cleaning.
Operations such as file transformations, filtering, sorting, grouping, joining, splitting, etc., are all possible inside Galaxy.

For more practice with such tools, please see our [Data Manipulation Olympics tutorial]({% link topics/introduction/tutorials/data-manipulation-olympics/tutorial.md %})


## Optional: Use an Interactive Tool

Galaxy also offers various *Interactive Tools*. For example, we could have performed this preprocessing with [OpenRefine](https://openrefine.org/) as well.
Or if we know a bit of programming in R or Python, we could have done these steps using [RStudio](https://posit.co/download/rstudio-desktop/) or [Jupyter Notebooks](https://jupyter.org/).
All of these have been integrated into Galaxy as *Interactive Tools*.

Using these interactive tools is not quite as reproducible as using standard Galaxy tools, but it is great for the exploratory analysis phase of research, especially if you are already familiar with these tools.

In this **optional section**, we will show you how to use such an interactive tool. Here we will use [OpenRefine](https://openrefine.org/), a powerful free, open source tool for working with messy data: cleaning it, transforming it from one format into another. We will use OpenRefine to perform the same task of adding an age column to our dataset.


> <hands-on-title> Launch OpenRefine </hands-on-title>
>
> 1. Click on **Interactive Tools** {% icon galaxy-interactive-tool %} in the **Activity Bar**
> 2. Search for OpenRefine
>
>    ![OpenRefine Interactive tool search result]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-search.png %})
>
> 3. You will see a tool form where you can select files to open
>    - *"Input file in tabular format"*: `2010 Winter Olympics Vancouver`
>    - **Run** {% icon workflow-run %} the tool
>
> 4. Click on **Interactive Tools** {% icon galaxy-interactive-tool %} in the **Activity Bar** again
>    - It may take a little time to start
>
> 5. Once it has started, click on the name to open it
>    - Clicking on the {% icon external-link %} icon will open it in a new tab
>
>    ![Olympics dataset as a table in OpenRefine]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-active.png %})
>
> 6. Click on **Open Project** on the left panel of OpenRefine
>    - Click on **Galaxy File**
>
>    ![Olympics dataset as a table in OpenRefine]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-open.png %})
>
> 5. You will see our Olympics dataset loaded in OpenRefine:
>
>    ![Olympics dataset as a table in OpenRefine]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-table.png %})
>
{: .hands_on}

Next, let's create the same age column in OpenRefine as we did earlier with regular tools.

> <hands-on-title> Edit dataset in OpenRefine </hands-on-title>
>
> 1. First, we tell OpenRefine to interpret the birthyear column as a number
>    - Click on the dropdown {% icon dropdown %} icon next to the birthyear column name
>    - **birthyear {% icon dropdown %} --> Edit Cells --> Common Transforms --> To number**
>
>    ![menu item to convert column to number]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-tonumber.png %})
>
> 2. The values in the column turned green
>
> 3. Click on column **birthyear {% icon dropdown %} --> Edit Column --> Add column based on this column**
>
> 4. Fill in the form
>    - *"New column name"*: `age`
>    - *"Expression"*: `2010-value`
>
>    ![create new column menu]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-create-age-column.png %})
>
> 3. You should now see a new column named "age"
>
>    ![age column added]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-newcolumn.png %})
>
>
{: .hands_on}

Now that we have transformed our dataset as needed, we want to export this table back to our Galaxy history so that we can continue working on it.

> <hands-on-title> Save OpenRefine dataset to Galaxy History </hands-on-title>
>
> 1. Click on the **Export** button in the top-right corner of OpenRefine
> 2. Select **Galaxy Exporter** from the dropdown
>
>    ![button to export to Galaxy]({% link topics/introduction/images/galaxy-intro-rdm/openrefine-exporter.png %})
>
> 3. You will get a message that your dataset has been exported to Galaxy
>    - Check your history and view the exported file
>
> 4. You can now **stop** {% icon stop %} your Interactive tool again
>
>    {% snippet faqs/galaxy/interactive_tools_stop.md tool="OpenRefine" %}
>
{: .hands_on}


Interactive tools can be a powerful addition to your Galaxy analysis.

> <tip-title>Tutorials about RStudio and Jupyter notebooks in Galaxy</tip-title>
>
> If you are interested in using R or Python programming in Galaxy, we recommend
> you have a look at the [Foundations of Data Science]({% link topics/data-science/ %}) topic in
> GTN for comprehensive tutorials.
>
{: .tip}


## Scaling up

Now that we have preprocessed our data, we can continue our analysis, but before we do that, let's explore some more
Galaxy RDM features that can help you manage your research data and analyses.


### Multiple histories

You can have multiple histories in Galaxy to organize your different analyses. We will now start a second history,
and show you how you can switch between histories and move data from one history to another.


> <hands-on-title> Create a second History </hands-on-title>
>
> 1. Create a **new History**
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Name** {% icon galaxy-pencil %} your history
>    - call it `Multi-Olympics Data Analysis`
>
{: .hands_on}

You have now created a new, empty history.

To avoid re-uploading our Olympics dataset and duplicating that data, we can simply copy the files from our previous history

> <hands-on-title> Copy datasets from another history  </hands-on-title>
>
> 1. **View** your histories **side by side**. Instructions are in the tip box below:
>
>    {% snippet faqs/galaxy/histories_side_by_side_view.md %}
>
>    ![multi-history view]({% link topics/introduction/images/galaxy-intro-rdm/multi-history.png %})
>
> 2. **Drag-and-drop** datasets between histories
>    - drag the Winter Olympics file to the new history
>    - do the same for the Summer Olympics file
>
>    ![multi-history view after file copy]({% link topics/introduction/images/galaxy-intro-rdm/multi-history-after.png %})
>
{: .hands_on}

We now have both our datasets in our new history. By doing it this way, rather than re-uploading the files, we do not increase our storage usage.

> <comment-title>The Galaxy History System </comment-title>
>
> Galaxy allows you to have as many histories as you like. Some tips and tricks for using histories:
>
> - Use a different **history for each analysis**
> - Always give your histories a **good name**. This makes it easy to find your histories later.
> - You can easily **switch back and forth between histories** as needed
>
>   {% snippet faqs/galaxy/histories_switch.md %}
>
> - And you can **search** your histories
>
>   {% snippet faqs/galaxy/histories_finding_histories.md %}
>
{: .comment}

We will continue our analysis in this second history and use collections and dataset tags to analyze multiple datasets simultaneously,
and keeping our data organized.


> <warning-title> Proceed with the correct history </warning-title>
>
> **Important!** We will now continue the rest of this tutorial in this new history (named `Multi-Olympics Data Analysis`)
>
> 1. If this is not your active history, please switch to it now (see the box below for instructions)
>
>    {% snippet faqs/galaxy/histories_switch.md %}
>
> 2. Your history should contain two datasets, and look something like this:
>
>    ![screenshot of history to continue with]({% link topics/introduction/images/galaxy-intro-rdm/history-multiolympics.png %})
>
{: .warning}

### Dataset tags

You may have noticed in our first history that the results from the **Compute** {% icon tool %} tool were named *Compute on dataset 1* and *Compute on dataset 3*. To make it a bit clearer for ourselves which dataset was generated from which input file, we can add **dataset tags** {% icon galaxy-tags %}

> <hands-on-title> Add dataset tags </hands-on-title>
>
> 1. Add three **dataset tags** {% icon galaxy-tags %} to the Winter Olympics file
>    - Make sure all tags start with a hashtag (`#`), then they will also be added to any datasets derived from it during analysis.
>    - tag 1: `#winter`
>    - tag 2: `#Vancouver`
>    - tag 3: `#2010`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md %}
>
> 2. Do the same for the Summer Olympics file:
>    - tag 1: `#summer`
>    - tag 2: `#Beijing`
>    - tag 3: `#2008`
>
> 3. Your history should now look something like this:
>    - a history with 2 files, each with dataset tags
>
>    ![datasets with tags added]({% link topics/introduction/images/galaxy-intro-rdm/datasets-tagged.png %})
>
{: .hands_on}


### Dataset collections

In order to easily run analysis on multiple datasets at once, we can create *dataset collections* in Galaxy:

> <hands-on-title> Create a collection </hands-on-title>
>
> 1. **Create a collection** {% icon param-collection %} out of our two Olympic datasets:
>    - Click on **Select Items** {% icon param-check %} (checkbox icon) at top-left of the history panel
>
>      ![select datasets from history]({% link topics/introduction/images/galaxy-intro-rdm/collection-create-select.png %})
>
>    - Check the box in front of both our datasets
>
>      ![checked both our datasets]({% link topics/introduction/images/galaxy-intro-rdm/collection-create-checked.png %})
>
>    - At the top, click on **All 2 Selected** {% icon dropdown%}
>    - Choose the option **Auto Build List**
>
>      ![Auto Build list menu option]({% link topics/introduction/images/galaxy-intro-rdm/collection-create-autobuild.png %})
>
>    - **Name your Collection**
>      - for example "Olympics Dataset"
>
>      ![dataset collection builder]({% link topics/introduction/images/galaxy-intro-rdm/collection.png %})
>
>    - Click **Build**
>
>    {% snippet faqs/galaxy/collections_autobuild_list.md %}
>
> 2. Your history now has a single item in it
>    - It tells you what is inside: *"a list with 2 tabular datasets"*
>
>    ![collection history item]({% link topics/introduction/images/galaxy-intro-rdm/collection-new.png %})
>
> 3. **Click on the collection** to see the files inside it
>
>    ![collection contents]({% link topics/introduction/images/galaxy-intro-rdm/collection-files.png %})
>
> 4. **Return to the regular history view** by clicking the link at the top of the history panel
>    - The link will be called something like "*<< History: Multi-Olympics Data Analysis*"
>
{: .hands_on}


We can now treat this collection the same way as a single dataset. If we use a collection as input to a tool, that tool will be run multiple times, once on each of the datasets inside the collection.
The output of the tool will again be a collection, this time with all the result files.


### Run a tool on a collection

Now that we have set up our inputs as a collection with tags, let's see how to run the **Compute** {% icon tool %} tool on both datasets in the collection at once.

Remember that you starred {% icon galaxy-star %} the compute tool, so you can use that to easily find it again now!

> <hands-on-title> Run a tool on a collection </hands-on-title>
>
> 1. {% tool [Compute - on rows](toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1) %} with the following parameters
>    - {% icon param-collection %} *"Input file"*: `Olympics Dataset` (collection)
>      - In front of this parameter, click on the {% icon param-collection %} icon to switch to collection input
>    - *"Input has a header line with column names?"*: `Yes`
>    - *"Expressions"*
>      - {% icon plus %} Insert Expressions
>        - *"Add expression"*: `int(c10)-int(c4)`
>        - *"Mode of the operation"*: `Append`
>        - *"The new column name"*: `age`
>    - Expand the **Error Handling** section at the bottom of the tool form
>      - {% icon param-toggle %} *"Autodetect column types"*: `No`
>      - *"If an expression cannot be computed for a row"*: `Skip the row`
>
>    {% snippet faqs/galaxy/tools_select_collection.md %}
>
> 2. **View** {% icon galaxy-eye %} the results
>
> > <question-title> What is our output? </question-title>
> >
> > 1. How many outputs were created? Are the files the same as in our first history?
> > 2. What happened with the tags?
> >
> > > <solution-title></solution-title>
> > > 1. One output collection was created, with two files inside. The files themselves are the same as before (a new 'age' column added at the end).
> > > 2. The tags from our input datasets were also added to the results
> > >
> > {: .solution}
> {: .question}
>
{: .hands_on}

Collections allow you to easily run tools on multiple datasets at once. We have 2 datasets in our collection, but you can have as many as
you like, even hundreds or thousands.

Now that we have everything in Galaxy set up for analysis, and our data pre-processed to the right format, we can start to
answer our research question, ***"What is the age distribution of Olympic athletes?"***.


# Analyse: Calculate results


![The RDM lifecycle with the analyse stage highlighted]({% link topics/introduction/images/galaxy-intro-rdm/rdm-analyse.png %}){: style="width:75%"}

The ***Analyse*** phase of the research life cycle is where you extract knowledge from your data in order to answer your research questions.
The details of this phase vary greatly depending on your domain, but Galaxy is a cross-domain platform and has a wide range of analysis
tools available for any scientific domain.


> <comment-title> Domain-specific analysis tools </comment-title>
>
> Because this is an intro tutorial, our "analysis" will be quite basic. But Galaxy offers thousands of tools covering
> a wide range of scientific domains. From life sciences to ecology, climate, astronomy, digital humanities, and many more.
>
> Galaxy has a lot of computational power behind it, so whether you need a simple calculation or a complex algorithm requiring a supercomputer,
> Galaxy can handle it.
>
> If you are interested in a specific domain, have a look at the following resources:
> - [Galaxy Special Interest Groups (SIGs)](https://galaxyproject.org/community/sig/)
> - Galaxy Labs (aka subsites or subdomains)
>
>   {% snippet faqs/galaxy/galaxy_labs.md %}
>
> And the following GTN resources:
> - [GTN Tutorials by Topic](https://training.galaxyproject.org)
> - [GTN Learning Pathways](https://training.galaxyproject.org/learning-pathways)
>
{: .comment}


## Plan our approach

Recall that our research question in this tutorial is ***"What is the age distribution of Olympic athletes?"***

> <question-title> What to do? </question-title>
>
> 1. How would you approach answering our research question?
> 2. Can you find tools in Galaxy that might help you do this?
>
> > <solution-title></solution-title>
> > 1. There are several things we might like to compute in order to answer our question, perhaps
> >    - What is the average age of our Olympians?
> >    - What is the standard deviation?
> >    - What ages are the oldest and youngest Olympians?
> >    - What does the histogram look like for the age distribution?
> >    - Create a boxplot for the age distribution
> >
> > 2. If you already know the name of the tool you want to use, you can simply enter this in the search bar. But often you might not know the name of the tool, then just search for some related keywords
> >
> >    Try searching for terms like:
> >    - statistics, mean, average, minimum, maximum, standard deviation, summary, column, histogram, boxplot
> >
> >    The tool **Summary Statistics - for any numerical column** {% icon tool %} looks interesting!
> >
> >    As does the **Histogram** {% icon tool %}
> >
> {: .solution}
{: .question}

Let's do some analysis based on our plan.

## Get summary statistics for our age column

> <hands-on-title> Summary Statistics </hands-on-title>
>
> 1. {% tool [**Summary Statistics** for any numerical column ](Summary_Statistics1) %} with the following parameters:
>    - {% icon param-collection %} *"Summary statistics on"*: output from **Compute** {% icon tool %}
>      - remember to switch to collection input {% icon param-collection %}
>    - *"Column or expression"*: `c16`
>
> 2. **View** {% icon galaxy-eye %} the results
>    - it should look something like this:
>
>      ```
>      #sum	mean	stdev	0%	25%	50%	75%	100%
>      114999	26.1243	5.01207	15	23	25	29	51
>      ```
>
> > <question-title> </question-title>
> >
> > 1. Which of these two Olympics game had the youngest contestants on average?
> > 2. What was the age of the oldest contestant in each Olympics?
> >
> > > <solution-title></solution-title>
> > > 1. The 2008 Summer Olympics. Compare the *mean* of each output. For the 2010 Winter Games, this was 26.1243, and for the 2008 Summer Games, it was 25.7341
> > > 2. The value of the 100th percentile indicates the highest value encountered. For 2008, this was 67 years, for 2010, it was 51.
> > >
> > {: .solution}
> {: .question}
>
{: .hands_on}

This is great, we know some summary statistics for the age distribution of the Olympics. Let's see if we can also create a visual representation.


## Create a histogram

A picture is worth a 1000 words, so let's see if we can plot the age distribution as well.
We already created a boxplot before; let's try a histogram this time. We will also use a tool rather than a Galaxy visualisation,
so that we get an output file with the plot in our history.

The tool we are going to use for this is **Histogram with ggplot2** {% icon tool %}. This tool will plot every compatible column in the
input dataset. Since we are only interested in the age column, we will extract this column first, and then plot it.

> <hands-on-title> Create a Histogram Plot </hands-on-title>
>
> 1. {% tool [Remove columns - by heading](toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Tabular file"*: output from **Compute** {% icon tool %} (remember to switch to **collection** {% icon param-collection %} input again)
>    - *"Header name"*: `age`
>    - {% icon param-toggle %} *"Keep named columns"*: `Yes`
>    - Click on **Run**
>
> 2. **View** {% icon galaxy-eye %} the outputs
>    - Make sur

tools_json:
["Compute - on rows", "toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.1", "Compute - on a row", "**Summary Statistics** for any numerical column", "Summary_Statistics1", "Remove columns - by heading", "toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/1.0", "Histogram with ggplot2", "toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_histogram/ggplot2_histogram/3.5.1+galaxy1", "**Image Montage** - with ImageMagick", "toolshed.g2.bx.psu.edu/repos/bgruening/imagemagick_image_montage/imagemagick_image_montage/7.1.2-2+galaxy1", "**Unzip** - a file", "toolshed.g2.bx.psu.edu/repos/imgteam/unzip/unzip/6.0+galaxy3", "Export datasets - to repositories", "export_remote"]


topic: introduction
tutorial: galaxy-intro-short
title: A short introduction to Galaxy
difficulty: introductory
time_estimation: 40m
requirements_json: []
tags_json: ["espa\u00f1ol"]
content_hash: 468ccc996a9cf0f54a3c469aa00a0563
last_modified: 2026-05-27T09:01:49.215800
zenodo_link: https://doi.org/10.5281/zenodo.582600
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-short/tutorial.html

content:
# Overview


* This is a short introduction to the Galaxy user interface - the web page that you interact with.
* We will cover key tasks in Galaxy: uploading files, using tools, viewing histories, and running workflows.

> <agenda-title></agenda-title>
> 1. TOC
> {:toc}
>
{: .agenda}

## Create an account on a Galaxy instance/server
If you already have an account, skip to the next section!

In Galaxy, *server* and *instance* are often used interchangeably. These terms basically mean that different regions have different Galaxy servers/instances, with slightly different tool installations and appearances. If you don't have a specific server/instance in mind, we recommend registering at one of the main public servers/instances, detailed below.

{% snippet faqs/galaxy/account_create.md %}

## What does Galaxy look like?

> <hands-on-title>Log in to Galaxy</hands-on-title>
> 1. Open your favorite browser (Chrome, Safari, Edge or Firefox as your browser, not Internet Explorer!)
> 2. Browse to your Galaxy instance
> 3. Log in or register
>
> ![Screenshot of Galaxy Australia with the register or login button highlighted](../../images/galaxy-login.png)
>
>   > <comment-title>Different Galaxy servers</comment-title>
>   >  This is an image of Galaxy Australia, located at [usegalaxy.org.au](https://usegalaxy.org.au/)
>   >
>   > The particular Galaxy server that you are using may look slightly different and have a different web address.
>   >
>   > You can also find more possible Galaxy servers at the top of this tutorial in **Available on these Galaxies**
>   {: .comment}
{: .hands_on}

The Galaxy homepage is divided into four sections (panels):
* The Activity Bar on the left: _This is where you will navigate to the resources in Galaxy (Tools, Workflows, Histories etc.)_
* Currently active "Activity Panel" on the left: _By default, the {% icon tool %} **Tools** activity will be active and its panel will be expanded_
* Viewing panel in the middle: _The main area for context for your analysis_
* History of analysis and files on the right: _Shows your "current" history; i.e.: Where any new files for your analysis will be stored_

![Screenshot of the Galaxy interface with aforementioned structure](../../images/galaxy_interface.png)

The first time you use Galaxy, there will be no files in your history panel.

# Key Galaxy actions

## Name your current history

Your "History" is in the panel at the right. It is a record of the actions you have taken.

> <hands-on-title>Name history</hands-on-title>
> 1. Go to the **History** panel (on the right)
> 2. Click {% icon galaxy-pencil %} (**Edit**) next to the history name (which by default is "Unnamed history")
>
>    ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value. An input box is below it.]({% link shared/images/rename_history.png %}){:width="250px"}
>
>    > <comment-title></comment-title>
>    >
>    > In some previous versions of Galaxy, you will need to click the history name to rename it as shown here:
>    > ![Screenshot of the galaxy interface with the history name being edited, it currently reads "Unnamed history", the default value.](../../../../shared/images/rename_history_old.png){:width="320px"}
>    {: .comment}
>
> 3. Type in a new name, for example, "My Analysis"
> 4. Click **Save**
>
> > <comment-title>Renaming not an option?</comment-title>
> > If renaming does not work, it is possible you aren't logged in, so try logging in to Galaxy first. Anonymous users are only permitted to have one history, and they cannot rename it.
> {: .comment}
>
{: .hands_on}

## Upload a file

The "Activity Bar" can be seen on the left-most part of the interface.

> <hands-on-title>Upload a file from URL</hands-on-title>
> 1. At the top of the **Activity Bar**, click the {% icon galaxy-upload %} **Upload** activity
>
>    ![upload data button shown in the galaxy interface](../../images/upload-data.png)
>
>    This brings up a box:
>
>    ![the complicated galaxy upload dialog, the 'regular' tab is active with a large textarea to paste subsequent URL](../../images/upload-box.png)
>
> 3. Click **Paste/Fetch data**
> 4. Paste in the address of a file:
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    ```
>
> 5. Click **Start**
> 6. Click **Close**
>
{: .hands_on}

Your uploaded file is now in your current history.
When the file has uploaded to Galaxy, it will turn green.

> <comment-title></comment-title>
> After this you will see your first history item (called a "dataset") in Galaxy's right panel. It will go through
> the gray (preparing/queued) and yellow (running) states to become green (success).
>
{: .comment}

> <details-title>Is this step taking a while?</details-title>
>   Sometimes during courses, data upload gets a little slow. You can also import data through a history link.
>
> 1. Import history from: [example input history](https://humancellatlas.usegalaxy.eu/u/wendi.bacon.training/h/short-introduction-to-galaxy--input)
>
>    {% snippet faqs/galaxy/histories_import.md %}
> 2. **Rename** {% icon galaxy-pencil %} the the history to your name of choice.
>
{: .details}

What is this file?

> <hands-on-title>View the dataset content</hands-on-title>
> 1. Click the {% icon galaxy-eye %} (eye) icon next to the dataset name, to look at the file content
>
>    ![galaxy history view showing a single dataset mutant_r1.fastq. Display link is being hovered.](../../images/eye-icon.png){:width="320px"}
{: .hands_on}

The contents of the file will be displayed in the central Galaxy panel. If the dataset is large, you will see a warning message which explains that only the first megabyte is shown.

This file contains DNA sequencing reads from a bacteria, in FASTQ format:

   ![preview of a fastq file showing the 4 line structure described in fig caption. 3 reads are shown.](../../images/fastq.png "A FastQ file has four lines per record: the record identifier (`@mutant-no_snps.gff-24960/`), the sequence (`AATG…`), the plus character (`+`), and then the quality scores for the sequence (`5??A…`)."){:width="620px"}

## Use a tool

Let's look at the quality of the reads in this file.

> <hands-on-title>Use a tool</hands-on-title>
> 1. Type **FastQC** in the tools panel search box (top)
> 2. Click the {% tool [FastQC](toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1) %} tool (it appears as **FastQC Read Quality reports** without the version number)
>
>    The tool will be displayed in the central Galaxy panel.
>
> 3. Select the following parameters:
>    - {% icon param-file %} *"Raw read data from your current history"*: the FASTQ dataset that we uploaded (should be already selected)
>    - No change in the other parameters
> 4. Click **Run Tool**
>
{: .hands_on}

This tool will run and two new output datasets will appear at the top of your history panel.

{% snippet faqs/galaxy/tutorial_mode.md %}

## View results

We will now look at the output dataset called *FastQC on data 1: Webpage*.

> <comment-title></comment-title>
> * Note that Galaxy has given this dataset a name according to both the tool name ("FastQC") and the input ("data 1") that it used.
> * The name "data 1" means the dataset number 1 in Galaxy's current history (our FASTQ file).
>
{: .comment}


> <hands-on-title>View results</hands-on-title>
> * Once it's green, click the {% icon galaxy-eye %} (eye) icon next to the "Webpage" output dataset.
>
>    The information is displayed in the central panel
>
>    ![Graph from fastqc's report. fastqc's images themselves are inaccessible, but this graph shows overall mostly green (good) sequences scores across the length of the read.](../../images/fastqc-out.png){:width="620px"}
{: .hands_on}

This tool has summarised information about all of the reads in our FASTQ file.

> <question-title></question-title>
>
> 1. What was the sequence length of reads in the input FASTQ file?
> 2. Do these reads have higher quality scores in the centre or at the ends?
>
>   > <solution-title></solution-title>
>   > 1. 150 bp
>   > 2. In the center
>   {: .solution}
{: .question}


## Run another tool

Let's run a tool to filter out lower-quality reads from our FASTQ file.


> <hands-on-title>Run another tool</hands-on-title>
> 1. Type **Filter by quality** in the tools panel search box (top)
> 2. Click the tool {% tool [Filter by quality](toolshed.g2.bx.psu.edu/repos/devteam/fastq_quality_filter/cshl_fastq_quality_filter/1.0.2+galaxy2) %}
> 3. Set the following parameters:
>    - {% icon param-file %} *"Input FASTQ file"*: our initial FASTQ dataset
>    - *"Quality cut-off value"*: 35
>    - *"Percent of bases in sequence that must have quality equal to / higher than cut-off value"*: 80
> 4. Click **Run Tool**
{: .hands_on}

After the tool has run, its output dataset will appear at the top of your History panel.
* This dataset will be called "Filter by quality on data 1".
* Remember that Galaxy has named this file according to the tool it used ("Filter by quality") and the input dataset ("data 1").
* The actual numbers in front of the datasets in the history are not important.

What are the results from this filtering tool?

We could click on the eye icon to view the contents of this output file, but it will not be very informative - we will just see a list of reads.

> <hands-on-title>Get metadata about a file</hands-on-title>
> 1. Click the dataset (around where its name is) in the History panel.
>
>    This expands the information about the file. _(By default all datasets are collapsed)_
>
>    ![Diagram of how to locate the information. As above, clicking the name expands the dataset, and an info section is shown with the filter settings. 1786 (14%) discarded.](../../images/filter-fastq1.png)
>
{: .hands_on}

> <question-title></question-title>
>
> How many reads have been discarded?
>
>   > <solution-title></solution-title>
>   > 1786 low-quality reads were discarded
>   {: .solution}
{: .question}

## Re-run that tool with changed settings

We can now try to filter our input reads to an even higher standard, and see how this changes the resulting output (an exploratory analysis). We will change the filter settings and re-run the tool.

> <hands-on-title>Re-run the tool</hands-on-title>
> 1. Click the {% icon galaxy-refresh %} icon (**Run Job Again**) for the output dataset of **Filter by quality** {% icon tool %}
>
>    ![A dataset is expanded showing the Run Job Again button highlighted](../../images/rerun.png)
>
>    This brings up the tool interface in the central panel with the parameters set to the values used previously to generate this dataset.
>
> 2. Change the settings to something even stricter
>
>    For example, you might decide you want 80 percent of bases to have a quality of 36 or higher, instead of 35.
>
> 3. Click **Run Tool**
> 4. View the results: Click the output dataset name to expand the information
>
>    > <comment-title></comment-title>
>    > Take care not to click the {% icon galaxy-eye %} (eye) icon as this will display the contents of the file in the central Galaxy panel instead.
>    {: .comment}
>
{: .hands_on}

> <question-title></question-title>
>
> How many reads were discarded under these new filtering conditions?
>
>   > <solution-title></solution-title>
>   > If you selected 80% of bases with 36 as quality cut-off, then 11517 reads (92%) should have been discarded, which indicates that we have gone too far with the filtering in this case.
>   {: .solution}
{: .question}

You can re-run a tool many times with different settings. Each time you re-run the tool, its new output datasets will appear at the top of your current history.

## Share your history

Let's imagine that you had a problem in your analysis and you want to ask for help. The easiest way to ask for help is to share your history. Try and create a link for your history and share it with...yourself!

{% snippet faqs/galaxy/histories_sharing.md %}


## Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning (at the bottom) to the end (on top). The history in Galaxy records details of every tool you run and preserves all parameter settings applied at each step. But when you need to analyze new data, it would be tedious to do each step one-by-one again. Wouldn't it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the `Extract workflow` option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click {% icon galaxy-history-options %} (**History options**) at the top of your history panel and select **Extract workflow**.
>
>    !['Extract Workflow' entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
>    ![Selection of steps for Extract Workflow from history. All three of fastqc, filter by quality, and the second filter by quality are selected.](../../images/intro_short_workflow_extract.png)
>
> 3. Replace the **Workflow name** to something more descriptive, for example: `QC and filtering`.
>
> 4. **Rename** the workflow input in the box at the top of second column to: `FASTQ reads`
>
> 5. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes. In this case, uncheck the second **Filter by quality** tool at the bottom, where we used a too high quality cut-off.
>
> 6. Click the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created.
>
{: .hands_on}

In a minute we will see how to find the extracted workflow and how to use it.

## Create a new history

Let's create a new history.

> <hands-on-title>New history</hands-on-title>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename your history, *e.g.* "Next Analysis"
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
{: .hands_on}

This new history does not have any datasets in it yet.

## Look at multiple histories

Where is your first history, called "My Analysis"?

> <hands-on-title>View histories in <i>History Multiview</i></hands-on-title>
> One of the ways to view multiple histories at once in Galaxy is through the **History Multiview**
> 1. There are multiple ways to get to the multview:
>    - Click the {% icon galaxy-multihistory %} **History Multiview** activity in the activity bar
>    - Or, click {% icon galaxy-history-options %} (**History options**) and then click the {% icon galaxy-columns %} **Show Histories side-by-side** option
>    ![History options menu dropdown showing you have 163 histories, and a show histories side-by-side button.](../../images/galaxy_interface_history_switch.png){:width="320px"}
>
>    A new page will appear with your histories displayed side-by-side here.
>
> 2. Copy a dataset into your new history
>    1. Click the FASTQ dataset in "My Analysis" history
>    2. Drag it into the "Next Analysis" history
>
>    ![Gif of copying datasets between histories in the side-by-side history view. For now this feature is not keyboard accessible, it is a known issue.](../../images/copy-dataset.gif "Copy a dataset between histories by dragging it")
>
>    This makes a copy of the dataset in the new history (without actually using additional disk space).
>
> 3. Click the Galaxy logo (Home) at the top-left (or **Analyze Data** on older versions of Galaxy) in the top panel to go back to your analysis window
>
{: .hands_on}

Your main Galaxy window will now show "Next Analysis" as the current history, and it will have one dataset in it.

> <comment-title></comment-title>
>   This is not the only way to view your histories in Galaxy:
>
> 1. An exhaustive list (table) of all your histories is available in the **My Histories** tab in the *Histories List* accessible via clicking the {% icon galaxy-histories-activity %} **Histories** activity
> 2. You can quickly switch to another history by clicking the {% icon switch-histories %} **Switch to history** button next to {% icon galaxy-history-options %} **History options**:
>
>    {% snippet faqs/galaxy/histories_switch.md %}
>
> _Here is a tutorial on [Understanding Galaxy history system]( {% link topics/galaxy-interface/tutorials/history/tutorial.md %} )_
{: .comment}

## Run workflow in the new history

Now that we have built our workflow, let's use it to re-create our small analysis in a single step. The same workflow could also be used on some new FASTQ data to quickly repeat the same analysis on different inputs.

> <hands-on-title>Run workflow</hands-on-title>
>
> 1. Click the {% icon galaxy-workflows-activity %} **Workflows** activity in the activity bar.
>
>    Here you have a list of all your workflows _(the **My Workflows** tab is active by default)_.
>
>    Your newly created workflow should be listed at the top:
>
>    ![Workflow list page showing a single workflow named QC and filtering.](../../images/intro_short_workflow_list.png)
>
>    You can see all available actions for the workflow on the workflow card, e.g. edit, copy, rename, share etc. Any other options (e.g.: delete, export etc.) are available by clicking the {% icon galaxy-dropdown %} **Workflow actions** button on the top right of the card.
>
> 2. Click the {% icon workflow-run %} (*Run workflow*) button on the bottom right of the workflow card.
>
>    The central panel will change to allow you to configure and launch the workflow.
>
>    ![Run workflow form with a single input: FASTQ reads. mutant_r1.fastq is selected as the input dataset for that parameter.](../../images/intro_short_run_workflow.png)
>
> 3. Check that the *"FASTQ reads"* input is set to the FASTQ dataset we have copied to the new history.
>
>    In this page we could change any parameter for the tools composing the workflow as we would do when running them one by one.
>
> 4. Click the **Run Workflow** button at the top-right of the screen.
>
>    You should see a message that the workflow was successfully invoked. Then jobs will start to run and datasets appear in your "Next Analysis" history, replicating the steps of your previous history.
>
{: .hands_on}


# Conclusion

Well done! You have completed the short introduction to Galaxy, where you named the history, uploaded a file, used a tool, viewed results and run a workflow. Additional tutorials are available for a more in-depth introduction to Galaxy's features.

tools_json:
["FastQC", "toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.74+galaxy1", "Filter by quality", "toolshed.g2.bx.psu.edu/repos/devteam/fastq_quality_filter/cshl_fastq_quality_filter/1.0.2+galaxy2"]


topic: introduction
tutorial: igv-introduction
title: IGV Introduction
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 79fa2016641c2ec96bcb49f19bb0559a
last_modified: 2026-05-15T22:28:13.730919
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/igv-introduction/tutorial.html

content:
# Using the Integrative Genomics Viewer

**Bioinformatics Galaxy Course**

**MPI-IE & University of Freiburg**

To start IGV, go to the IGV downloads page: https://www.broadinstitute.org/software/igv/download.

## User Interface

## Main Window

![MainWindowIGV](https://www.broadinstitute.org/software/igv/sites/cancerinformatics.org.igv/files/images/igv_desktop_callouts.jpg)


**Tool bar** ![callout_1.jpg](../../images/callout_1.jpg)

- The [tool bar](https://www.broadinstitute.org/software/igv/?q=ToolBar) provides access to commonly used functions. The [menu bar](https://www.broadinstitute.org/software/igv/?q=MenuBar) and [pop-up menus](http://www.broadinstitute.org/software/igv/?q=PopupMenus) (not shown) provide access to all other functions.

**Chromosome** ![callout_2.jpg](../../images/callout_2.jpg)

- The red box on the chromosome ideogram indicates which portion of the chromosome is displayed. When zoomed out to display the full chromosome, the red box disappears from the ideogram.

**Ruler** ![callout_3.jpg](../../images/callout_3.jpg)

- The ruler reflects the visible portion of the chromosome. The tick marks indicate chromosome locations. The span lists the number of bases currently displayed.

**Tracks** ![callout_4.jpg](../../images/callout_4.jpg)

- IGV displays data in horizontal rows called tracks. Typically, each track represents one sample or experiment. This example shows methylation, gene expression, copy number and mutation data.

**Features** ![callout_5.jpg](../../images/callout_5.jpg)

- IGV also displays features, such as genes, in tracks. By default, IGV displays data in one panel and features in another, as shown here. Drag-and-drop a track name to move a track from one panel to another. Combine data and feature panels by selecting that option on the General tab of the [Preferences window](https://www.broadinstitute.org/software/igv/?q=Preferences).

**Track names** ![callout_6.jpg](../../images/callout_6.jpg)

- Track names are listed in the far left panel. Legibility of the names depends on the height of the tracks; i.e., the smaller the track the less legible the name.

**Attributes** ![callout_7.jpg](../../images/callout_7.jpg)

- Attribute names are listed at the top of the attribute panel. Colored blocks represent attribute values, where each unique value is assigned a unique color. Hover over a colored block to see the attribute value. Click an attribute name to sort tracks based on that attribute value.


## Menu Bar

**File**

- Load from File - Displays genomic data from one or more files.

- Load from URL - Displays genomic data from a file identified by URL.

- Load from Server - Displays genomic data from the IGV data server.

- Load from DAS - Displays genomic data from a Distributed Annotation Server (DAS)

- New Session - Unloads all currently loaded data, as if you exited and restarted IGV

- Open Session - Opens a previously saved session file.

- Save Session - Saves your current settings to a named session file

- Save Image - Saves a snapshot of the IGV window to a graphics file, omitting the menu bar and tool bar.  Can save in .png, .jpg, or .svg format.

- Exit - Closes IGV.

**Genomes**

- Load Genome from File - Loads a genome into IGV from your file system.

- Load Genome from URL - Loads a genome into IGV from a web URL.

- Load Genome from Server - Loads a genome into IGV from the IGV data server.

- Create .genome File - Save your genome file in .genome format.

- Manage Genome List - Choose which genomes to display in the genome drop-down menu in the tool bar.

**View**

- Preferences - Opens a tabbed menu of data display preferences.

- Color Legends - Displays color legends for track data, which may be modified.

- Show Name Panel - Shows/hides the track name panel.

- Set Name Panel Width - Resets the track name panel width.

- Show Attribute Display - Shows/hides attributes and attribute values.

- Select Attributes to Show - Shows/hides selected attributes and attribute values.

- Show Header Panel - Shows/hides the chromosome location header panel.

- Reorder Panels - Allows the user to reorder the display panels.

- Go to - View and select loci visited in your navigation history.

**Tracks**

- Sort Tracks - Sorts track data.

- Group Tracks - Groups track data.

- Filter Tracks - Filters track data.

- Fit Data to Window - Sets the track height to display all of the data, or as much data as possible.

- Set Track Height - Sets the track height to a specified value.

**Regions**

- Region Navigator - Opens the region navigator.

- Gene Lists - Opens the gene lists window.

- Export Regions - Saves currently defined regions of interest to a BED file. If no regions of interest are defined, no BED file is created.

- Import Regions - Imports regions of interest from a BED file.

**Tools**

- Run Batch Script - Executes a series of sequential tasks.  Users can load at .txt file that contains a list of commands, one per line, that will be run by IGV.  The accepted commands are the same as the [IGV Port Commands](https://www.broadinstitute.org/software/igv/batch).

- Run igvtools - Launches the igvtools interface window.

- Find Motif - Search for a particular nucleotide sequence in the reference genome.

- Gitools Heatmaps - Data and results are represented as browsable heatmaps. Data can be exported from IGV in gitools format, or loaded directly into a running gitools session. See http://www.gitools.org for details.

- BEDTools - A fast, flexible suite of tools used to compare sets of genomic features. IGV uses BEDTools to compare features in loaded tracks, and view the results as a new track.

**GenomeSpace**

- Load File from GenomeSpace - Load a file into IGV from your GenomeSpace directory.

- Load Genome from GenomeSpace - Load a genome into IGV from your GenomeSpace directory.

- Save Session to GenomeSpace - Save current IGV session to your GenomeSpace directory.

- Load Session from GenomeSpace - Load a previous session from your GenomeSpace directory.

- Logout - Log out of GenomeSpace

- Register - Register a new account at [GenomeSpace](http://www.genomespace.org/)

**Help**

- User Guide... - Displays the IGV [User Guide](https://www.broadinstitute.org/software/igv/UserGuide).



- Help Forum... - In your default web browser, opens the home page for the [igv-help forum](https://groups.google.com/forum/#!forum/igv-help).



- About IGV - Displays IGV version and build number.



## Tool Bar


**Genome drop-down box** ![icon_genome_dropdown](../../images/icon_genome_dropdown.jpg)

- Loads a genome.

**Chromosome drop-down box** ![igv_chromdrop](../../images/igv_chromdrop.jpg)

- Zooms to a chromosome.

**Search box** ![searchbox](../../images/searchbox.jpg)

- Displays the chromosome location being shown.
To scroll to a different location, enter the gene name, locus, or track name and click Go.

**Whole genome view** ![icon_genome](../../images/icon_genome.jpg)

- Zooms to whole genome view.

![icon_backforward](../../images/icon_backforward.jpg)

- Moves backward and forward through views of the genome like the back and forward buttons in a web browser.

**Refresh** ![icon_refresh](../../images/icon_refresh.jpg)

- Refreshes the display.

**Define a region** ![icon_region_of_interest](../../images/icon_region_of_interest.jpg)

- Defines a region of interest on the chromosome.

![icon_squish](../../images/icon_squish.jpg)

- Reduces the row height on all tracks to fit all data for the region in view into the window; will also expand tracks (to their maximum preferred size) to fill the view, if needed.

![icon_popuponoff](../../images/icon_popuponoff.jpg)

- Toggles the pop-up information windows in IGV on or off.

**Zoom slider** ![icon_zoom](../../images/icon_zoom.jpg)

- Zooms in and out on a chromosome.

## Pop-up Menus

To select tracks and display the pop-up menu, do one of the following:

- Right-click a track to select it and display the pop-up menu.

- Right-click an attribute value to select all tracks with that attribute value and display the pop-up menu. Tip: Keep in mind that right-clicking an attribute may select tracks that are not visible in the data panel. Scroll down the data panel to view all the selected tracks.

- Control-click track names (Mac: Command-click) to select the tracks, then right-click one of the selections to display the pop-up menu.

Commands in the track pop-up menu change the display options for the selected tracks. Most changes made via the pop-up menu are lost when you exit IGV unless you [save the session](https://www.broadinstitute.org/software/igv/Sessions). In a few cases, changing the pop-up menu also changes an option in the Preferences window; these changes are persistent.

## Using IGV: Examples

### LOOKING AT ENCODE RNA-SEQ DATA


- Open IGV (.igv.sh from command line)

- Load the human genome :  In the toolbar, Click Genome > Load Genome from Server > Search and select Human (hg19)

- Load the encode RNA-Seq BAM files: Go to File > Load from ENCODE > Select C2C12 RNA-Seq BAM files (See image in the Pdf)

![load_encode_RNA-seq_BAM](../../images/load_encode_RNA-seq_BAM.png)


- Go to your favorite location in the genome by entering the location in the Search box, zoom in to the reads by using Zoom button.

- You can change the track properties by right-click on the track. Select View as pairs to see the left and right pairs of paired-end reads.

![view_pairs](../../images/view_pairs.png)

- You can make a sashimi plot to see reads aligning to each junction. Right-click > Sashimi plot.

![sashimi_plot](../../images/sashimi_plot.png)

- Save what you see by going to File > Save Image.



### LOOKING AT ENCODE ChIP-Seq DATA


- Go to File > Load from ENCODE > Select C2C12 ChIP-Seq bigwig files

![ChIP-Seq_bigwig](../../images/ChIP-Seq_bigwig.png)

- Reset the track heights to make them comparable. Right click on the track > Set data range. Then change the scale.

![data_range](../../images/data_range.png)

- You can also change the track colors, name and chart type (line,bar,heatmap).

- Save the image (File > Save Image)

## POPULAR FILE FORMATS THAT YOU CAN LOAD IN TO IGV


-  BAM

-  BED

-  BedGraph

-  bigBed

-  bigwig

-  FASTA

-  GFF/GTF

-  MAF

-  SAM/BAM

-  VCF

-  PSL

-  Narrowpeak

-  Many more

## SOME OTHER THINGS YOU CAN DO
 - Take multiple images from regions in a bed file. First make a batch file from the bed file using bedtools (bedtoIgv). Then load the script in Tools > Run Batch Script.

- Find a given motif. Tools > Find Motif.

- Set startup preferences. View > preferences.

- For more, have a look at the IGV documentation.





## FOR IGV HELP AND DOCUMENTATION

**GOTO: https://www.broadinstitute.org/igv/ or Click on Help in IGV menu.**


topic: introduction
tutorial: galaxy-cite
title: Best Practices for Citing Galaxy
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["fair", "how-to-cite-galaxy"]
content_hash: 1052319be2a4ed5bdf36d3c775952a71
last_modified: 2026-05-27T09:01:49.212678
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-cite/tutorial.html

content:
# Overview

This is a short primer on how to cite Galaxy in your publication and also how to prepare your Methodology based on your Galaxy history or histories. Following these steps will make it easy to cite Galaxy, share your methods, and adhere to best practices for FAIR and open science.


> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Citing Galaxy
Citing Galaxy comes in two main actions: 

1. Citing the global **Galaxy Project** that provides the core code that all Galaxy instances operate on. 
2. Citing the **instance of Galaxy**, you did your work on.

## Citing Galaxy Project
Please cite Galaxy Project’s primary publication: 

- ***The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update***
  - Nucleic Acids Research, Volume 52, Issue W1, 5 July 2024, Pages W83–W94, [https://doi.org/10.1093/nar/gkae410](https://doi.org/10.1093/nar/gkae410)

This is also found on the Galaxy Project Hub with more specific examples of how to cite individual aspects of Galaxy Project - [https://galaxyproject.org/citing-galaxy/](https://galaxyproject.org/citing-galaxy/)

> <comment-title></comment-title>
> The main Galaxy Project publication will be included in the list if you follow the instructions below to extract tool citations from your History.
>
{: .comment}

## Citing a Galaxy Service

### Service Descriptions
Each public Galaxy service benefits from being able to track the amazing work done by researchers using its service. Provided below are the DOIs for the Galaxy Australia, France and Main `usegalaxy.*` services. The linked papers are available on Zenodo in the Galaxy Project Zenodo Community at [https://zenodo.org/communities/galaxy/](https://zenodo.org/communities/galaxy/):

- Galaxy Australia - [https://doi.org/10.5281/zenodo.17298823](https://doi.org/10.5281/zenodo.17298823)
- Galaxy France - [https://doi.org/10.5281/zenodo.17298858](https://doi.org/10.5281/zenodo.17298858)
- Galaxy Main - [https://doi.org/10.5281/zenodo.17298864](https://doi.org/10.5281/zenodo.17298864)

We recommend including the relevant service name and DOI link in your Methods.


### Acknowledging a Service
Each public Galaxy service is provided through a complex mixture of funding, governance and infrastructure providers. As such, each service provides a cut/paste **Acknowledgement** statement that supports their particular configuration and this can be found on each service. If your publication supports Acknowledgements, we strongly recommend you include the service-recommended **Acknowledgements**.

**Acknowledgement** statements for the four main `usegalaxy.*` services can be found using the following links:

- Galaxy Australia - [https://site.usegalaxy.org.au/about#acknowledgement-statement](https://site.usegalaxy.org.au/about#acknowledgement-statement)
- Galaxy Europe - [https://usegalaxy-eu.github.io/about](https://usegalaxy-eu.github.io/about)
- Galaxy France - [https://www.france-bioinformatique.fr/en/services/acknowledgement/](https://www.france-bioinformatique.fr/en/services/acknowledgement/)
- Galaxy Main - [https://usegalaxy.org/#about](https://usegalaxy.org/#about)

# Writing a Methods section based on your Galaxy work

## Making a History public

We provide the following recommendations and process for making your History persistent and your options for public archiving of your History. Sharing your History will make it easier for others to understand or replicate your analysis. You can also add **annotations** and **tags** to ensure your history aligns to the FAIR principles.

### Preparing a History

We recommend cleaning up your History when preparing to make your History public.

> <hands-on-title>Clean up Unneeded Datasets</hands-on-title>
>
> 1. `Delete` and `Delete (permanently)` (purge) any failed jobs.
> 2. `Delete` and `Delete (permanently)` (purge) any datasets/job outputs that were used for testing and are not required for the final, published analysis.
> 3. `Delete` and `Delete (permanently)` (purge) any datasets that you don't want to share publicly (e.g. restricted data, unpublished results) - but make sure you save them to a private history first.
> 
> {% snippet faqs/galaxy/datasets_deleting.md %}
>
{: .hands_on}

We also suggest providing a description/context of the History in the `History Annotation`.

> <hands-on-title>Annotating a History</hands-on-title>
>
> 1. Add an annotation to your History.
>
> {% snippet faqs/galaxy/histories_annotation.md %}
>
{: .hands_on}

Users can add tags to their Galaxy histories to organise and connect them to specific analyses. Tags improve the findability and traceability of your work, aligning with the FAIR principles. Applying relevant tags before publishing your history is recommended to enhance its clarity and usefulness for both yourself and others.

> <hands-on-title>Adding Tags to a History</hands-on-title>
>
> 1. Add Tags to your History
>
> {% snippet faqs/galaxy/histories_tagging.md %}
>
{: .hands_on}

### Sharing a History

Galaxy provides a number of options for sharing histories which allows others to import and access the datasets, parameters, and steps of your history.

> <hands-on-title>Sharing a History</hands-on-title>
>
> 1. Choose one or several ways to share your History.
>
> {% snippet faqs/galaxy/histories_sharing.md %}
>
{: .hands_on}

### Exporting a History to Zenodo

Integration of InvenioRDM and Zenodo into Galaxy now enables you to export your history directly to Zenodo. Publishing your history on Zenodo will generate a DOI for your history, which you can include in your publication and share with other researchers to make your work easily citable.

For instructions on connecting to Zenodo and exporting your history, see the following tutorial on the Galaxy Training Network:

- [Integrating InvenioRDM-compatible Repositories with Galaxy]({% link topics/fair/tutorials/inveniordm-integration/tutorial.md %})

## Making a Workflow public

We recommend publishing a workflow alongside your data and in your article. In addition to providing a complete description of your method, the published workflow will also make your approach reproducible and citeable.

If you have not created and used a workflow for your analysis, Galaxy can automatically create a workflow based on the analysis you have performed in a history.

> <hands-on-title>Extract a Workflow from a History</hands-on-title>
>
> 1. Extract a Workflow from a Galaxy History
>
> {% snippet faqs/galaxy/workflows_extract_from_history.md %}
>
{: .hands_on}

For instructions on preparing your workflow, see the following tutorial on the Galaxy Training Network:

- [Annotate, prepare tests and publish Galaxy workflows in workflow registries]({% link topics/galaxy-interface/tutorials/workflow-fairification/tutorial.md %})

Two repository options that we recommend for publishing your Workflows are:

- WorkflowHub ([https://workflowhub.eu/](https://workflowhub.eu/))
- Dockstore ([https://dockstore.org/](https://dockstore.org/)) 

## Citing Public Workflows

If you used a workflow that was published on Galaxy, in Galaxy's workflow library, [the IWC](https://iwc.galaxyproject.org/)], or a repository such as WorkflowHub, then we recommend you cite the original workflow. You can cite a workflow as you would any other software. If the information is available, we recommend including the creator’s name, the date of creation, the name of the repository, and the workflow’s DOI or URL.

## Citing Public Data

If you used a public data source in your work, we recommend you cite both the data source publication DOI and the dataset itself. Most citation styles will include options for citing datasets. You should include the repository name and accession code in your Methods section to make it easy to find.

## Extracting Tool Citations from a Galaxy History

Galaxy can all citations of the tools used in your history. You can directly extract them from your history. It is the quickest way to add all these tools to your references, and you might find this helpful when writing up your methods.

> <hands-on-title>Extracting Tool Citations</hands-on-title>
>
> 1. Export a list of tool citations from your History
>
> {% snippet faqs/galaxy/cite-tools-used-in-a-history.md %}
> 
> > <comment-title></comment-title>
> > 
> > You will also get the Galaxy Project publication in all History citation exports!
> > 
> {: .comment}
> 
{: .hands_on}

 
# Conclusion

In this tutorial, we have covered how to cite Galaxy Project and the specific Galaxy server that you used and how to prepare your methodology based on your Galaxy History.


topic: introduction
tutorial: galaxy-intro-ngs-data-managment
title: NGS data logistics
difficulty: introductory
time_estimation: 1H30M
requirements_json: []
tags_json: []
content_hash: e55b6d79d377a5e2e4da911a21aeb208
last_modified: 2026-05-27T09:01:49.213828
zenodo_link: https://zenodo.org/records/15354240
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-ngs-data-managment/tutorial.html

content:
<!--

ALL IMAGES USED IN THIS TUTORIAL ARE AVAILABLE IN EDITABLE FORM HERE:

https://drive.google.com/drive/folders/0B7a9KfygumZFYWxIZlJqSkJwc2c?resourcekey=0-C7W_lazWNzjkKiplb9y01A&usp=sharing

-->

In this tutorial, we will look at practical aspects of manipulation of next-generation sequencing data. We will start with the FASTQ format produced by most sequencing machines and will finish with a table of variants present in four human samples we will analyze.

# The Story

To make this tutorial as realistic as possible we wanted to use an example from the real world. We will start with four sequencing datasets (fastq files) representing four individuals positive for malaria---a life-threatening disease caused by *Plasmodium* parasites---transmitted to humans through the bites of infected female *Anopheles* mosquitoes.

Our goal is to understand whether the malaria parasite ([*Plasmodium falciparum*](https://brc-analytics.dev.clevercanary.com/data/organisms/5833)) infecting these individuals is resistant to [Pyrimethamine](https://en.wikipedia.org/wiki/Pyrimethamine)---an antimalarial drug. Resistance to Pyrimethamine is conferred by a mutation in `PF3D7_0417200` (*dhfr*) gene ({% cite Cowman1988 %}). Given sequencing data from four individuals we will determine which one of them is infected with a *Plasmodium falciparum* carrying mutations in this gene.

An outline of our analysis looks like this:

![analysis_outline](../../../galaxy-interface/images/collections/collection_lifecycle.svg "This analysis begins with a set of fastq files representing four individuals. The sequencing was performed using a paired-end protocol. This means that each sample is represented by two files: forward reads (red) and reverse reads (blue). Before analysis begins these files are combined into a <i>Collection</i>. This allows processing all of them at once. The data is mapped against a reference genome of <i>P. falciparum</i>. This produces a series of BAM files. The BAM files are further passed through a variant caller. This produces VCF files. VCF files are then converted to tabular format and concatenated to create one final dataset. This dataset contains the answer to our question: which of the individual carries drug resistant mutations.").

# From reads to variants

In this tutorial we will use data from four infected individuals sequenced within the [MalariaGen](https://www.malariagen.net/data_package/open-dataset-plasmodium-falciparum-v70/) effort. We use the following four samples:

| Accession | Location |
|------------|------------|
| [ERR636434](https://www.ncbi.nlm.nih.gov/sra/?term=ERR636434) | Ivory coast |
| [ERR636028](https://www.ncbi.nlm.nih.gov/sra/?term=ERR636028) | Ivory coast |
| [ERR042232](https://www.ncbi.nlm.nih.gov/sra/?term=ERR042232) | Colombia |
| [ERR042228](https://www.ncbi.nlm.nih.gov/sra/?term=ERR042228) | Colombia |

These accessions correspond to datasets stored in the [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) at NCBI.

Let's do that 🚀

## Upload data into Galaxy

For this tutorial we down-sampled the data (made datasets smaller) to make sure that you can go through it quickly. We deposited these fastq datasets in a Zenodo library. Let's upload these datasets into Galaxy!

> <details-title>What is fastq data?</details-title>
>
> [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) is not a very well defined format. In the beginning various manufacturers of sequencing instruments were free to interpret FASTQ as they saw fit, resulting in a multitude of FASTQ flavors. This variation stemmed primarily from different ways of encoding quality values as described [on the Wikipedia article for FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) (below you will find an explanation of quality scores and their meaning). Today, the FASTQ Sanger version ({% cite Cock2009 %}) of the format is considered to be the standard form of FASTQ. Galaxy is using FASTQ Sanger as the only legitimate input for downstream processing tools and provides a number of utilities for converting FASTQ files ({% cite Blankenberg2010 %}) into this form (see **FASTQ Quality Control** section of Galaxy tools which is available on some usegalaxy.* instances).
>
> The FASTQ format looks like this:
>
> ```
> @M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
> CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCTGAACCAGCCAAGTAGCGTGCAG
> +
> ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE<,,CD,CEFC,@E9<FCFCF?9
> @M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
> CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACTGATCCAGCCATGCCGCGTGCAG
> +
> ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF<,CEF,CFGGD8FFCFCFEGCF
> @M02286:19:000000000-AA549:1:1101:11116:1322 1:N:0:23
> CCTACGGGAGGCAGCAGTAGGGAATCTTCGGCAATGGACGGAAGTCTGACCGAGCAACGCCGCGTGAGT
> +
> AAC<CCF+@@>CC,C9,F9C9@9-CFFFE@7@:+CC8-C@:7,@EFE,6CF:+8F7EFEEF@EGGGEEE
> ```
>
> Each sequencing read is represented by four lines:
>
> 1. `@` followed by read ID and optional information about sequencing run
> 2. sequenced bases
> 3. `+` (optionally followed by the read ID and some additional info)
> 4. Quality scores for each base of the sequence encoded as [ASCII symbols](https://en.wikipedia.org/wiki/ASCII)
>
> **Paired end data**
>
> It is common to prepare pair-end and mate-pair sequencing libraries. This is highly beneficial for a number of applications discussed in subsequent topics. For now, let's just briefly discuss what these are and how they manifest themselves in FASTQ form.
>
> ![Paired-end and mate-pair reads](../../images/pe_mp.png)
> In paired end sequencing (left) the actual ends of rather short DNA molecules (less than 1kb) are determined, while for mate pair sequencing (right) the ends of long molecules are joined and prepared in special sequencing libraries. In these mate pair protocols, the ends of long, size-selected molecules are connected with an internal adapter sequence (i.e. linker, yellow) in a circularization reaction. The circular molecule is then processed using restriction enzymes or fragmentation. Fragments are enriched for the linker and outer library adapters are added around the two combined molecule ends. The internal adapter can then be used as a second priming site for an additional sequencing reaction in the same orientation or sequencing can be performed from the second adapter, from the reverse strand (From Ph.D. dissertation by <a href="https://ul.qucosa.de/api/qucosa%3A11231/attachment/ATT-0/">Martin Kircher</a>
>
> Thus in both cases (paired-end and mate-pair) a single physical piece of DNA (or RNA in the case of RNA-seq) is sequenced from two ends and so generates two reads. These can be represented as separate files (two FASTQ files with first and second reads) or a single file were reads for each end are interleaved. Here are examples:
>
> *Two single files*
>
> File 1
>
> ```
> @M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
> CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCT
> +
> ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE
> @M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
> CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACT
> +
> ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF
> ```
>
> File 2
>
> ```
> @M02286:19:000000000-AA549:1:1101:12677:1273 2:N:0:23
> CACTACCCGTGTATCTAATCCTGTTTGATACCCGCACCTTCGAGCTTA
> +
> --8A,CCE+,,;,<CC,,<CE@,CFD,,C,CFF+@+@CCEF,,,B+C,
> @M02286:19:000000000-AA549:1:1101:15048:1299 2:N:0:23
> CACTACCGGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATC
> +
> -6AC,EE@::CF7CFF<<FFGGDFFF,@FGGGG?F7FEGGGDEFF>FF
> ```
>
> > <comment-title>Read order is important</comment-title>
> > Note that read IDs are **identical** in two files and they are listed in **the same** order. In some cases read IDs in the first and second file may be appended with `/1` and `/2` tags, respectively (however, this is not guaranteed).
> {: .comment}
>
>
> *Interleaved file*
>
> ```
> @1/1
> AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
> +
> EGGEGGGDFGEEEAEECGDEGGFEEGEFGBEEDDECFEFDD@CDD<ED
> @1/2
> CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
> +
> GHHHDFDFGFGEGFBGEGGEGEGGGHGFGHFHFHHHHHHHEF?EFEFF
> @2/1
> AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
> +
> HHHHHHEGFHEEFEEHEEHHGGEGGGGEFGFGGGGHHHHFBEEEEEFG
> @2/2
> CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
> +
> HHHHHHHHHHHHGHHHHHHGHHHHHHHHHHHFHHHFHHHHHHHHHHH
> ```
>
> In this tutorial paired-end data is represented as two different files.
>
{: .details}

> <hands-on-title>Upload accessions into Galaxy</hands-on-title>
>
> 1. Go to your Galaxy instance of choice such as one of the [UseGalaxy.* instances](https://galaxyproject.org/usegalaxy/) or any other.
> 1. Click *Upload Data* button:
> ![Data upload button](../../images/upload_button.png)
> 1. In the dialog box that would appear click "*Paste/Fetch*" button
> ![Choose local files button](../../images/paste.png)
> 1. Paste the following accessions into the box (red box in the image below):
> ```
> https://zenodo.org/records/15354240/files/ERR042228_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR042228_R.fq.gz
> https://zenodo.org/records/15354240/files/ERR042232_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR042232_R.fq.gz
> https://zenodo.org/records/15354240/files/ERR636028_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR636028_R.fq.gz
> https://zenodo.org/records/15354240/files/ERR636434_F.fq.gz
> https://zenodo.org/records/15354240/files/ERR636434_R.fq.gz
> ```
> 1. Change datatype to `fastqsanger.gz` (green box in the image below)
> ![Name dataset and change datatype](../../images/upload_name_datatype.svg)
> 1. Click *Start* button
> 1. Close dialog by pressing **Close** button
> This will create eight datasets in your history on the right side of the interface:
> ![Four samples and eight files](../../images/f_and_r.svg)
> There are eight datasets because each sample has *forward* and *reverse* read files associated with it, so 4 &#215; 2 = 8
>
{: .hands_on}

## Bundle data into *Collection*

We are going to perform exactly the same analysis on all four samples. So it does not make sense to repeat the same operation eight times (imagine if you had a hundred or a thousand datasets). So before we go any further we will bundle the datasets we have in the history into a *Collection*. *Collections* in Galaxy are logical groupings of datasets that reflect the semantic relationships between them in the experiment / analysis. In this case, we will create a *paired collection* that will have two "levels"  (For more information on Collections see the [Collections tutorial]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}):

![Paired collection](../../images/paired_collection.svg "A paired collection is nested: the first level contains samples and the second contains individual forward and reverse reads associated with each sample")

> <hands-on-title>Creating a paired-collection</hands-on-title>
>
> To create a paired collection follow the steps shown in the video below (the video is 2 minutes long 😁):
>
> <div style="text-align: center;">
> <iframe width="560" height="315" src="https://www.youtube.com/embed/OW4n61BIHAc?si=namdcFYfUITktKRz" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
> </div>
>
> Explore the collections by first **clicking** on the collection name in the history panel. This takes you inside the collection and shows you the datasets in it.  You can then navigate back to the outer level and, finally, full history.
>
{: .hands_on}

<!-- The above embedding uses iframe because include _includes/youtube.html does not work from within hands-on section -->

## Assessing the quality of the data with **fastp** and **multiqc**

Raw fastq data is often contaminated with fragments of sequencing adapters used in library preparation and contain low quality reads or reads with low quality portions. {% tool fastp %} can automatically detect widely used sequencing adapters and remove them as well as prune low quality segments from reads. So, let's pass our data through **fastp**:

> <hands-on-title>Running <b>fastp</b></hands-on-title>
>
> Run {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.24.0+galaxy4) %} with the following parameters:
>    - "*Single-end or paired reads*": `Paired collection` (<font color="red">red outline</font>).
>    - "*Select paired collection(s)*": Collection we created in the previous step (<font color="red">red arrow</font>).
>
> ![fastp interface](../../images/fastp.svg)
>
> Fastp modifies files by removing standard Illumina adapters and applies a number of quality filters generating "Cleaned up data" shown above) as well as HTML and JSON reports as three collections:
>
> ![fastp history items](../../images/fastp_history.svg)
>
{: .hands_on}

You can click on individual HTML reports to get an idea about the quality of the data and degree of "cleanup". However, clicking on each dataset individually can become problematic if the number of datasets is large (you don't want to click on hundred datasets, do you?). We can visualize the QC data provided by **fastp** for **ALL** samples by feeding its JSON output to {% tool multiqc %}:

> <hands-on-title>Running <b>multiqc</b> on <b>fastp</b> JSON data</hands-on-title>
>
> Run {% tool [multiqc](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %} with the following parameters:
>
>    - "*Which tool was used to generate logs?*": `fastp` (<font color="red">red outline</font>).
>    - "*Output of fastp*": JSON output generated by **fastp** at the previous step (<font color="red">red arrow</font>).
>
> ![multiqc interface](../../images/multiqc.svg)
>
> `multiqc` will produce two outputs, but the one you care about has a word "Webpage" in it:
>
> ![multiqc history item](../../images/multiqc_history.svg)
>
> Click on the {% icon galaxy-eye %} (eye) icon and you will see the QC report.
>
{: .hands_on}

Figure below shows one of the plots produced by `multiqc`---distribution of quality values across positions for forward reads after processing.

![distribution of quality values across positions for forward reads after processing](../../images/multiqc_f.png "Quality score distribution after filtering. Here you can see that two samples have very high quality values and 100 bp reads. Two other samples have somewhat lower but still acceptable values and shorter, 80bp, reads.")

> <details-title>What are base qualities? What is good and what is bad?</details-title>
>
> **What are base qualities?**
>
> As we've seen above, FASTQ datasets contain two types of information:
>
> - *sequence of the read*
> - *base qualities* for each nucleotide in the read.
>
> The base qualities allow us to judge how trustworthy each base in a sequencing read is. The following excerpt from an excellent [tutorial](https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf) by Friederike D&uuml;ndar, Luce Skrabanek, Paul Zumbo explains what base qualities are:
>
> > <comment-title>From "Introduction to differential gene expression analysis using RNA-seq"</comment-title>
> > Illumina sequencing is based on identifying the individual nucleotides by the fluorescence signal emitted upon their incorporation into the growing sequencing read. Once the fluorescence intensities are extracted and translated into the four letter code. The deduction of nucleotide sequences from the images acquired during sequencing is commonly referred to as base calling.
> ><br><br>
> > Due to the imperfect nature of the sequencing process and limitations of the optical instruments, base calling will always have inherent uncertainty. This is the reason why FASTQ files store the DNA sequence of each read together with a position-specific quality score that represents the error probability, i.e., how likely it is that an individual base call may be incorrect. The score is called [Phred score](http://www.phrap.com/phred/), $$Q$$, which is proportional to the probability $$p$$ that a base call is incorrect, where $$Q = −10lg(p)$$. For example, a Phred score of 10 corresponds to one error in every ten base calls ($$Q = −10lg(0.1)$$), or 90% accuracy; a Phred score of 20 corresponds to one error in every 100 base calls, or 99% accuracy. A higher Phred score thus reflects higher confidence in the reported base.
> ><br><br>
> > To assign each base a unique score identifier (instead of numbers of varying character length), Phred scores are typically represented as ASCII characters. At http://ascii-code.com/ you can see which characters are assigned to what number.
> ><br><br>
> > For raw reads, the range of scores will depend on the sequencing technology and the base caller used (Illumina, for example, used a tool called Bustard, or, more recently, RTA). Unfortunately, Illumina has been anything but consistent in how they calculated and ASCII-encoded the Phred score (see below)! In addition, Illumina now allows Phred scores for base calls with as high as 45, while 41 used to be the maximum score until the HiSeq X. This may cause issues with downstream applications that expect an upper limit of 41.
> {: .comment}
>
> ![Illumina quality score](../../images/illumina_qs.png)
>
> Base call quality scores are represented with the Phred range. Different Illumina (formerly Solexa) versions
> used different scores and ASCII offsets. Starting with Illumina format 1.8, the score now represents the standard
> Sanger/Phred format that is also used by other sequencing platforms and the sequencing archives.
>
> ![FASTQ quality score](../../images/fastq_qs.png)
>
>The ASCII interpretation and ranges of the different Phred score notations used by Illumina and the original Sanger interpretation. Although the Sanger format allows a theoretical score of 93, raw sequencing reads typically do not exceed a Phred score of 60. In fact, most Illumina-based sequencing will result in maximum scores of 41 to 45 (image from Wikipedia)
>
> **What is good and what is bad?**
>
> One of the first steps in the analysis of NGS data is seeing how good the data actually is. [FastqQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/) is the most famous tool allowing you to assess the quality of FASTQ datasets (and deciding whether to blame or not to blame whoever has done sequencing for you). [Falco](https://github.com/smithlabcode/falco) is an alternative tool for quality control. Falco is a improved implementation of FastQC for high throughput sequence quality control.
>
> ![Good quality in FastQC](../../images/qc_plot.svg)
> Here, you can see **FastQC** base quality reports (the tools gives you many other types of data) for two datasets: <b>A</b> and <b>B</b>. The A dataset has long reads (250 bp) and very good quality profile with no qualities dropping below phred score of 30. The B dataset is significantly worse with ends of the reads dipping below phred score of 20. The B reads may need to be trimmed for further processing. **Falco** will generate almost the same reports about the quality of the reads.")
{: .details}

## Mapping reads

We can proceed with mapping reads. Galaxy has a number of mappers including **bowtie**, **bwa-mem**, and **bwa-mem2**. For this analysis we will use {% tool bwa-mem2 %}---the latest version of this popular and "battle-tested" tool.

### Upload reference genome

The key question when mapping reads against a genome is whether the index for this genome---a datastructure **bwa-mem2** uses to quickly find matches---is already installed on Galaxy or not. Let's assume that it is **NOT** present in Galaxy. In this case you will need to upload the genome. In this case we will use reference genome of 3D7 strain *P. falciparum*.

> <hands-on-title>Uploading the genome for <i>P. falciparum</i></hands-on-title>
>
> To download the genome paste the following URL into the {% tool Upload %} tool:
>
> ```
> https://zenodo.org/records/15354240/files/GCF_000002765.6.fa.gz
> ```
> The only difference is that here with the example we've seen above is that you need to set datatype (green box) to `fasta.gz`:
> ![Genome upload](../../images/upload_genome.svg)
>
{: .hands_on}

### Map the reads

Now we can map the reads against the uploaded genome:

> <hands-on-title>Map sequencing reads with <b>bwa-mem</b></hands-on-title>
>
> Run {% tool [BWA-MEM2](toolshed.g2.bx.psu.edu/repos/iuc/bwa_mem2/bwa_mem2/2.2.1+galaxy4) %} with the parameters shown in the image below:
>
>    - "*Will you select a reference genome from your history or use a built-in index?*": `Use a reference genome from history and build index if necessary` (<font color="red">red outline</font>).
>    - "*Use the following dataset as the reference*": Choose the reference we uploaded at the previous step (<font color="red">red arrow</font>).
>    - "*Single or Paired-end reads*": `Paired collection` (<font color="green">green outline</font>).
>    - "*Select a paired collection*": Select the collection of fastq datasets produced by **fastp** (<font color="green">green arrow</font>).
>
> ![BWA-MEM2 interface](../../images/bwamem2.svg)
>
{: .hands_on}

**bwa-mem2** produces four BAM outputs which we will process further and will use to call variants.

> <details-title>What are BAM files?</details-title>
>
> **SAM/BAM data**
>
> The [SAM/BAM](https://samtools.github.io/hts-specs/SAMv1.pdf) format is an accepted standard for storing aligned reads (it can also store unaligned reads and some mappers such as {% tool [Map with BWA](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.19) %} are accepting unaligned BAM as input). The binary form of the format (BAM) is compact and can be rapidly searched (if indexed). In Galaxy, BAM datasets are always indexed (accompanies by a .bai file) and sorted in coordinate order. In the following discussion, we once again rely on [tutorial](https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf) by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo.
>
> The Sequence Alignment/Map (SAM) format is, in fact, a generic nucleotide alignment format that describes the alignment of sequencing reads (or query sequences) to a reference. The human readable, TAB delimited SAM files can be compressed into the Binary Alignment/Map format. These BAM files are bigger than simply gzipped SAM files, because they have been optimized for fast random access rather than size reduction. Position-sorted BAM files can be indexed so that all reads aligning to a locus can be efficiently retrieved without loading the entire file into memory.
>
> As shown below, SAM files typically contain a short header section and a very long alignment section where each row represents a single read alignment. The following sections will explain the SAM format in a bit more detail. For the most comprehensive and updated information go to [SAM/BAM and related specifications GitHub repository.](https://github.com/samtools/hts-specs)
>
> ![BAM structure](../../images/bam_structure.png)
> Each line of the optional header section starts with <tt>@</tt>, followed by the appropriate abbreviation (e.g., SQ for sequence dictionary which lists all chromosomes names (SN) and their lengths (LN)). The vast majority of lines within a SAM file typically correspond to read alignments where each read is described by the 11 mandatory entries (black font) and a variable number of optional fields (grey font). From <a href="https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf">a tutorial</a> by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo."
>
> **SAM Header**
>
> The header section includes information about how the alignment was generated and stored. All lines in the header section are tab-delimited and begin with the “@” character, followed by tag:value pairs, where tag is a two-letter string that defines the content and the format of value. For example, the “@SQ” line in the header section contains the information about the names and lengths of the **reference** sequences to which the reads were aligned. For a hypothetical organism with three chromosomes of length 1,000 bp, the SAM header should contain the following three lines:
>
> ```
> @SQ SN:chr1 LN:1000
> @SQ SN:chr2 LN:1000
> @SQ SN:chr3 LN:1000
> ```
>
> **SAM alignment section**
>
> The optional header section is followed by the alignment section where each line corresponds to one sequenced read. For each read, there are 11 mandatory fields that always appear in the same order:
>
> ```
> <QNAME> <FLAG> <RNAME> <POS> <MAPQ> <CIGAR> <MRNM> <MPOS> <ISIZE> <SEQ> <QUAL>
> ```
>
> If the corresponding information is unavailable or irrelevant, field values can be ‘0’ or ‘*’ (depending on the field, see below), but they cannot be missing! After the 11 mandatory fields, a variable number of optional fields can be present. Here is an example of one single line of a real-life SAM file (you may need to scroll sideways):
>
> ```
> ERR458493 .552967 16 chrI 140 255 12 M61232N37M2S * 0 0 CCACTCGTTCACCAGGGCCGGCGGGCTGATCACTTTATCGTGCATCTTGGC BB?HHJJIGHHJIGIIJJIJGIJIJJIIIGHBJJJJJJHHHHFFDDDA1+B NH:i:1 HI:i:1 AS:i:41 nM:i:2
> ```
>
> The following table explains the format and content of each field. The `FLAG`, `CIGAR`, and the optional fields (marked with pale blue background) are explained in more detail below. The number of optional fields can vary widely between different SAM files and even between reads within in the same file. The field types marked in blue are explained in more detail in the text below.
>
> ![SAM fields](../../images/sam_fields.png)
>
> **`FLAG` field**
>
> The FLAG field encodes various pieces of information about the individual read, which is particularly important for Paired-End (PE) reads. It contains an integer that is generated from a sequence of bits (0, 1). This way, answers to multiple binary (Yes/No) questions can be compactly stored as a series of bits, where each of the single bits can be addressed and assigned separately.
>
> The following table gives an overview of the different properties that can be encoded in the FLAG field. The developers of the SAM format and samtools tend to use the hexadecimal encoding as a means to refer to the different bits in their documentation. The value of the FLAG field in a given SAM file, however, will always be the decimal representation of the sum of the underlying binary values (as shown in Table below, row 2).
>
> ![SAM flag](../../images/sam_flag.png)
> The <tt>FLAG</tt> field of SAM files stores information about the respective read alignment in one single decimal number. The decimal number is the sum of all the answers to the Yes/No questions associated with each binary bit. The hexadecimal representation is used to refer to the individual bits (questions). A bit is set if the corresponding state is true. For example, if a read is paired, <tt>0x1</tt> will be set, returning the decimal value of 1. Therefore, all <tt>FLAG</tt> values associated with paired reads must be uneven decimal numbers. Conversely, if the <tt>0x1</tt> bit is unset (= read is not paired), no assumptions can be made about <tt>0x2</tt>, <tt>0x8</tt>, <tt>0x20</tt>, <tt>0x40</tt> and <tt>0x80</tt> because they refer to paired reads. From <a href="https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf">a tutorial</a> by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo."
>
> In a run with single reads, the flags you most commonly see are:
>
> - 0: This read has been mapped to the forward strand. (None of the bit-wise flags have been set.)
> - 4: The read is unmapped (`0x4` is set).
> - 16: The read is mapped to the reverse strand (`0x10` is set)
>
> (`0x100`, `0x200` and `0x400` are not used by most aligners/mappers, but could, in principle be set for single reads.) Some common `FLAG` values that you may see in a PE experiment include:
>
>
> |**69** (= 1 + 4 + 64) | The read is paired, is the first read in the pair, and is unmapped.|
> |**77** (= 1 + 4 + 8 + 64) | The read is paired, is the first read in the pair, both are unmapped.|
> |**83** (= 1 + 2 + 16 + 64) | The read is paired, mapped in a proper pair, is the first read in the pair, and it is mapped to the reverse strand.|
> |**99** (= 1 + 2 + 32 + 64) | The read is paired, mapped in a proper pair, is the first read in the pair, and its mate is mapped to the reverse strand.|
> |**133** (= 1 + 4 + 128) | The read is paired, is the second read in the pair, and it is unmapped.|
> |**137** (= 1 + 8 + 128) | The read is paired, is the second read in the pair, and it is mapped while its mate is not.|
> |**141** (= 1 + 4 + 8 + 128) | The read is paired, is the second read in the pair, but both are unmapped.|
> |**147** (= 1 + 2 + 16 + 128) | The read is paired, mapped in a proper pair, is the second read in the pair, and mapped to the reverse strand.|
> |**163** (= 1 + 2 + 32 + 128) | The read is paired, mapped in a proper pair, is the second read in the pair, and its mate is mapped to the reverse strand.|
>
>
> **`CIGAR` string**
>
> `CIGAR` stands for *Concise Idiosyncratic Gapped Alignment Report*. This sixth field of a SAM file
> contains a so-called CIGAR string indicating which operations were necessary to map the read to the reference sequence at that particular locus.
>
> The following operations are defined in CIGAR format (also see figure below):
>
> - **M** - Alignment (can be a sequence match or mismatch!)
> - **I** - Insertion in the read compared to the reference
> - **D** - Deletion in the read compared to the reference
> - **N** - Skipped region from the reference. For mRNA-to-genome alignments, an N operation represents an intron. For other types of alignments, the interpretation of N is not defined.
> - **S** - Soft clipping (clipped sequences are present in read); S may only have H operations between them and the ends of the string
> - **H** - Hard clipping (clipped sequences are NOT present in the alignment record); can only be present as the first and/or last operation
> - **P** - Padding (silent deletion from padded reference)
> - **=** - Sequence match (not widely used)
> - **X** - Sequence mismatch (not widely used)
>
> The sum of lengths of the **M**, **I**, **S**, **=**, **X** operations must equal the length of the read. Here are some examples:
>
> ![CIGAR](../../images/cigar.png)
> From <a href="https://web.archive.org/web/20240422192254/https://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf">a tutorial</a> by Friederike D&uuml;ndar, Luce Skrabanek, and Paul Zumbo.")
>
> **Optional fields**
>
> Following the eleven mandatory SAM file fields, the optional fields are presented as key-value
> pairs in the format of `<TAG>:<TYPE>:<VALUE>`, where `TYPE` is one of:
>
> - `A` - Character
> - `i` - Integer
> - `f` - Float number
> - `Z` - String
> - `H` - Hex string
>
> The information stored in these optional fields will vary widely depending on the mapper and new tags can be added freely. In addition, reads within the same SAM file may have different numbers of optional fields, depending on the program that generated the SAM file. Commonly used optional tags include:
>
> - `AS:i` - Alignment score
> - `BC:Z` - Barcode sequence
> - `HI:i` - Match is i-th hit to the read
> - `NH:i` - Number of reported alignments for the query sequence
> - `NM:i` - Edit distance of the query to the reference
> - `MD:Z` - String that contains the exact positions of mismatches (should complement the CIGAR string)
> - `RG:Z` - Read group (should match the entry after ID if @RG is present in the header.
>
> Thus, for example, we can use the NM:i:0 tag to select only those reads which map perfectly to the reference (i.e., have no mismatches). While the optional fields listed above are fairly standardized, tags that begin with `X`, `Y`, and `Z` are reserved for particularly free usage and will never be part of the official SAM file format specifications. `XS`, for example, is used by TopHat (an RNA-seq analysis tool we will discuss later) to encode the strand information (e.g., `XS:A:+`) while Bowtie2 and BWA use `XS:i:` for reads with multiple alignments to store the alignment score for the next-best-scoring alignment (e.g., `XS:i:30`).
>
> **Read Groups**
>
> One of the key features of SAM/BAM format is the ability to label individual reads with readgroup tags. This allows pooling results of multiple experiments into a single BAM dataset. This significantly simplifies downstream logistics: instead of dealing with multiple datasets one can handle just one. Many downstream analysis tools such as variant callers are designed to recognize readgroup data and output results on per-readgroup basis.
>
> One of the best descriptions of BAM readgroups is on [GATK support site](https://gatkforums.broadinstitute.org):
>
> ![Read groups](../../images/rg.png)
>
> GATK forum also provides the following example:
>
> ![Read group example](../../images/rg_example.png)
>
> **Manipulating SAM/BAM datasets**
>
> We support four major toolsets for processing of SAM/BAM datasets:
>
>  * [DeepTools](https://deeptools.readthedocs.io) - a suite of user-friendly tools for the visualization, quality control and normalization of data from deep-sequencing DNA sequencing experiments.
>  * [SAMtools](http://www.htslib.org/) - various utilities for manipulating alignments in the SAM/BAM format, including sorting, merging, indexing and generating alignments in a per-position format.
>  * [BEDtools](https://bedtools.readthedocs.io/en/latest/) - a toolkit originally written for BED format was expanded for analysis of BAM and VCF datasets.
>  * [Picard](https://broadinstitute.github.io/picard/) - a set of Java tools for manipulating high-throughput sequencing data (HTS) data and formats.
{: .details}

### Remove duplicates

In many cases there could be artifacts that can be detected in mapped reads. One of such artifacts are duplicate reads.

> <details-title>What are read duplicates?</details-title>
>
> **PCR duplicates**
>
> Preparation of sequencing libraries (at least at the time of writing) for technologies such as Illumina (used in this example) involves PCR amplification. It is required to generate sufficient number of sequencing templates so that a reliable detection can be performed by base callers. Yet, PCR has its biases, which are especially profound in cases of multitemplate PCR used for construction of sequencing libraries ({% cite Kanagawa2003 %}).
>
> ![PCR duplicates](../../images/pcr-duplicates.png)
> Analyzing molecules aligning with the same outer coordinates, a mapping quality of at least 30 and a length of at least 30 nucleotide, resulted in an average coverage of 12.9 per PCR duplicate and an empirical coverage distribution similar to an exponential/power law distribution (left upper panel). This indicates that many molecules are only observed for deeper sequencing while other molecules are available at higher frequencies. Analyzing length (left middle panel) and GC content (left lower panel) patterns as well as the combination (right panel) shows higher PCR duplicate counts for a GC content between 30% to 70% as well as for shorter molecules compared to longer molecules. This effect may be due to an amplification bias from the polymerase or the cluster generation process necessary for Illumina sequencing. (From Ph.D. dissertation by <a href=\"https://ul.qucosa.de/api/qucosa%3A11231/attachment/ATT-0/\">Martin Kircher</a>)."
>
> Duplicates can be identified based on their outer alignment coordinates or using sequence-based clustering. One of the common ways for identification of duplicate reads is the `MarkDuplicates` utility from [Picard](https://broadinstitute.github.io/picard/command-line-overview.html) package, which we will use later in this tutorial.
>
> **Sampling coincidence duplicates**
>
> However, one has to be careful when removing duplicates in cases when the sequencing targets are small (e.g., sequencing of bacterial, viral, or organellar genomes as well as amplicons). This is because when sequencing target is small reads will have the same coordinates by chance and not because of PCR amplification issues. The figure below illustrates the fine balance between estimates allele frequency, coverage, and variation in insert size (from ({% cite Zhou2014 %})):
>
> ![Sampling bias](../../images/sampling-bias.png)
>The Variant Allele Frequency (VAF) bias determined by coverage and insert size variance. Reads are paired-end and read length is 76. The insert size distribution is modeled as a Gaussian distribution with mean at 200 and standard deviation shown on the X-axis. The true VAF is 0.05. The darkness at each position indicates the magnitude of the bias in the VAF ({% cite Zhou2014 %}).")
{: .details}

Removing duplicates is particularly important for identification of sequence variants as they can affect their frequencies. It is performed with the {% tool MarkDuplicates %} tool:

> <hands-on-title>Remove duplicates</hands-on-title>
>
> Run {% tool [MarkDuplicates](toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/3.1.1.0)%} with the following parameters:
>    - {% icon param-file %} *"Select SAM/BAM dataset or dataset collection"*: `bam_output` (output of {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.19) %}) as shown with the <font color="red">red arrow</font>.
>    - *"If true do not write duplicates to the output file instead of writing them with appropriate flags set"*: `Yes` as shown with <font color="green">green arrow</font>.
>
> ![Mark Duplicates](../../images/markdups.svg)
{: .hands_on}

## Calling variants

Now we are ready to proceed with variant calling. For this purpose we will use a tool called **lofreq**. One thing to keep in mind in our case is that the samples are from human blood. In humans *Plasmodium* parasites exist in **haploid** state (only zygote is diploid and this stage happens in mosquito). **lofreq** is specifically designed for calling variants in this scenario.

### Realign reads

One of the key issues with accurate identification of sequence variants is normalizing indels (insertions and deletions).

> <details-title>What is the problem with indels?</details-title>
>
>Left aligning of indels (a variant of re-aligning) is extremely important for obtaining accurate variant calls. For illustrating how left-aligning works, we expanded on an example provided by Tan et al. [2015](https://pubmed.ncbi.nlm.nih.gov/25701572/). Suppose you have a reference sequence and a sequencing read:
>
>
>```
>Reference GGGCACACACAGGG
>Read      GGGCACACAGGG
>```
>
>If you look carefully you will see that the read is simply missing a `CA` repeat. But it is not apparent to a mapper, so some of possible alignments and corresponding variant calls include:
>
>```
>Alignment                 Variant Call
>
>GGGCACACACAGGG            Ref: CAC
>GGGCAC--ACAGGG            Alt: C
>
>GGGCACACACAGGG            Ref: ACA
>GGGCA--CACAGGG            Alt: A
>
>GGGCACACACAGGG            Ref: GCA
>GGG--CACACAGGG            Alt: G
>```
>
>While we cannot know which of these arrangements is correct, left-aligning guarantees consistency.
>
{: .details}

{% tool Realign reads %} is a part of **lofreq** package. It corrects misalignment around insertions and deletions:

> <hands-on-title>Realign reads around indels</hands-on-title>
>
> Run {% tool [Realign reads](toolshed.g2.bx.psu.edu/repos/iuc/lofreq_viterbi/lofreq_viterbi/2.1.5+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Reads to realign"*: output of {% tool MarkDuplicates %} as shown with the <font color="red">red arrow</font>.
>    - *"Choose the source for the reference genome"*: `History`
>        - {% icon param-file %} *"Reference"*: set this to the [previously uploaded reference genome](#upload-reference-genome) (<font color="green">green outline and arrow</font>).
>
> ![Realign reads](../../images/realign_lofreq.svg)
>
{: .hands_on}

### Call variants

We are now ready to actually call variants:

> <hands-on-title>Call variants</hands-on-title>
>
> Run {% tool [Call variants](toolshed.g2.bx.psu.edu/repos/iuc/lofreq_call/lofreq_call/2.1.5+galaxy3) %} with the following parameters:
>    - {% icon param-file %} *"Input reads in BAM format"*: Output of {% tool Insert indel qualities %} (<font color="red">red arrow</font>)
>    - *"Choose the source for the reference genome"*: `History` (<font color="green">green outline</font>).
>        - {% icon param-file %} *"Reference"*: set this to the [previously uploaded reference genome](#upload-reference-genome) (<font color="green">green outline and arrow</font>).
>    - *"Types of variants to call"*: `SNVs and indels` (<font color="blue">blue outline and arrow</font>).
>    - *"Variant calling parameters"*: `Configure settings` (<font color="orange">orange outline</font>).
>        - In *"Coverage"*:
>            - *"Minimal coverage"*: `10` (<font color="orange">orange arrow</font>).
>        - In *"Base-calling quality"*:
>            - *"Minimum baseQ"*: `20` (<font color="orange">orange arrow</font>).
>            - *"Minimum baseQ for alternate bases"*: `20`  (<font color="orange">orange arrow</font>).
>        - In *"Mapping quality"*:
>            - *"Minimum mapping quality"*: `20` (<font color="orange">orange arrow</font>).
>
> ![Call variants with lofreq](../../images/lofreq.svg)
{: .hands_on}

The output of this step is a collection of [VCF](https://en.wikipedia.org/wiki/Variant_Call_Format) files containing information on all variants found between the reads and the reference genome.

## Annotating variants

### Preparing {% tool SnpEff %} database

We will now annotate the variants we called in the previous step with the effect they may have on the *Plasmodium* phenotype. In order to do this we need to create a database that can be used by {% tool SnpEff %}. This process requires a reference genome (which we already uploaded and a list of genes present in this genome---a file we have not uploaded yet.

> <hands-on-title>Uploading gene annotations for <i>P. falciparum</i></hands-on-title>
>
> To download gene annotations paste the following URL into the {% tool Upload %} tool as was shown previously in this tutorial already twice (for fastq files and for reference genome).
>
> ```
> https://zenodo.org/records/15354240/files/GCF_000002765.6_GCA_000002765.ncbiRefSeq.gtf.gz
> ```
> Set datatype (<font color="green">green</font> box) to `gtf.gz`:
>
> ![Genome upload](../../images/gtf_upload.svg)
>
{: .hands_on}

Now we can run {% tool SnpEff build %}:

> <hands-on-title>Prepare snpEff database with <b>SnpEff Build</b></hands-on-title>
>
> Run {% tool [SnpEff Build](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff_build_gb/5.2+galaxy0) %} with the following parameters:
>
>    - *"Name for the database"*: `Pf` (<font color="red">red arrow</font>).
>    - *"Input annotations are in"*: `GTF` (<font color="green">green arrow</font>).
>    - *"GTF dataset to build database from"*: Select the GTF file uploaded at the previous step (<font color="blue">blue arrow</font>).
>    - *"Choose the source for the reference genome"*: `History` (<font color="orange">orange outline</font>).
>    - *"Genome in FASTA format"*: Choose reference genome uploaded earlier (<font color="orange">orange arrow</font>).
>
> ![SnpEff Build](../../images/snpeff_build.svg)
>
{: .hands_on}

### Annotate variant effects with **SnpEff eff**

We are now ready to annotate variants with {% tool SnpEff eff %} (here "eff" stands for effect):

> <hands-on-title>Annotate variant effects</hands-on-title>
>
> Run {% tool [SnpEff eff](toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/5.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Sequence changes (SNPs, MNPs, InDels)"*: Output of {% tool lofreq %} we've run previously <font color="red">red arrow</font>)
>    - *"Genome source"*: `Custom snpEff database in your history` <font color="green">green outline</font>)
>      - "*SnpEff5.2 Genome Data*": set to the SnpEff database we built in the previous step <font color="green">green arrow</font>)
>
>![snpEff eff](../../images/snpeff.svg)
>
{: .hands_on}

### Create table of variants using **SnpSift Extract Fields**

Previous step generated a set of annotated VCF files. These can be visualized in a genome browser, but we will go a different way. We will convert them into tab-delimited files that can be easily processed.

> <hands-on-title>Create table of variants</hands-on-title>
>
> Run {% tool [SnpSift Extract Fields](toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_extractFields/4.3+t.galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Variant input file in VCF format"*: Output of {% tool SnpEff eff %} from previous step (<font color="red">red arrow</font>)
>    - *"Fields to extract"*: Copy the following string into a box highlighted with (<font color="green">green outline</font>):
>
> ```
>CHROM POS REF ALT QUAL DP AF SB DP4 ANN[*].EFFECT ANN[*].IMPACT ANN[*].GENE ANN[*].AA_POS ANN[*].HGVS_C ANN[*].HGVS_P
> ```
>
>    - *"One effect per line"*: `Yes` (<font color="blue">blue arrow</font>)
>
> ![snpsift extract fields](../../images/snpsift_extract_fields.svg)
>
{: .hands_on}

Let's explain which fields we are extracting:

| # | Field | Meaning |
|---|----------|-------------|
| 1 | `CHROM` | Name of the chromosome |
| 2 | `POS` | Position within that chromosome |
| 3 | `REF` | Reference allele (the one present in the *reference* |
| 4 | `ALT` | Alternative allele (the one observed in the reads) |
| 5 | `QUAL` | Variant quality computed by **lowfreq** |
| 6 | `DP` | Read depth (the number of reads covering this position) |
| 7 | `AF` | Alternative allele frequency |
| 8 | `SB` | Strand bias P-value from Fisher's exact test calculated by lofreq |
| 9 | `DP4` | Depth for Forward Ref Counts, Reverse Ref Counts, Forward Alt Counts, Reverse Alt Counts |
| 10 | `ANN[*].EFFECT` | Expected effect of the variant on the protein |
| 11 | `ANN[*].IMPACT` | Expected impact of the phenotype |
| 12 | `ANN[*].GENE` | Name of the gene containing the variant |
| 13 | `ANN[*].AA_POS `| Amino acid position within the gene affected by this variant |
| 14 | `ANN[*].HGVS_C` | Nucleotide change in nucleotide coordinates relative to the start of the coding region |
| 15 | `ANN[*].HGVS_P` | Amino acid change in amino acid coordinates relative to the start of the protein |

### Collapse data into a single dataset

We now extracted meaningful fields from VCF datasets. But they still exist as a collection. To move towards secondary analysis we need to **collapse** this collection into a single dataset. "Collapsing" simply concatenates the content of collection elements and attaches sample IDs, so that we know which line in the concatenated file corresponds to which sample:

![Collapsing a collection](../../images/collapse.svg "Collapsing combines a collection into a single file dataset and includes names of collection elements as a new column. Here each line in the collapsed dataset (right) is pre-pended with the element's name such as 1, 2, 3, and 4.")

For more information about collapsing collections, please watch the following YouTube video:

{% include _includes/youtube.html id="ypuFZ1RKMIY" title="Dataset Collections: Collapsing a collection" %}

> <hands-on-title>Collapse a collection</hands-on-title>
>
> Run {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: Output of {% tool SnpSift Extract Fields %} from the previous step (<font color="red">red arrow</font>).
>    - "*Keep one header line*": `Yes` (<font color="green">green arrow</font>)
>    - "*Prepend File name*": `Yes` (<font color="blue">blue arrow</font>)
>    - "*Where to add dataset name*": `Same line and each line in dataset` (<font color="orange">orange outline</font>)
>
> ![Collapse tool UI](../../images/collapse_ui.svg)
>
{: .hands_on}

You can see that this tool takes lines from all collection elements (in this tutorial we have two), add element name as the first column, and pastes everything together. So, if we have a collection as an input:

> <code-in-title>A collection with two items (<b>15</b> columns)</code-in-title>
> A collection element named `ERR042228.fq`
>
>```
>NC_004318.2   676399   C    T     1344.0   45   0.977778   0   0,0,23,22   missense_variant    MODERATE PF3D7_0415200   914     c.2740G>A    p.Glu914Lys
>NC_004318.2   676631   C    T     2000.0   60   0.966667   0   0,0,32,27   synonymous_variant  LOW      PF3D7_0415200   836     c.2508G>A    p.Lys836Lys
>```
>
>A collection element named `ERR042232.fq`:
>
>```
>NC_004318.2   671152   G    A     1324.0   42   0.952381   0   0,1,20,21   synonymous_variant  LOW      PF3D7_0415100   90      c.270G>A   p.Gln90Gln
>NC_004318.2   671641   A    T     1300.0   43   0.906977   0   0,0,16,27   missense_variant    MODERATE PF3D7_0415100   253     c.759A>T   p.Glu253Asp
>```
{: .code-in}

We will have a single dataset as the output:

> <code-out-title>A single dataset (<b>16</b> columns)</code-out-title>
>
>then the {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} will produce this:
>
>```
>ERR042228.fq NC_004318.2   676399   C    T     1344.0   45   0.977778   0   0,0,23,22   missense_variant    MODERATE PF3D7_0415200   914     c.2740G>A    p.Glu914Lys
>ERR042228.fq NC_004318.2   676631   C    T     2000.0   60   0.966667   0   0,0,32,27   synonymous_variant  LOW      PF3D7_0415200   836     c.2508G>A    p.Lys836Lys
>ERR04223

tools_json:
["fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.24.0+galaxy4", "multiqc", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3", "BWA-MEM2", "toolshed.g2.bx.psu.edu/repos/iuc/bwa_mem2/bwa_mem2/2.2.1+galaxy4", "Map with BWA", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa/0.7.19", "MarkDuplicates", "toolshed.g2.bx.psu.edu/repos/devteam/picard/picard_MarkDuplicates/3.1.1.0", "Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.19", "Realign reads", "toolshed.g2.bx.psu.edu/repos/iuc/lofreq_viterbi/lofreq_viterbi/2.1.5+galaxy0", "Call variants", "toolshed.g2.bx.psu.edu/repos/iuc/lofreq_call/lofreq_call/2.1.5+galaxy3", "SnpEff Build", "toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff_build_gb/5.2+galaxy0", "SnpEff eff", "toolshed.g2.bx.psu.edu/repos/iuc/snpeff/snpEff/5.2+galaxy0", "SnpSift Extract Fields", "toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_extractFields/4.3+t.galaxy0", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "Filter", "Filter1"]


topic: introduction
tutorial: galaxy-intro-101
title: Galaxy Basics for genomics
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: []
content_hash: b8d5afc9aa8a3b64f4284752c0d98331
last_modified: 2026-05-27T09:01:49.213001
zenodo_link: https://doi.org/10.5281/zenodo.4104428
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/galaxy-intro-101/tutorial.html

content:
This tutorial aims to familiarize you with the Galaxy user interface. It will teach you how to perform basic tasks such as importing data, running tools, working with histories, creating workflows, and sharing your work.

{% snippet faqs/galaxy/analysis_results_may_vary.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Setting the stage: Exons and SNPs

We start with the question: In human chromosome 22, which exon has the highest number of single nucleotide polymorphisms (SNPs)?

> <comment-title>Background</comment-title>
> Not everyone has the same background and that's ok! You may have studied different organisms with different gene models, or you may not be familiar with the biological aspect at all. The biological background is not necessary for following this tutorial, we just like to provide research context for any tutorial. Here are brief explanations of some of the concepts mentioned in this tutorial:
>
> - **Nucleotides** are the A, C, T, and Gs that make up DNA.
> - **Chromosomes** can be thought of as a very long piece of DNA (string of A, C, T, Gs) Some organisms have chromosomes, in this tutorial we will use Human chromosome number 22.
> - **Features** are regions of the chromosome that are interesting for one reason or another. Some examples of features include genes, terminators, transcription start sites, and repeat regions.
> - **Genes** are one kind of interesting feature, a region that will be [transcribed](https://en.wikipedia.org/wiki/Transcription_(biology)) into RNA before being [translated](https://en.wikipedia.org/wiki/Translation_(biology)) into proteins.
> - **Exons** are fundamental components of [eukaryotic](https://en.wikipedia.org/wiki/Eukaryote) [genes](https://en.wikipedia.org/wiki/Gene#Structure_and_function). A typical eukaryotic gene contains numerous *exons* separated by *introns*. An entire gene containing both exons and introns is *transcribed* into a pre-messenger RNA or pre-mRNA. During maturation introns are excised from the pre-mRNA in a process called *[splicing](https://www.nature.com/scitable/topicpage/rna-splicing-introns-exons-and-spliceosome-12375/)*.  A mature messenger RNA, or simply mRNA, is then *translated* into protein during the process of *translation*.
>
>
>   ![Eukaryotic gene model](../../images/DNA_exons_introns.png "An original piece of DNA containing introns and exons has the introns cut out before the exons are joined together to form the mRNA. (Image from <a href="https://commons.wikimedia.org/wiki/File:DNA_exons_introns.gif">WikiMedia</a>, under the Public Domain)")
>
> **SNP** is an abbreviation for [single-nucleotide polymorphism](https://en.wikipedia.org/wiki/Single-nucleotide_polymorphism). It is pronounced as "snip". SNPs are single nucleotide differences between a sequenced individual compared to some reference sequence; where one individual might have an A, another could have a C in that position. Databases of SNPs have been created for many organisms and they include any single nucleotide deviation from the reference sequence which appears in some percentage of the population (e.g. >1%). These are often especially interesting to geneticists as the causes of certain inheritable diseases.
{: .comment}

You may be familiar with the [UCSC Genome Browser](https://genome.ucsc.edu/) or another resource like it, and know that you can find the data there.
But even with your data in hand, you still have the question: "how do I actually compute this?" There is really a straightforward answer: **Galaxy**. So let's try it...

# Get your workspace ready

Browse to your favorite [Galaxy instance](https://galaxyproject.org/use/) and log in or register.

{% snippet faqs/galaxy/account_create.md %}

The Galaxy interface consists of three main parts:

1. The available **Tools** are listed on the left
2. The **Central Panel** will let you run analyses and view outputs
3. Your analysis **History** is recorded on the right


![Galaxy interface screenshot showing History panel on the right, tools panel on the left, and main panel in the center](../../images/galaxy_interface.png "<b>Galactic triptych</b>: the three panels of Galaxy interface: <em>Tools</em>, <em>Center Panel</em>, and <em>History</em>. Here you also see the 'Activity bar' that can be turned on and off (Don't see the Activity Bar? See a Tip below).")
{: .comment}


{% snippet faqs/galaxy/activity_bar_how_to_enable.md %}


When you start Galaxy for very first time, your history will be empty. Let's add some data to it.

> <hands-on-title>Create history</hands-on-title>
>
> 1. Make sure you start from an empty analysis history.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename your history** to be meaningful and easy to find. You can do this by clicking on the title of the history (which by default is *Unnamed history*) and typing **Galaxy 101** as the name. Do  not forget to hit the <kbd>Enter</kbd> key on your keyboard to save it.
>
>    ![Rename the history](../../../../shared/images/rename_history.svg)
>
{: .hands_on}

# Analysis

## Locate Exons

First we need to get some data into our history. You can upload files from your computer, or Galaxy can also fetch data directly from external sources. We know UCSC has exon locations for humans and we can use Galaxy to import the data for chromosome 22, directly from the UCSC table browser.

> <hands-on-title>Upload SNPs and Exons</hands-on-title>
> 1. At the top of the **Tools** panel (on the left), click {% icon galaxy-upload %} **Upload Data**
>
>    ![upload data button](../../images/upload-data.png)
>
>    This brings up the upload interface:
>
>    ![filebox](../../images/upload-box.png)
>
> 2. Click **Paste/Fetch data** and paste in the following URLs in the box that appears.
>
>    ```
>    https://zenodo.org/record/4104428/files/UCSC-hg38-chr22-Coding-Exons.bed
>    https://zenodo.org/record/4104428/files/UCSC-hg38-chr22-dbSNP153-Whole-Gene-SNPs.bed
>    ````
>
> 3. Click **Start**, and then **Close**
>
> 4. When they are ready, **rename** {% icon galaxy-pencil %} the datasets to `Exons` and `SNPs`, respectively.
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
{: .hands_on}

For this tutorial, we made the input datasets available on Zenodo for you. However, these datasets can also obtained directly from UCSC, without leaving Galaxy.
Below we describe how you can do this, but it is not necessary for this tutorial. Note that since the data in UCSC is updated frequently, you might get slightly different results in the rest of this tutorial if you use these files.

> <comment-title></comment-title>
> In order to get the datasets from the UCSC server, you need to have an account in an instance.
{: .comment}


> <tip-title>Optional: Obtaining the Data directly from UCSC</tip-title>
>
> This tool works a bit differently than most Galaxy tools, but if you wish to obtain the newest data from UCSC, you can do that as follows:
>
> > <hands-on-title>Obtaining Exons from UCSC</hands-on-title>
> >
> > 1. **UCSC Main** {% icon tool %} table browser:
> >
> >     In the tool menu, navigate to `Get Data -> UCSC Main table browser`
> >
> >     ![`UCSC Main table browser` menu entry](../../images/101_01.png)
> >
> >     When you click on this tool, you will be taken to the **UCSC table browser**, which looks something like this:
> >
> >     ![`UCSC table browser` tool, first screen for exons](../../images/101_02.png)
> >
> >     Now set the following parameters:
> >     - *"clade"*: `Mammal`
> >     - *"genome"*: `Human`
> >     - *"assembly"*: `Dec. 2013 (GRCh38/hg38)`
> >     - *"group"*: `Genes and Gene Predictions`
> >     - *"track"*: `GENCODE v36` (or a more recent version)
> >     - *"table"*: `knownGeneV36`
> >     - {% icon param-text %} *"region"* should be changed to `position` with value `chr22`
> >     - *"output format"* should be changed to `BED - browser extensible data`
> >     - {% icon param-check %} *"Send output to"* should have the option `Galaxy` checked
> >
> >     > <comment-title></comment-title>
> >     > If the *"table"* drop down menu does not show the `knownGene` option. Set *"group"* to `All tables` and scroll down.
> >     {: .comment}
> >
> > 2. Click on the **Get output** button and you will see the next screen:
> >
> >    ![`UCSC table browser` tool, second screen for exons](../../images/101_03.png)
> >
> >    Change **Create one BED record per** to `Coding Exons` and then click on the **Send query to Galaxy** button.
> >
> >     > <comment-title></comment-title>
> >     > After this you will see your first history item in Galaxy's right panel. It will go through
> >     > the gray (preparing/queued) and yellow (running) states to become green (success):
> >     >
> >     > ![`UCSC Main on Human: knownGene` dataset is green](../../images/101_04.png)
> >     >
> >     > You might need to login to Galaxy again.
> >     {: .comment}
> >
> > 3. When the dataset is green, click on the {% icon galaxy-eye %} (eye) icon to **view the contents** of the file. It should look something like this:
> >
> >    ![Contents of the `UCSC Main on Human: knownGene` dataset](../../images/101_exons.png)
> >
> >    Each line represents an exon, the first three columns are the genomic location, and the fourth column contains the name of the exon.
> >
> > 4. Let's rename our dataset to something more recognizable.
> >    - Click on the {% icon galaxy-pencil %} **pencil icon** for the dataset to edit its attributes
> >    - In the central panel, change the **Name** field to `Exons`
> >    - Click the **Save** button
> >
> >    Your history should now look something like this:
> >
> >    ![Rename dataset to `Exons`](../../images/101_rename.png)
> {: .hands_on}
>
> We now have information about the exon locations, but our original question was which exon contains the largest number of SNPs, so let's get some information about SNP locations from UCSC as well:
>
> > <hands-on-title>Obtaining SNPs from UCSC</hands-on-title>
> > Again, search for **UCSC Main table browser** Browser on the tools search bar {% icon tool %} and set the following parameters
> >
> > 1. **UCSC Main** {% icon tool %} table browser:
> >    - *"group"* should be changed to `Variation`
> >    - {% icon param-text %} *"region"* should be changed again to `position` with value `chr22`
> >    - *"output format"* should be changed again to `BED - browser extensible data`
> >
> >    ![`UCSC table browser` tool, first screen for SNPs](../../images/101_06.png)
> >
> >    The *"track"* setting shows the version of the SNP database to get. In this example it is version 151, but you may select the latest one. Your results may vary slightly from the ones in this tutorial when you select a different version, but in general it is a good idea to select the latest version, as this will contain the most up-to-date SNP information.
> >
> > 2. Click on the **Get output** button to find a form similar to this:
> >
> >    ![`UCSC table browser` tool, second screen for SNPs](../../images/101_07.png)
> >
> >    Make sure that *"Create one BED record per"* is set to `Whole Gene` (Whole Gene here really means Whole Feature), and click on **Send query to Galaxy**. A second item will appear in your analysis history.
> >
> > 3. Now **rename** {% icon galaxy-pencil %} your new dataset to `SNPs` so we can easily remember what the file contains.
> >
> {: .hands_on}
>
{: .tip}

## Find exons with the most SNPs

Our objective is to find which exon contains the most SNPs. Therefore we have to intersect the file with the exon locations with the file containing the SNP locations (here "intersect" is just a fancy word for printing SNPs and exons that overlap side-by-side).

> <comment-title>Search bar</comment-title>
>
> Different Galaxy servers may have tools available under different sections, therefore it is often useful to use the **search bar** at the top of the tool panel to find your tool.
>
> Additionally different servers may have multiple, similarly named tools which accomplish similar functions. For these tutorials, you should select precisely the one that is described. However, in your real analyses, you'll need to search among the various options to find the one that works for you.
>
{: .comment}

> <hands-on-title>Finding Exons</hands-on-title>
>
> To find intersection we will be using `intersect intervals` {% tool [bedtools intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} from [BEDTools](https://bedtools.readthedocs.io/en/latest/content/overview.html) package.
>
> 1. {% tool [bedtools intersect intervals](toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0) %} find overlapping the intervals in various ways:
>
>    If you opened the tutorial on any usegalaxy.* instance, you can click on the tool name and will open the correct tool with correct version. Or, you can search for this tool yourself. Enter the word `intersect` in the search bar of the tool panel, and select the
>    tool named `bedtools Intersect intervals`
>
>    - *"File A to intersect with B"*: Select `Exons`
>    - *"Combined or separate output files"*: Select `One output file per 'input B' file"`
>    - *"File B to intersect with A"*: `SNPs`
>    - *"What should be written to the output file?"*: `Write the original entry in B for each overlap...`, which means that only matches are included in the output (i.e. only exons with SNPs in it and only SNPs that fall in exons)
>
>    The interface of the tool should look like this:
>    ![Bedtool intersect interval interface](../../images/101_bed_intersect_interface.png)
>
> 2. Click **Run Tool**.
> 3. Wait for the job to finish.
> 4. View the resulting file (with the {% icon galaxy-eye %} (eye) icon). If everything went okay, you should see a file that looks similar to this:
>
>    ![Contents of the `Join` output dataset](../../images/101_joined.png)
>
>    Here column 4 contains exon IDs (e.g., `ENST00000252835.5_cds_0_0_chr22_15528159_f`) and column 10 lists SNP IDs (e.g., `rs9617249`). Remember that variations are possible due to using different versions of UCSC databases: as long as you have similar looking columns you did everything right!
>    > <tip-title>Where do I find tool documentation?</tip-title>
>    > All Galaxy tools include documentation. If you scroll down on this page, you will find the help of the tool.
>    {: .tip}
>
>    > <comment-title>If things didn't work...</comment-title>
>    >
>    > Did the Intersect tool error with a memory failure? Or is this step executing for a long time? Most likely a setting was missed when extracting the data from the UCSC Table Browser. Try again, double checking that:
>    >
>    >  * For both SNP and EXON: *"region"* is actually changed to `position` with value `chr22`.
>    >  * For EXON: *"Create one BED record per"* `Coding Exons` is selected (*not* `Whole Gene` as for the SNP data).
>    >  * Carefully inspect the remaining Table Browser settings if these two most common reasons for problems were correct in your tool executions.
>    {: .comment}
>
{: .hands_on}

Let's take a look at this dataset. The first six columns correspond to the exons, and the last six columns correspond to the SNPs. Column 4 contains the exon IDs, and column 10 contains the SNP IDs. In our screenshot you see that the first lines in the file all have the same exon ID but different SNP IDs, meaning these lines represent different SNPs that all overlap the same exon.

> <question-title></question-title>
> For the first 3 exons in your file, what is the number of SNPs that fall into that exon?
>
> > <solution-title></solution-title>
> > At the time of writing, for hg38/GENCODE v29, joined with "Common dbSNPs(153)", using <kbd>ctrl+f</kbd> (<kbd>cmd+f</kbd> on Mac OS) to look for how many times each is used:
> >
> > Gene | Occurences
> > ---- | ----------
> > `ENST00000252835.5_cds_0_0_chr22_15528159_f` | 2
> > `ENST00000643195.1_cds_0_0_chr22_15528192_f` | 2
> > `ENST00000343518.11_cds_0_0_chr22_15690078_f` | 4
> {: .solution}
{: .question}


## Count the number of SNPs per exon

Since each line in our file represents a single overlap between SNP and exon, we can find the total number of SNPs in an exon, simply by counting the number of lines that have the same exon ID. However, to be more "proper" we will instead count the number of unique SNP IDs per exon. So let's do this for all the exons in our file:

> <hands-on-title>Counting SNPs</hands-on-title>
>
> 1. {% tool [Datamash](toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.8+galaxy0) %} (operations on tabular data):
>
>    - *"Input tabular dataset"*: select the output dataset from **bedtools intersect intervals** {% icon tool %}
>    - *"Group by fields"*: `Column: 4` (the column with the exon IDs)
>
>    ![Datamash tool interface](../../images/101_datamash_1.png)
>
>    - Scroll tool interface down to *"Operation to perform on each group"*
>      - *"Type"*: `Count Unique values`
>      - *"On column"*: `Column: 10` (this column contains SNPs ids like `rs2236639`. This we will count occurrences of unique SNP ids for each exon)
>
>    ![Operation to perform on each group](../../images/101_datamash_2.png)
>
> 2. Click **Run Tool**. Your new output dataset will look something like this:
>
>    ![Contents of the `Group` output dataset](../../images/101_14.png)
>
{: .hands_on}

This file contains only two columns. The first contains the exon IDs, and the second the number of times that exon ID appeared in the file - in other words, how many SNPs were present in that exon.

> <question-title></question-title>
> How many exons are there in total in your file?
>
> > <solution-title></solution-title>
> > Each line now represents a different exon, so you can see the answer to this when you expand the history item, as in the image above. The exact number you see for your dataset may be slightly different due to the updates to the exon and SNPs information in UCSC. In our case the dataset contains 4,241 lines, which is equal to the number of exons overlapped by at least one SNP.
> >
> {: .solution }
{: .question}

## Sort the exons by SNPs count

Now that we have a list of all exons, and the number of SNPs they contain, we would like to know which exon has the *highest number* of SNPs. We can do this by sorting the file on the second column.

> <hands-on-title>Sorting</hands-on-title>
>
> 1. {% tool [Sort](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1) %} data in ascending or descending order:
>
>    - *"Sort Query"*: Output from **Datamash** {% icon tool %}
>    - In *"Column selections"* set the following:
>       - *"on column"*: `Column: 2`
>       - *"in"*: `Descending order`
>       - *"Flavor"*: `Fast numeric sort`
>
> 2. Click **Run Tool**
>
> 3. Examine the output file.
>
>    ![Contents of the `Sort` output dataset](../../images/101_16.png)
>
>    The file should look similar to before sorting, but now the exons with the highest number of SNPs are at the top.
{: .hands_on}


> <question-title></question-title>
> Which exon has the highest number of SNPs in your file?
>
> > <solution-title></solution-title>
> > When this tutorial was last updated, `ENST00000253255.7_cds_0_0_chr22_46256561_r` had 27 SNPs.
> > Keep in mind this may depend on your settings when getting the data from UCSC.
> {: .solution}
{: .question}

## Select top five exons

Let's say we want a list with just the top-5 exons with highest number of SNPs.

> <hands-on-title>Select first</hands-on-title>
>
> 1. {% tool [Select first](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/9.5+galaxy0) %} lines from a dataset (head):
>
>    - *"File to select"*: The output from **Sort** {% icon tool %} (previous step of the analysis)
>    - "*Operation*": `Keep first lines`
>    - *"Number of lines"*: `5`
>
> 2. Click **Run Tool**.
>
> 3. Examine the output file, this should contain only the first 5 lines of the previous dataset.
>
{: .hands_on}

## Recover exon info

Congratulations! You have now determined which exons on chromosome 22 have the highest number of SNPs, but what else can we learn about them? One way to learn more about a genetic location is to view it in a genome browser. However, in the process of getting our answer, we have lost information about the location of these exons on the chromosome. But fear not, Galaxy saves all of your data, so we can recover this information quite easily.

> <hands-on-title>Compare two Datasets</hands-on-title>
>
> 1. {% tool [Compare two Datasets](comp1) %} to find common or distinct rows:
>
>    - *"Compare"*: `Exons`
>    - *"Using column"*: `Column: 4`
>    - *"against"*: the output from **Select first** {% icon tool %} (previous step of the analysis)
>    - *"and column"*: `Column: 1`
>    - *"to find"*: `Matching rows of 1st dataset`
>
> 2. Click **Run Tool**
> 3. Examine your output file. It should contain the locations of your top 5 exons:
>
>    ![Contents of the `Compare two Datasets` output dataset](../../images/101_19.png)
{: .hands_on}

## Display data in UCSC genome browser

A good way to learn about these exons is to look at their genomic surrounding. This can be done by using genome browsers. Galaxy can launch a genome browser such as IGV on your local machine, and it can connect to online genome browsers as well. An example of such an online genome browser is the UCSC genome browser.

> <hands-on-title>UCSC genome browser</hands-on-title>
>
> 1. First, check that the **database** of your latest history dataset is `hg38`. If not, click on the {% icon galaxy-pencil %} pencil icon and modify the **Database/Build:** field to `Human Dec. 2013 (GRCh38/hg38) (hg38)`.
>
>    {% snippet faqs/galaxy/datasets_change_dbkey.md dbkey="Human Dec. 2013 (GRCh38/hg38) (hg38)" %}
>
> 2. Second, check that the **format** of your latest history dataset is `bed`. If not, click on the {% icon galaxy-pencil %} pencil icon and modify the **Datatype** field to `bed`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="bed" %}
>
> 3. Click on the dataset in your history to expand it, then click on the {% icon galaxy-barchart %} (**Visualize**) icon
>
>    ![visualize button on the expanded history item](../../images/101_ucsc_visualize.png)
>
> 4. To **visualize the data in UCSC genome browser**, click on `display at UCSC (main)` now visible at the top (blue box) of the Visualize menu.
>
>    ![`display at UCSC main` link](../../images/101_displayucsc.png)
>
>    This will upload the data to UCSC as custom track. To see your data look at the `User Track` near the top. You can enter the coordinates of one of your exons at the top to jump to that location.
>
>    ![`User Track` shown in the UCSC genome browser](../../images/101_21.png)
{: .hands_on}

UCSC provides a large number of tracks that can help you get a sense of your genomic area, it contains common SNPs, repeats, genes, and much more (scroll down to find all possible tracks).

# Histories and workflows: A brief introduction

In Galaxy your analyses live in histories such as your current one. Histories can be very large, and you can have as many histories as you want. You can control your histories (switching, copying, sharing, creating a fresh history, etc.) in the {% icon galaxy-history-options %} **History Options** menu on the top of the history panel:

![History options menu](../../images/history_options_menu.png "'History options' allows for a variety of history operations")

## List your histories

You can create as many histories as you want. If you create a new history, your current history does not disappear. You can view your histories in two ways: (1) as a list or (2) side by side (also known as "Multiview"). Two Tip boxes below explain both of these approaches:

{% snippet faqs/galaxy/histories_list.md %}

{% snippet faqs/galaxy/histories_side_by_side_view.md %}


![Overview of histories with their datasets](../../images/histories_side_by_side_view.png "Histories side-by-side view: in this view you drag datasets between histories, switch between histories, create new histories etc.")

You can always return to your analysis view by clicking on Galaxy Icon in the top menu bar.

## Convert your analysis history into a workflow

When you look carefully at your history, you can see that it contains all the steps of our analysis, from the beginning to the end. By building this history we have actually built a complete record of our analysis with Galaxy preserving all parameter settings applied at every step. But when you receive new data, or a new report is requested, it would be tedious to do each step over again. Wouldn't it be nice to just convert this history into a workflow that we will be able to execute again and again?

Galaxy makes this very easy with the `Extract workflow` option. This means any time you want to build a workflow, you can just perform the steps once manually, and then convert it to a workflow, so that next time it will be a lot less work to do the same analysis.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. **Clean up** your history: remove any failed (red) jobs from your history by clicking on the {% icon galaxy-delete %} button.
>
>    This will make the creation of the workflow easier.
>
> 2. Click on "**History options**" dropdown at the top of your history panel and select **Extract workflow**.
>
>    ![`Extract Workflow` entry in the history options menu](../../images/history_menu_extract_workflow.png)
>
>    The central panel will show the content of the history in reverse order (oldest on top), and you will be able to choose which steps to include in the workflow.
>
>    ![Selection of steps for `Extract Workflow` from history](../../images/101_25.png)
>
> 3. Replace the **Workflow name** to something more descriptive, for example `Find exons with the highest number of features`.
>
>    While we created this workflow initially to analyse SNPs, if we had similarly formatted datasets we could use this workflow to find different features.
>
> 4. If there are any steps that shouldn't be included in the workflow, you can **uncheck** them in the first column of boxes.
>
> 5. Click on the **Create Workflow** button near the top.
>
>    You will get a message that the workflow was created. But where did it go?
>
> 6. Click on **Workflows** in the left menu of Galaxy (Activity Bar). Here you have a list of all your workflows. Your newly created workflow should be listed at the top:
>
>    ![`Your workflows` list](../../images/workflow_cards.png "Workflow are listed in the center pane of the Galaxy interface as cards.")
{: .hands_on}

## The workflow editor

We can examine the workflow in Galaxy's workflow editor. Here you can view/change the parameter settings of each step, add and remove tools, and connect an output from one tool to the input of another, all in an easy and graphical manner. You can also use this editor to build workflows from scratch.

> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. Click on the triangle to the right of your workflow name.
>
>    ![`Edit` option in the menu for the latest workflow](../../images/open_wf_editor.png "To open the interactive workflow editor click the 'Edit' button.")
>
> 2. Select **Edit** to launch the workflow editor. You should see something like this:
>
>    ![Workflow editor](../../images/wf_editor_open.png "Workflow editor interface. It can be used for creation and editing of workflows of any complexity.")
>
>    When you click on a workflow step, you will get a view of all the parameter settings for that tool on the right-hand side of your screen.
>
> 3. Re-arrange the boxes so you can clearly see the data flow. The default automatic layout hides some of the connections due to overlapping and box placement.
>
> 4. Make sure the  **check boxes** for `out_file1` in the `Select First` and `Compare two Datasets` tools are selected. Make sure that everything else is not selected.
>
>    Now, when we run the workflow, we will only see the final two outputs, i.e. the table with the top-5 exons and their SNP counts, and the file with exons ready for viewing in a genome browser.
>
> 5. The box named `Exons` is named ok, but we want to change `SNPs` since this workflow is not specific to SNPs
>
>    - **Click** on the box corresponding to the `SNPs` input dataset
>    - change the **Label** to `Features` on the right-hand side of your screen.
>
> 6. Let's also **rename the outputs**:
>
>    - Click on the `Select first` tool in the workflow editor
>    - In the menu on the right click on `Configure Output: 'out_file1'`
>        - Under `Rename dataset`, and enter a descriptive name for the output dataset like `Top 5 exon IDs`
>
>    ![Rename the output of the `Select first` tool step](../../images/configure_wf_output.png "The datasets generated by the workflow execution can be automatically renamed using  the 'Rename dataset' option")
>
> 7. **Repeat** this for the output of the `Compare two Datasets` tool, naming it `Top 5 exons`
>
> 8. **Save your workflow** (important!) by clicking on the {% icon galaxy-save %} icon at the top right of the screen.
>
> 9. **Return** to the analysis view by clicking on the Galaxy Icon at the top left of the menu bar.
>
{: .hands_on}

> <comment-title></comment-title>
> We could **validate** our newly built workflow by running it on the same input datasets that we used at the start of this tutorial, in order to make sure we do obtain the same results.
{: .comment}

## Run workflow on different data

Now that we have built our workflow, let's use it on some different data. For example, let's find out which exons have the highest number of repeat elements.

> <hands-on-title>Run workflow</hands-on-title>
>
> 1. Create a **new history** and give it a name.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. We will need the list of exons again. We don't have to get this from UCSC again, we can just **copy** it from our previous history. To do this:
>
>    - Bring up a list of all you histories:
>
>    {% snippet faqs/galaxy/histories_list.md %}
>
>    - Display the "Galaxy 101" history in the Center pane by clicking "Open in Center pane" button:
>    ![Opening a history in the center pane](../../images/open_in_center_pane_button.png)
>
>    - Drag and drop `Exons` dataset into the empty history created at step 1
>    ![Drag and drop of `Exons` dataset in the history overview](../../images/drag_n_drop_dataset.gif)
> 3. Click on Galaxy at the left to return to the main analysis window
>
> 4. {% tool [Upload Data](upload1) %} the Repeats file from [Zenodo](https://doi.org/10.5281/zenodo.4104428)
>
>    ```
>    https://zenodo.org/record/4104428/files/UCSC-hg38-chr22-Repeats.bed
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    > <tip-title>Obtaining the Data from UCSC</tip-title>
>    >
>    > Again, for reproducibility we obtain the data from Zenodo ensuring that the results will never change, allowing us to do good science! However if you wish to obtain UCSC data:
>    >
>    > > <hands-on-title>Obtaining Exons from UCSC</hands-on-title>
>    > >
>    > > 1. **UCSC Main** {% icon tool %} table browser:
>    > >
>    > >     In the tool menu, navigate to `Get Data -> UCSC Main - table browser`
>    > >
>    > >     Now set the following parameters:
>    > >     - *"clade"*: `Mammal`
>    > >     - *"genome"*: `Human`
>    > >     - *"assembly"*: `Dec. 2013 (GRCh38/hg38)`
>    > >     - *"group"*: `Repeats`
>    > >     - {% icon param-text %} *"region"* should be changed to `position` with value `chr22`
>    > >     - *"output format"* should be changed to `BED - browser extensible data`
>    > >     - {% icon param-check %} *"Send output to"* should have the option `Galaxy` checked
>    > >
>    > >   Click on **get output** and then **Send query to Galaxy** on the next screen.
>    > {: .hands_on}
>    {: .tip}
>
> 5. **Rename** {% icon galaxy-pencil %} the dataset to `Repeats`
>
> 6. Open the **workflow menu** (top menu bar). Find the workflow you made in the previous section, and select the option `Run`.
>
>    ![`Run` option in the workflow menu](../../images/run_workflow_button.png)
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
>    The central panel will change to allow you to configure and launch the workflow.
>
> 7. Select appropriate datasets for the inputs as shown below, then scroll down and click `Run workflow`.
>    - Select `Expand to full workflow form`
>    - {% icon param-file %} *"Exons"*: the `Exons` file you copied from our previous history
>    - {% icon param-file %} *"Features"*: the `Repeats` file we downloaded from UCSC
>
>    ![Settings for running the workflow](../../images/workflow_setting_inputs.png)
>
>    > <comment-title>Potential workflow issues</comment-title>
>    >
>    >  * Galaxy validates the workflow inputs to ensure they're correct. It may show a validation error at the start, until you select `Exons` for the Exons input, and your repeats for the Features input.
>    >  * If you see an "Invalid column choice" error, you need to specify which column you want to use. If you have to type the column number, you need to type just the number e.g. `4` (not `Column 4` or anything else).
>    {: .comment}
>
> 8. Once the workflow has started, you will initially be able to see all its steps, but the unimportant intermediates may disappear after they complete successfully (based on the workflow settings):
>
>    ![Datasets appearing in the history](../../images/workflow_running.png)
{: .hands_on}

> <comment-title>Unhiding hidden datasets</comment-title>
> Because most intermediate steps of the workflow could be hidden, once it is finished you may only see the final two datasets. If we want to view the intermediate files after all, you can click the "## hidden" just below the history's name.
{: .comment}

> <question-title></question-title>
> Which exon had the highest number of repeats? How many repeats were there?
{: .question}

# Share your work

One of the most important features of Galaxy comes at the end of an analysis. When you have published striking findings, it is important that other researchers are able to reproduce your in-silico experiment. Galaxy enables users to easily share their workflows and histories with others.

{% snippet faqs/galaxy/histories_sharing.md %}

> <hands-on-title>Share history and workflow</hands-on-title>
>
> 1. Share one of your histories with your neighbour.
> 2. See if you can do the same with your workflow!
> 3. Find the history and/or workflow shared by your neighbour. Histories shared with specific users can be accessed by those users in their {% icon galaxy-gear %} history menu under `Histories shared with me`.
{: .hands_on}

# Conclusion


{% icon trophy %} Well done! You have just performed your first analysis in Galaxy. You also created a workflow from your analysis so you can easily repeat the exact same analysis on other datasets. Additionally you shared your results and methods with others.

tools_json:
["bedtools intersect intervals", "toolshed.g2.bx.psu.edu/repos/iuc/bedtools/bedtools_intersectbed/2.31.1+galaxy0", "Datamash", "toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.8+galaxy0", "Sort", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1", "Select first", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_head_tool/9.5+galaxy0", "Compare two Datasets", "comp1", "Upload Data", "upload1"]


