# tutorials

topic: contributing
tutorial: github-interface-contribution
title: Contributing with GitHub via its interface
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 7ef36a1fe946f2196badb9c08282a272
last_modified: 2026-02-26T21:12:14.709501
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/github-interface-contribution/tutorial.html

content:
All the training material which you find on [{{ site.url }}{{ site.baseurl }}/]({{ site.baseurl }}/) is stored on a [GitHub](https://github.com) repository ([{{ site.github_repository }}]({{ site.github_repository }})), a code hosting platform for version control and collaboration. GitHub interface is quite intuitive and simplifies the contributions from anyone.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to use GitHub interface to contribute to the training material:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# GitHub

The GitHub repository for the training material is: [{{ site.github_repository }}]({{ site.github_repository }}).

> <hands-on-title>Familiarization with GitHub</hands-on-title>
>
> 1. Go on the GitHub repository: [{{ site.github_repository }}]({{ site.github_repository }})
>
>    ![GitHub interface](../../images/github_interface.png "Interface of the GitHub repository of the training material")
>
> 2. Click on `CONTRIBUTORS.yaml` file
>
>    ![Click on CONTRIBUTOR](../../images/github_contributor.png)
>
> 3. View the file
>
>    You should see something like:
>
>    ![Click on CONTRIBUTOR](../../images/github_contributor_interface.png "CONTRIBUTOR file")
>
{: .hands_on}

This `CONTRIBUTORS.yaml` file is where we collect the information (name, email, etc) about the different contributors to display them on our [Hall of Fame]({% link hall-of-fame.md %}). You will add your information there. But first you need to sign in to GitHub to be able to change this file.

> <hands-on-title>Sign in to GitHub</hands-on-title>
>
> 1. Create a GitHub account (if you do not have one already)
>
>    ![GitHub signup](../../images/github_sign_up.png)
>
> 2. Sign in (once you have a GitHub account)
>
{: .hands_on}

# Edit a file

You can now modify the `CONTRIBUTORS.yaml` file to add your information there

> <hands-on-title>Edit a file</hands-on-title>
>
> 1. Open the `CONTRIBUTORS.yaml` file on GitHub
> 2. Click on {% icon hands_on %} icon (top right of the file)
>
>    ![GitHub edit](../../images/github_edit.png)
>
>    A new page will open:
>
>    ![GitHub contributor edit page](../../images/github_contributor_edit.png "CONTRIBUTOR file in edit mode")
>
> 4. Modify the `CONTRIBUTORS.yaml` by adding yourself
>
>    You should use your GitHub username and add it followed by `:` (the `:` is important) at the correct position given the alphabetical order.
>
> 5. Scroll down to the bottom of the file
> 6. Fill the **Propose file change** form
>
>    It can also be named **Commit changes** for the ones with write accesses to the repository
>
>    1. Fill the box "Update CONTRIBUTORS.yaml" with "Add < GitHub username > as contributor" (replace < GitHub username > by your GitHub username)
>
>       > <comment-title>Commit messages</comment-title>
>       > This a commit message: a description explaining why a particular change was made. Theses messages capture the history of the changes, so other contributors can understand what have been done and why
>       {: .comment}
>
>    2. Leave "Add an optional extended description..." empty
>
>    ![Propose file change form](../../images/github_propose_change_form.png)
>
> 7. Click on **Propose file change**
>
>    ![Submit propose file change form](../../images/github_propose_change_form_submit.png)
>
{: .hands_on}

Without realizing it, GitHub let you create your first branch (named here `patch-1`) and add a changement on this branch.

> <comment-title>Branching</comment-title>
> Branching is the way to work on different versions of a repository at one time. By default your repository has one branch named `main` which is considered to be the definitive branch. When you create a branch off the main branch, you're making a copy, or snapshot, of `main` as it was at that point in time.
>
> By changing a file in this branch, it will diverge from the `main` branch. It will contain data that is only on this new branch.
{: .comment}

# Open a Pull Request

Then the addition of your information in the `CONTRIBUTORS.yaml` file is currently only on your branch `patch-1`. Not on the `main` branch and so not only on the [Hall of Fame]({% link hall-of-fame.md %}). You can't add or push directly to the `main` branch, so you need to create what we call a pull request.

The GitHub interface guides you through this process: after clicking on **Propose file change**, a new page opens:

![Pull request form](../../images/github_pr_interface.png "Pull request form")

> <hands-on-title>Edit a file</hands-on-title>
>
> 1. Open and read the [CONTRIBUTING.md]({{ site.github_repository }}/blob/main/CONTRIBUTING.md) file
> 2. Come back to the pull request
> 3. Fill in the pull request description
>
>    ![Pull request description](../../images/github_pr_form.png)
>
>    1. Add a title for the Pull Request
>    2. Add a message explaining the changes you made
>
>       This message is a good way to introduce yourself and to explain the message you made. Be kind and descriptive. It helps the reviewers to understand why you did your changes and if it should be intergrated into the `main` branch (and then website).
>
>       > <comment-title>Pull request messages</comment-title>
>       > GitHub uses [Markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet), a simple Markup language, to render the Pull request messages. You can then add bold test, lists, images, etc.
>       {: .comment}
>
> 4. Scroll down and check the changes you made
>
>    - In green with `+`: what you added
>    - In red with `-`: what you deleted
>
> 5. Click on **Create pull request** or switch to **Create draft pull request** from the dropdown menu
>
>    ![Create pull request dropdown in the GitHub interface is shown, the draft pull request button option is highlighted](../../images/PR_create_menu.png)
>
>    >  <details-title>"Draft pull requests"</details-title>
>    >  [Creating a pull request as a draft](https://github.blog/2019-02-14-introducing-draft-pull-requests/#tag-your-work-in-progress) serves as an indication that you are still working on the content.
>    >  Reviewers may comment on the current state and give general feedback, but they will know that they are not looking at the final version of your contribution.
>    >
>    >  In the Galaxy Training Material repository we have also disabled the computationally most expensive automated tests on draft pull requests, and we encourage you to use the draft stage as a small contribution to sustainable computing.
>    {: .details}
{: .hands_on}

You have created your first pull request!

Your pull request will be reviewed. There are two possible outcomes:

1. Your pull request is accepted. Congratulations! Your changes will be merged into the main branch of the original repository. The website will be re-built and you will be in the [Hall of Fame]({% link hall-of-fame.md %})
2. Your pull request needs modifications: the reviewers will ask for some changes, possibly because the automatic tests are failing.

# Update a Pull Request

One of the reviewers of your pull request asked you to add your name after your GitHub username in the `CONTRIBUTORS.yaml` file.

> <hands-on-title>Update a Pull Request</hands-on-title>
>
> 1. Go to the list of pull request tab on GitHub
> 2. Click on your pull request
>
>     You can see here the comments the reviewers left you
>
> 3. Click on **Files changed** tab and see the changes you made
>
>    ![Pull request files changed tab](../../images/github_pr_file_changed.png)
>
> 4. Click on {% icon hands_on %} icon
> 5. Add your name below your GitHub username
>
>    It should look like:
>
>    ```
>    bebatut:
>         name: Bérénice Batut
>    ```
>
> 5. Navigate to the bottom of the file
> 6. Fill the **Commit changes** form, similarly to the **Propose file change** form before
> 7. Make sure the **Commit directly to the `patch-1` branch** is selected
>
>    ![Commit directly to the `patch-1` branch](../../images/github_pr_commit.png)
>
> 8. Click on **Commit changes**
>
>    ![Submit propose file change form](../../images/github_pr_commit_changes.png)
>
>    The pull request should be automatically updated
>
> 9. Check that the new changes are added to the pull request on GitHub
>
{: .hands_on}


# Close the Pull Request

Great! You now know how to make pull request on GitHub, and how to make changes after a review.
Reviewers can now approve and merge your pull request.

Because this was just a practice pull request, let's close it again.


> <hands-on-title>Close the Pull Request</hands-on-title>
>
> Once you have run through all these steps, please close the pull request again.
>
> 1. Go to the [list of pull request tab on GitHub](https://github.com/galaxyproject/training-material/pulls)
> 2. Click on your pull request
> 3. Scroll to the bottom of the page
> 3. Click on *"Close pull request"* button
>
> Whenever you add your first real contribution, you can add yourself to the `CONTRIBUTORS.yaml` file in that PR.
>
{: .hands_on}


# Conclusion


With this tutorial, you learn how to use GitHub to change a file, create a pull request and then contribute to the training material. What you have learned here can be applied to any file.

> <details-title>More about GitHub</details-title>
> Via the GitHub interface, you can also go further: create file, branch directly, etc.
> To learn that, we recommend you to read the [GitHub guide](https://guides.github.com/activities/hello-world/)
{: .details}


topic: contributing
tutorial: create-new-tutorial-tours
title: Creating Interactive Galaxy Tours
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 59e4326faf738b141c88d70a9db322c9
last_modified: 2026-02-26T21:12:14.704298
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-tours/tutorial.html

content:
Galaxy is a great solution to train the bioinformatics concepts:

- numerous bioinformatics tools are available (almost 5,000 in the ToolShed)
- it can be used by people without any computer science skills
- it trains to use technology, outlining available resources and efforts that have made them accessible to researchers
- it is scalable

In 2016, the Galaxy Training Network decide to set up a new infrastructure for delivering easily Galaxy related training material. The idea was to develop something open and online based on a community effort, as most of the time in Galaxy.

We take inspiration from [Software Carpentry](https://software-carpentry.org). We collected everything on a GitHub repository: [https://github.com/galaxyproject/training-material ](https://github.com/galaxyproject/training-material). We decided a structure based on tutorials with hands-on, fitting both for online self-training but also for workshops, grouped in topics. Each tutorial follows the same structure and comes with a technical support to be able to run.

In this tutorial, you will understand how to design and develop a new tutorial fitting in this training material repository. As doing helps to understand, we will develop a small tutorial to explain BLAST with the full infrastructure to be able to run this tutorial anywhere.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# A Galaxy Interactive Tour

A Galaxy Interactive Tour is a way to go through an entire analysis, step by step inside Galaxy in an interactive and explorative way. It is a great pedagogic way to run the tutorial directly inside Galaxy.

![Demonstration of an Interactive Tour](../../../dev/images/galaxy_tour_demo.gif "Demonstration of an Interactive Tour")

A tour is a YAML file like:

```
id: galaxy_ui
name: Galaxy UI
description: A gentle introduction to the Galaxy User Interface
title_default: "Welcome to Galaxy"

steps:
    - title: "Welcome to Galaxy"
      content: "This short tour will guide you through Galaxy's user interface.<br>
                You can navigate with your arrow keys and leave the tour at any time point
                with 'Escape' or the 'End tour' button."
      backdrop: true

    - title: "Upload your data"
      element: ".upload-button"
      intro: "Galaxy supports many ways to get in your data.<br>
              Use this button to upload your data."
      position: "right"
      postclick:
        - ".upload-button"
```

- at the top some metadata related to the Tour:
    - `id`: ID of the tour
    - `name`: name of the tour
    - `description`: a short description of the tour
    - `title_default`: a title
- several steps corresponding to the different boxes

    Each step is beginning with a dash `-` and with possible arguments

    Argument | Description
    ---  | ---
    `title`  | Header of each step-container
    `content` | Text that is shown to the user
    `element` | [JQuery Selector](https://api.jquery.com/category/selectors/) of the element you want to describe / click
    `placement` | Placement of the text box relative to the selected element
    `preclick` or `postclick` | Elements that receive a click() event before (`preclick`) or after (`postclick`) the step is shown
    `textinsert` | Text to insert if element is a text box (e.g. tool search or upload)
    `backdrop` | `true/false`:  Show a dark backdrop behind the popover and its element, highlighting the current step

    [Full reference of the properties](https://bootstraptour.com/api/)

The YAML file of a tour can be integrated in a Galaxy instance by placing the YAML file in the `config/plugins/tours` directory of the Galaxy code and restarting the Galaxy instance

# Creating a Galaxy Interactive Tour

[A Web browser plugin](https://github.com/TailorDev/galaxy-tourbuilder) is available to help the creation and the test (on the fly) of an interactive tour.

<blockquote class="imgur-embed-pub" lang="en" data-id="a/0YVvz"><a href="https://imgur.com/a/0YVvz">Galaxy Tour Builder by TailorDev</a></blockquote><script async src="https://s.imgur.com/min/embed.js" charset="utf-8"></script>

> <hands-on-title>Install and start the plugin</hands-on-title>
>
> 1. Install the plugin using the app store of your web-browser:
>     - [Chrome Web Store](https://chrome.google.com/webstore/detail/galaxy-tour-builder/mdfbapknmcpnbmggahhaegehbbbmhmgg)
>     - [Mozilla Add-ons (Firefox)](https://addons.mozilla.org/en-US/firefox/addon/galaxy-tour-builder/)
>     - [Opera add-ons](https://addons.opera.com/en/extensions/details/galaxy-tour-builder/)
>
> 2. Load the webpage of any Galaxy instance
> 3. Start the plugin by clicking on the icon with Galaxy icon close to the address bar
{: .hands_on}

We can now create easily a Galaxy Interactive Tour and test it on the fly.

> <hands-on-title>Create a Galaxy Interactive Tour</hands-on-title>
>
> 1. Create a Galaxy Interactive Tour for "BLAST" tutorial
> 2. Test it with the plugin
> 3. Copy the YAML content and add it to a file
> 2. Add the file to the `tours` directory of the tutorial
> 3. Test it on a local Galaxy instance
{: .hands_on}

# Conclusion


topic: contributing
tutorial: github-contribution
title: Contributing to the Galaxy Training Network with GitHub
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 45ae850ac715d990d18761ed3382d617
last_modified: 2026-02-26T21:12:14.709141
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/github-contribution/tutorial.html

content:
Most of the GTN content is written in [GitHub Flavored Markdown](https://guides.github.com/features/mastering-markdown/) with some metadata (or variables) found in [YAML](http://yaml.org/) files. Everything is stored on a [GitHub](https://github.com) repository: [{{ site.github_repository }}]({{ site.github_repository }}).

The process of development of new content is open and transparent, using git and following the [GitHub flow](https://guides.github.com/introduction/flow/), which we will walk you through:

![Open source development process](../../images/oss_development_process.png "Open source development process")

1. Create a fork
1. Clone your fork of this repository to create a local copy on your computer
1. Create a new branch in your local copy for each significant change
2. Commit the changes in that branch
1. Push that branch to your fork on GitHub
1. Submit a pull request from that branch to the [original repository]({{ site.github_repository }})
1. If you receive feedback, make changes in your local clone and push them to your branch on GitHub: the pull request will update automatically
1. Pull requests will be merged by the training team members after at least one other person has reviewed the Pull request and approved it.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to contribute to the GitHub repository:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Create a GitHub account, if you have not already

> <hands-on-title>Create a GitHub account</hands-on-title>
>
> 1. Go to the [GitHub website](https://github.com)
> 2. Create an account & follow the instructions. If anything is unclear, you can check [their documentation on account creation](https://docs.github.com/en/get-started/start-your-journey/creating-an-account-on-github).
{: .hands_on}

# Create a fork of this repository on GitHub

{% include _includes/cyoa-choices.html option1="Desktop" option2="Command-line" default="Desktop"
       text="You can engage with GitHub via either an application on your computer, called GitHub Desktop, or through using the Command-line. If you're not used to using the Command-line, or you just like a user interface, then use the GitHub Desktop." %}

<div class="Desktop" markdown="1">

> <hands-on-title>Download GitHub Desktop</hands-on-title>
>
> 1. Go to the [GitHub Desktop website](https://desktop.github.com/download/)
> 2. Select the **Download for (your kind of computer)**
> 3. Follow any instructions
{: .hands_on}

</div>

A fork is a copy of a repository. [Forking](https://help.github.com/articles/fork-a-repo/) a repository allows you to freely experiment with changes without affecting the original project:

![Explanation of the forking](../../images/PR_schema_01.png "Creation of a fork"){: width="900px"}

Forking a repository is a simple two-step process:

> <hands-on-title>Fork the repository</hands-on-title>
>
> 1. Go on the GitHub repository: [{{ site.github_repository }}]({{ site.github_repository }})
> 2. Click on **Fork** (top-right corner of the page)
>
>    ![Fork](../../images/PR_fork.jpg)
>
> > <comment-title></comment-title>
> > When you click the **Fork** button GitHub will show you a list with your user account and any groups where you can create a fork of this repository (starting with `@`). If you see yourself only below "You have existing forks of this repository:", it means you already have a fork and you have nothing to do.
> {: .comment}
>
{: .hands_on}

# Clone the GitHub repository on your computer

To modify the content of the repository, you need a copy of it on your computer. This step of importing a git repository is called "cloning":

![Explanation of the cloning](../../images/PR_schema_02.png "Cloning a repository"){: width="900px"}

<div class="Desktop" markdown="1">

> <hands-on-title>Clone the GitHub repository</hands-on-title>
>
> 1. Get the URL of your fork. You can find this on the main page of your fork under the green button:
>    1. Click on **Code** (right)
>
>       ![Get fork URL](../../images/clone_code_button.png)
>
>    2. Select **Open with GitHub Desktop**
>
>       Your computer might ask if it's ok to open the website Select *Allow* if so.
>     
>       ![Clone a Repository](../../images/clone_a_repo.png)
>
>    2. If you are happy with the location of the folder, select **Clone**
>       This may take some time the first time (as the GTN repository is quite large!), but eventually the folder will be in your computer.
{: .hands_on}

</div>

<div class="Command-line" markdown="1">

> <hands-on-title>Clone the GitHub repository</hands-on-title>
>
> 1. Get the URL of your fork. You can find this on the main page of your fork under the green button:
>    1. Click on **Code** (right)
>
>       ![Get fork URL](../../images/clone_code_button.png)
>
>    2. Copy the URL
>
>       It should be something like `https://github.com/< Your GitHub Username >/training-material.git`
>
> 1. Open a terminal
> 2. Navigate with `cd` to the folder in which you will clone the repository
> 3. Clone the repository with the command:
>
>    ```
>    $ git clone https://github.com/< Your GitHub Username >/training-material.git
>    ```
>
> 4. Navigate to the repository
>
>    ```
>    $ cd training-material
>    ```
{: .hands_on}

> <comment-title></comment-title>
> If you already have a local copy of the GitHub repository, you need to update it before doing any changes. To learn how to do that, please follow the [last section](#stay-up-to-date).
{: .comment}

</div>

# Create a new branch

You have now your repository locally and you want to modify it. For this example tutorial, you will add yourself as contributor of the project to appear on the [Hall of Fame]({% link hall-of-fame.md %}).

In GitHub flow, there is a concept: one new feature or change = one branch.

When you're working on a project, you're going to have a bunch of different features or ideas in progress at any given time – some of which are ready to go, and others which are not. Branching exists to help you manage this workflow. You should develop different features on different branches to help keep the changes you make simple and easy to review.

![Divergence of a branch compared to main](../../images/PR_branches_01.png "When you create a branch in your project, you're creating an environment where you can try out new ideas. Changes you make on a branch don't affect the main branch")

Here for this tutorial, you will create a branch called "my_new_branch" in which you will modify the `CONTRIBUTORS.yaml` file, the file used to generate the [Hall of Fame]({% link hall-of-fame.md %}).

<div class="Desktop" markdown="1">

> <hands-on-title>Create a branch</hands-on-title>
>
> 1. Go to *Current Branch*
>
>       ![Click on Current Branch](../../images/desktop-select_current_branch.png)
>
> 2. Either select an existing branch by searching, or (as in this case) type a name for a new branch:
>    ```
>    my_new_branch
>    ```
>       ![Select a branch option](../../images/desktop-select_a_branch.png)
>       ![Type in new branch name](../../images/desktop-new_branch.png)
>
> 3. Select **Create New Branch**
>
>       ![Create branch confirmation](../../images/desktop-create_branch.png)
>
> 4. Select **Create Branch**
{: .hands_on}


</div>

<div class="Command-line" markdown="1">

> <hands-on-title>Create a branch</hands-on-title>
>
> 1. List the existing branch
>
>    ```
>    $ git branch
>      * main
>    ```
>
>    The branch on which you are is shown with the `*`
>
> 2. Create a new branch
>
>    ```
>    $ git checkout -b my_new_branch
>    Switched to a new branch 'my_new_branch'
>    ```
>
> 3. List the existing branch to check that the branch has been created and you are now on it
{: .hands_on}

</div>

This branch is added to your local copy:

![Creation of a branch on the local copy of the repository](../../images/PR_schema_03.png "Creation of a branch on the local copy of the repository")

# Make your changes on this branch

You have created your first branch! Now you want to make the change in the `CONTRIBUTING.yaml` file. By changing a file in this branch, it will diverge from the `main` branch. It will contain data that is only on this new branch:

![Divergence of the branch compared to main](../../images/PR_branches_02.png "The changes on your branch will not be on the main branch")

<div class="Desktop" markdown="1">
> <hands-on-title>Make changes in a branch</hands-on-title>
>
> 1. Select **Open the repository in your external editor** if you have one. Otherwise, you will need to find a text editor. You can consider using [Visual Studio Code](https://code.visualstudio.com), for example. The text editors allow you to navigate through folders and type in code with automatically detected formatting, which is very helpful!
>
>       ![Open in editor](../../images/desktop-open_in_editor.png)
>
> 2. Scroll down to the `CONTRIBUTORS.yaml` file and follow the text in that file to add yourself.
>
>       ![Find the contributors file](../../images/desktop-contributors_file.png)
>
> 3. Save the file
>
> 4. Go back to your GitHub Desktop, which should now show that file change. Add a useful *commit message* and a *description* saying what you did.
>
>       ![Find the contributors file](../../images/desktop-commit.png)
>
> 5. Select **Commit to my_new_branch**
>
{: .hands_on}
</div>


<div class="Command-line" markdown="1">

> <hands-on-title>Make changes in a branch</hands-on-title>
>
> 1. Open with your favorite text editor the `CONTRIBUTORS.yaml` file that is on your computer
> 2. Add yourself in the `CONTRIBUTORS.yaml` file
>
>    You should use your GitHub username and add it followed by `:` at the correct position given the alphabetical order
>
> 2. Save the file
> 3. Check the changes you made
>
>    ```
>    $ git status
>    On branch my_new_branch
>    Changes not staged for commit:
>      (use "git add/rm <file>..." to update what will be committed)
>      (use "git checkout -- <file>..." to discard changes in working directory)
>
>        modified:   CONTRIBUTORS.yaml
>
>    no changes added to commit (use "git add" and/or "git commit -a")
>    ```
>
> 4. Add the file and commit the changes
>
>    ```
>    $ git add CONTRIBUTORS.yaml
>    $ git commit -m "Add ..."
>    ```
>
> 5. Check that there are no more changes to commit with `git status`
>
{: .hands_on}

</div>


# Push your branch on your GitHub repository

The changes you made on your branch are only on the local copy of the repository. To propagate them online, you need to push them on your fork on GitHub:

<div class="Desktop" markdown="1">
> <hands-on-title>Push the changes</hands-on-title>
>
> 1. Select **Publish branch**
>       ![Publish branch](../../images/desktop-publish_branch.png)
>
{: .hands_on}

</div>


<div class="Command-line" markdown="1">

> <hands-on-title>Push the changes</hands-on-title>
>
> 1. Push the changes to the GitHub repository
>
>    ```
>    $ git push origin my_new_branch
>    ```
>
>    > <details-title>"push origin"</details-title>
>    > When you `git push`ed, you specified `origin`. Git repositories can know that forks exist in multiple places. When you clone one, it creates a "remote" (a remote repository) which it names `origin`, set to the URL that you used when you cloned. By having multiple remotes, you can manage more complex workflows.
>    {: .details}
>
{: .hands_on}

</div>

To check this has worked, you can go to your GitHub account.

> <hands-on-title>Push the changes</hands-on-title>
> 1. Go to your GitHub repository on a web browser
> 2. Change to the "my_new_branch" branch:
>    1. Click on **Branch: main** (left)
>
>       ![Selecting branch on GitHub](../../images/PR_branch_github.png)
>
>    2. Select the branch "my_new_branch"
>
> 4. Check that your name is in the `CONTRIBUTORS.yaml` file
>
{: .hands_on}

![Pushing changes to the fork from the local copy](../../images/PR_schema_04.png "Pushing changes from the local copy to the fork on GitHub")

# Open a pull request

You pushed your changes to GitHub, but currently they are only on your fork. You want to have these changes in the main GitHub repository in order to appear on our [Hall of Fame]({% link hall-of-fame.md %}) online. You can't add or push directly the main GitHub repository, so you need to create what we call a pull request:

![Pull request](../../images/PR_schema_05.png "Pull Requests provide a way to notify project maintainers about the changes you'd like them to consider")

<div class="Desktop" markdown="1">
> <comment-title></comment-title>
> You will see that you can also make pull requests directly from GitHub Desktop. That also works great!
{: .comment}

</div>

> <hands-on-title>Create a pull request</hands-on-title>
>
> 2. Go to your GitHub repository
> 1. Click on **Compare & pull request**
>
>    !["Opening a pull request"](../../images/PR_button.png)
>
> 3. Check that the selected branch are correct: **main** on the left and your branch name on the right
>
>    ![Branches in PR](../../images/PR_branch_check.png)
>
> 2. Fill in the pull request description
>
>    ![PR description](../../images/PR_description.png)
>
>    1. Add a title for the Pull Request
>    2. Add a message explaining the changes you made (Be kind <i class="fa fa-smile-o" aria-hidden="true"></i>)
>    3. Click on **Create pull request** or switch to **Create draft pull request** from the dropdown menu
>
>       ![Create pull request dropdown in the GitHub interface is shown, the draft pull request button option is highlighted](../../images/PR_create_menu.png)
>
>       >  <details-title>"Draft pull requests"</details-title>
>       >  [Creating a pull request as a draft](https://github.blog/2019-02-14-introducing-draft-pull-requests/#tag-your-work-in-progress) serves as an indication that you are still working on the content.
>       >  Reviewers may comment on the current state and give general feedback, but they will know that they are not looking at the final version of your contribution.
>       >
>       >  In the Galaxy Training Material repository we have also disabled the computationally most expensive automated tests on draft pull requests, and we encourage you to use the draft stage as a small contribution to sustainable computing.
>       {: .details}
>
> 3. Go to **Pull requests** to check if it is there
{: .hands_on}

Once the pull is open, it will be reviewed. There are two possible outcomes:

1. Your pull request is accepted. Congratulations! Your changes will be merged into the main branch of the original repository. The website will be re-built and you will be in the [Hall of Fame]({% link hall-of-fame.md %})
2. Your pull request needs modifications: the reviewers will ask for some changes, possibly because the automatic tests are failing.

# Make any requested changes

<div class="Desktop" markdown="1">

Three things can happen:

1. A reviewer commits changes to your pull request (they fix something themselves)
2. A reviewer suggests changes to your pull request (they think something will be helpful, but want you to check and decide)
3. A reviewer posts comments, asking you to make some sort of change.

On larger pull requests, you can end up with all three! Here's how to deal this.

## 1: A reviewer commits their own changes to your pull request

These changes currently only exist in your GitHub repository online, not on your local branch. In case you have to make any further changes, it's important you *fetch* those changes onto your local branch. This is how you stay up to date.

> <hands-on-title>Fetch origin</hands-on-title>
> 1. On your GitHub Desktop, select **Fetch origin**
>       ![Fetch origin](../../images/desktop-fetch_origin.png)
>
{: .hands_on}

You can then make further changes locally, keeping your local and online branch synched.

## 2: A reviewer suggests changes

> <hands-on-title>Commit suggested changes</hands-on-title>
>
> 1. On your Pull Request link where you see the suggested changes, commit them by clicking on the "Commit suggestion" button (or engage further, or edit them) on GitHub
>
>       ![Fetch origin](../../images/desktop-commit_suggestion.png)
>
> 2. Now that your branch is updated, you need to **Fetch origin**
>
>       ![Fetch origin](../../images/desktop-fetch_origin.png)
>
{: .hands_on}

You can then make further changes locally, keeping your local and online branch synched (are you noticing a pattern?).

## 3: A reviewer posts comments to be addressed

</div>

For example, one of the reviewers of your pull request may have asked you to add your name after your GitHub username in the `CONTRIBUTORS.yaml` file, if you didn't already.

<div class="Desktop" markdown="1">

> <hands-on-title>Make further changes</hands-on-title>
>
> 1. Just like when you made your initial branch and edited the file in your text editor, go to GitHub Desktop | **Current Branch** (make sure it is the correct one!) | **Open the repository in your external editor**
>
> 2. Navigate to the `CONTRIBUTORS.yaml` file and make any suggested changes
>
> 3. Save the file
>
> 4. Commit the changes to your branch (as before)
>
> 5. Select **Push origin** to push these changes from your local folder to your online GitHub branch (which is linked to your pull request)
>
>       ![Push origin](../../images/desktop-push_origin.png)
>
>    The pull request should be automatically updated
>
> 6. Check that the new changes are added to the pull request on GitHub
>
{: .hands_on}

</div>


<div class="Command-line" markdown="1">



> <hands-on-title>Make further changes</hands-on-title>
>
> 1. Make any requested changes in the `CONTRIBUTORS.yaml` file
>
>    It should look like
>
>    ```yaml
>    bebatut:
>         name: Bérénice Batut
>    ```
>
> 2. Check the changes that you made
>
>    ```
>    $ git status
>    On branch my_new_branch
>    Changes not staged for commit:
>      (use "git add/rm <file>..." to update what will be committed)
>      (use "git checkout -- <file>..." to discard changes in working directory)
>
>        modified:   CONTRIBUTORS.yaml
>
>    no changes added to commit (use "git add" and/or "git commit -a")
>    ```
>
> 3. Add the file and commit the changes
>
>    ```
>    $ git add CONTRIBUTORS.yaml
>    $ git commit -m "Add ..."
>    ```
>
> 4. Check that there are no more changes to commit with `git status`
> 5. Push the new changes to GitHub
>
>    ```
>    $ git push origin my_new_branch
>    ```
>
>    The pull request should be automatically updated
>
> 6. Check that the new changes are added to the pull request on GitHub
>
{: .hands_on}

</div>

# Check the automatic tests

When a pull request is opened, some automated tests are automatically launched on [Travis](http://travis-ci.com/) to be sure that the changes do not break the website, the URL are valid, etc.

On the bottom of your pull request, you can see the status of the tests:

- Yellow (with circle)

    ![Running](../../images/PR_test_yellow.png)

    The tests are still running

- Red (with cross)

    ![Failed tests](../../images/PR_test_red.png)

    When it is red, you can investigate why by clicking on **Details**. You will be redirected on [Travis](http://travis-ci.com/) where you can see the logs of the tests. Get in touch with us on [Gitter]({{ site.gitter_url }}) if you need help to understand the issue.

- Green (with tick)

    ![Passed tests](../../images/PR_test_green.png)

    The tests passed. Good job!

    Even it is green, we recommend to check the result of the tests, as some of tests are allowed to fail (to avoid too much noise).




<div class="Command-line" markdown="1">

# Stay up to date

You now want to work on a new tutorial or make some other new changes. However since you get a local copy, some changes have happened to the original GitHub repository. You need then to update your local copy of the repository before changing anything.

> <hands-on-title>Update the local copy</hands-on-title>
>
> 1. Move to the `main` branch
>
>    ```
>    $ git checkout main
>    ```
>
> 2. Add a reference to the original GitHub repository
>
>    ```
>    $ git remote add upstream https://github.com/galaxyproject/training-material.git
>    ```
>
>    > <comment-title>Error "remote upstream already exists"</comment-title>
>    > If you have done step 2 before and try to `remote add` again, git will tell you that a "remote upstream already exists". In this case you can safely continue to step 4.
>    {: .comment}
>
> 4. Update the local copy of the repository by "pulling" in the content of the original GitHub repository
>
>    ```
>    $ git pull upstream main
>    ```
>
{: .hands_on}


You can now restart the GitHub flow to propose new changes: start by [creating a new branch](#create-a-new-branch).
</div>


# Finishing the pull request

Great! You now know how to make pull request on GitHub, and how to make changes after a review. Reviewers can now approve and merge your pull request, however, there's no point adding you as a contributor if you don't plan to contribute! So you have to make a choice - do you plan to contribute now, later, or never?

{% include _includes/cyoa-choices.html disambiguation="whencontribute"  option1="Contribute-now" option2="Contribute-later" option3="Contribute-never" default="Contribute-now"
       text="Your choice impacts how you will continue with your pull request." %}

<div class="Contribute-never" markdown="1">

If you do not intend to contribute to Galaxy, please close the pull request as per below.

> <hands-on-title>Close the Pull Request</hands-on-title>
>
> 1. Go to the [list of pull requests tab on the GitHub website](https://github.com/galaxyproject/training-material/pulls)
> 2. Click on your pull request
> 3. Scroll to the bottom of the page
> 3. Click on "Close pull request" button
>
{: .hands_on}

</div>

<div class="Contribute-later" markdown="1">

If you plan to contribute to Galaxy, but not anytime soon, we ask that you close your Pull Request. When you come back to contribute to Galaxy, you can add any changes to the branch that you are working on, and open a fresh pull request following the same instructions above!

> <hands-on-title>Close the Pull Request</hands-on-title>
>
> 1. Go to the [list of pull requests tab on the GitHub website](https://github.com/galaxyproject/training-material/pulls)
> 2. Click on your pull request
> 3. Scroll to the bottom of the page
> 3. Click on "Close pull request" button
>
{: .hands_on}

</div>

<div class="Contribute-now" markdown="1">

Thank you so much for being willing to contribute! The easiest way to start contributing is to look through training materials to find typos, poor figure legends or alt text - basically, a text-based issue that you can correct. If you're feeling bold, you can also look at some of these [ideas for ways to contribute]({% link faqs/gtn/contributors_getting_started.md %}).

1. Find a typo/error
2. Fix it locally, just the way you edited the `CONTRIBUTORS.yaml` file above.
3. Commit this change to your branch (the same one with yourself added to the `CONTRIBUTORS.yaml` file)
4. Push this change from your local machine to the Github site
5. Check the change is reflected in your Pull Request

The steps are the same as your initial contribution to the `CONTRIBUTORS.yaml` file. You will then, as before, engage further with reviewers to finalise your Pull Request, and eventually (if all goes well!) your Pull Request will be approved and merged!

{% icon congratulations %} Congrats, you'll have made your first contribution to the Galaxy Training Network!

</div>

# Conclusion


With this tutorial, you have learned some basics `git` commands and principles:

![Summary of the links between GitHub, fork and local repository](../../images/PR_global_schema.png "Summary of the links between GitHub, fork and local repository")

You also learned the GitHub flow and its cycle:

1. Create a new branch in your local copy
2. Commit the changes in that branch
1. Push that branch to your fork on GitHub
1. Submit a pull request from that branch to the main repository
1. Wait for feedbacks and make requested changes
1. Update your local copy
1. Restart the cycle

You can now contribute and help us to improve our tutorials!

> <details-title>Git in Depth</details-title>
> This tutorial was a quick introduction to explain the basics of contributing to the training material. We recommend that everyone follow a more detailed git tutorials:
> - [Software Carpentry tutorial](http://swcarpentry.github.io/git-novice/) which explains a bit more in detail some git commands and the concept of remote, local and stagging
> - [Learn Git Branching](https://learngitbranching.js.org/) by GitHub to test the different git commands
>
> You should also download, print and keep always with you the [Git Cheat Sheet](https://services.github.com/on-demand/downloads/github-git-cheat-sheet.pdf)
{: .details}


topic: contributing
tutorial: meta-analysis-plot
title: Single Cell Publication - Data Plotting
difficulty: advanced
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["R"]
content_hash: 42239cf6a85f0c8a5e8e2839660ac310
last_modified: 2026-02-26T21:12:14.717800
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/meta-analysis-plot/tutorial.html

content:
We'll use some ggplot and tidyverse code to plot the data we collected in part 1

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


We'll use tidyverse (which includes things like magrittr (`%>%`) and ggplot2) to load our data. Reshape2 provides the cast/melt functions which can be used to reshape specific datasets into formats that are easier to plot.

```R
library(tidyverse)
library(reshape2)
# This path will probably need to be changed depending on where you downloaded that dataset.
data = read_tsv("sc.tsv")
```

## Data Cleaning

Let's start by cleaning our data a bit. We've got a couple problems with it:

- there might be some null mergedAt dates, we'll want to remove those
- we probably want to ignore a couple of classes of datasets that were added/removed (images, and the explicit 'ignore' type.)

```R
clean = data %>% 
	select(num, class, additions, deletions, mergedAt) %>% 
	filter(!is.na(mergedAt)) %>% 
	group_by(num, class, mergedAt, month=floor_date(mergedAt, 'month'), quarter=floor_date(mergedAt, 'quarter')) %>% 
	summarise(additions=sum(additions), deletions=-sum(deletions)) %>%
	filter(class != "ignore") %>%
	filter(class != "image") %>%
	arrange(mergedAt) %>%
	as_tibble()
```

We'll setup a 'theme' for our plot that mainly consists of using the black and white theme which is quite elegant and readable, and then making some font sizes a wee bit larger:

```R
theme = theme_bw() + theme(
  axis.text=element_text(size=14), 
  plot.title=element_text(size=18), 
  axis.title=element_text(size=16),
  legend.title=element_text(size=14),
  legend.text=element_text(size=14))
```

We want all of our plots to look the same which is why we use this trick, it helps us keep a consistent aesthetic without having to re-type the configuration every single time.

Let's get to plotting!

## Plot: Lines added/removed by date/time period

We have a dataset that looks like:

```R
clean %>% head(10)
```

We'll want to plot the changes versus the time (either quarter or month), and
then maybe plot them differently based on the class of the addition/removal. So that translates into an aesthetics statement like `aes(x=time, y=additions, fill=class)`

In ggplot2 you can plot data some different ways, if you provide the datasets upfront, e.g. `data %>% ggplot()` you'll generally do something like `data %>% ggplot(aes(x=a, y=b)) + geom_something()` and `geom_something` will take the data from the ggplot call. However if you want to plot multiple series, you can also provide the data directly to the `geom_*` functions like so:

```R
ggplot() +
  geom_col(data=clean, aes(x=quarter, y=additions, fill=class)) +
  geom_col(data=clean, aes(x=quarter, y=deletions, fill=class)) + scale_fill_brewer(palette = "Paired") +
  geom_point(data=clean, aes(x=mergedAt, y=0), shape=3, alpha=0.3, color="black") +
  theme +
  xlab("Quarter") + ylab("Lines added or removed") + guides(fill=guide_legend(title="Category")) +
  ggtitle("Lines added or removed by file type across GTN Single Cell Learning Materials")
ggsave("sc-lines-by-quarter.png", width=12, height=5)

ggplot() +
  geom_col(data=clean, aes(x=month, y=additions, fill=class)) + 
  geom_col(data=clean, aes(x=month, y=deletions, fill=class)) + scale_fill_brewer(palette = "Paired") +
  theme +
  geom_point(data=clean, aes(x=mergedAt, y=0), shape=3, alpha=0.3, color="black") +
  xlab("Month") + ylab("Lines added or removed") + guides(fill=guide_legend(title="Category")) +
  ggtitle("Lines added or removed by file type across GTN Single Cell Learning Materials") 
ggsave("sc-lines-by-month.png", width=12, height=5)
```

![very inconsistent lines added by quarter some quarters have more and some have less but it is mostly workflow and then tutorial lines being added or removed.](./images/sc-lines-by-quarter.png)

In this case we plotted the data for additions and deletions separately, and we additionally added points based on the actual date of the PRs to visualise their density.


Let's produce @gtn:nomadscientist's favourite running sum plots in addition. We'll start by reshaping our data. Currently we have data that looks like:

time | variable | value
-- | --- | ---
today | measure1 | 1
today | measure2 | 10
yesterday | measure1 | 30
yesterday | measure2 | 5

And we'll re-shape this to look like this, which will make it easier to calculate changes over the course of a specific series:

time | measure1 | measure2
--- | --- | ---
today | 1 | 10
yesterday | 30 | 5


We'll use the `dcast` function to do that:

```
clean %>% select(month, class, additions)  %>% dcast(month ~ class, value.var="additions") %>% head()
```

That doesn't look quite right so, let's change how the data is aggregated:

```R
# cumulative
clean %>% select(month, class, additions)  %>% dcast(month ~ class, value.var="additions", fun.aggregate = sum)
```

Let's do it for real now:

```R
cumulative = clean %>% select(month, class, additions)  %>% 
  dcast(month ~ class, value.var="additions", fun.aggregate = sum) %>%
  mutate(across(bibliography:workflows, cumsum)) %>% 
  reshape2::melt(id.var="month")

cumulative %>% ggplot(aes(x=month, y=value, color=variable)) + geom_line() + 
  theme_bw() + theme +
  xlab("Month") + ylab("Lines added") + guides(color=guide_legend(title="Category")) +
  ggtitle("Cumulative lines added by category, across GTN Single Cell materials")
ggsave("sc-lines-cumulative.png", width=12, height=8)
```

![workflows and tutorials are the highest with on the order of 25k lines added, it used to be quite jagged with code being thrown over the wall but now it's rather smooth with consistent constant updates.](./images/sc-lines-cumulative.png)

## Contributions over time

Let's again start with some cleaning, namely removing all rows with 0 records, and removing future records (at the time of writing.)

```R
roles = read_tsv("sc-roles.tsv")
w = roles %>% 
  filter(count != 0) %>% 
  filter(!grepl("2025", date)) %>%
  filter(!grepl("2024-12-01", date))

w %>% 
  ggplot(aes(x=date, y=count, color=area)) + 
  theme +
  xlab("Date") + ylab("Unique Contributors") + guides(color=guide_legend(title="Contribution Area")) +
  ggtitle("Contributors over time to GTN Single Cell Materials") +
  geom_line()
ggsave("sc-contribs.png", width=12, height=6)
```

![plot of several lines increasing over time in a very discrete way, authorship remains highest](./images/sc-contribs.png)

## New X over Time

Let's plot all of the new single cell things added over time, all the new FAQs, Tutorials, Slides, Etc:

```R
added_by_time = read_tsv("single-cell-over-time.tsv")
added_by_time %>% dcast(date ~ `type`) %>% 
  arrange(date) %>% 
  mutate(across(event:workflow, cumsum)) %>% 
  melt(id.var="date") %>% 
  as_tibble() %>% arrange(date) %>% 
  ggplot(aes(x=date, y=value, color=variable)) + geom_line() + 
  theme_bw() + theme +
  xlab("Date") + ylab("New Single Cell Items") + guides(color=guide_legend(title="Contribution Type")) +
  ggtitle("New Single Cell events, FAQs, news, slides, tutorials, videos and workflows in the GTN")
ggsave("sc-files-cumulative.png", width=6, height=6)
```

We may also want to know how many changes there have been since the start date of this study, e.g. October 1st, 2020:

```R
since_oct = added_by_time %>% dcast(date ~ `type`) %>% 
  arrange(date) %>% 
  mutate(across(event:workflow, cumsum)) %>% 
  filter(date > as.Date("2020-10-01"))

# Pull out the first/last date as our start ane dnwend
start_date = (since_oct %>% head(n=1))$date
end_date = (since_oct %>% tail(n=1))$date

# Table of our changes.
since_oct %>% 
  filter(date == start_date | date == end_date) %>%  # Just those rows
  mutate(date = case_when(date == start_date ~ 'start', date == end_date ~ 'end')) %>% # Relabel the start/end as literal string start and end
  pivot_longer(-date) %>% pivot_wider(names_from=date, values_from=value) %>% # Transpose the data
  mutate(increase=end - start) %>% # And calculate our increase
  select(name, increase) %>% arrange(-increase)
```

## Pageviews

GTN uses the Galaxy Europe Plausible server for collecting metrics (you can change your preferences in [your GTN privacy preferences]({% link user/privacy.md %})).
We can download the data from the server and plot it, filtering by our preferred start/end dates and filters (namely that page includes `/topics/single-cell/`).
Unfortunately the data is downloaded as a zip file which we'll then need to extract data from:

```R
system("wget 'https://plausible.galaxyproject.eu/training.galaxyproject.org/export?period=custom&date=2024-11-07&from=2022-08-01&to=2024-11-07&filters=%7B%22page%22%3A%22~%2Ftopics%2Fsingle-cell%2F%22%7D&with_imported=true&interval=date' -O sc-stats.zip")
```

We can use the unzip function to read a single file directly from the zip:

```R
views = read_csv(unzip("sc-stats.zip", "visitors.csv"))
```

With that we're ready to plot. We're going to use a new feature for our plot,
`annotate`. Annotation allows you to draw arbitrary features atop your plot, in
this case we're going to draw rectangles to indicate outages and events that
might have affected our data.

```R
y = 3300
xoff = 3
views %>% group_by(date=floor_date(date, 'week')) %>% 
  summarise(date, visitors=sum(visitors), pageviews=sum(pageviews)) %>% 
  filter(visitors != 0 | pageviews != 0) %>%
  melt(id.var="date")  %>% 
  ggplot(aes(x=date, y=value, color=variable)) + geom_line() + 
  annotate("rect", xmin = as.Date("2023-10-01"), xmax = as.Date("2023-10-27"), ymin = 0, ymax = y,  alpha = .2) + # Outage
  annotate("rect", xmin = as.Date("2023-05-22") - xoff, xmax = as.Date("2023-05-26") - xoff, ymin = 0, ymax = y,  alpha = .2) + # Smorg3
  annotate("rect", xmin = as.Date("2024-10-07") - xoff, xmax = as.Date("2024-10-11") - xoff, ymin = 0, ymax = y,  alpha = .2) + # GTA
  annotate("rect", xmin = as.Date("2024-09-16") - xoff, xmax = as.Date("2024-09-20") - xoff, ymin = 0, ymax = y,  alpha = .2) + # Bootcamp
  theme_bw() + theme +
  scale_y_continuous(expand = c(0, 0), limits = c(0, NA)) +
  xlab("Date") + ylab("Count") + guides(color=guide_legend(title="Metric")) +
  ggtitle("GTN Single Cell Visits")
ggsave("sc-pageviews.png", width=14, height=4)
```

for our lovely pageview plot!

![pageviews over time, it's relatively stable except for the large annotated outage, and a large spike around the bootcamp date](./images/sc-pageviews.png)

> <tip-title>Do you want to re-do this analysis? For your community?</tip-title>
> [Discuss this with us](https://github.com/galaxyproject/training-material/issues/new?title=GTN%20Metadata%20reuse) and we can perhaps generalise this analysis, to reduce the amount of data processing you need to do, and make it more accessible for everyone!
{: .tip}


topic: contributing
tutorial: python
title: Teaching Python
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 249c7b9de061bf9e0bfe872bf8549758
last_modified: 2026-02-26T21:12:14.719629
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/python/tutorial.html

content:
Improving student learning and out comes should always be a goal of teaching. Here we present several strategies to improve student experiences during courses by focusing on how they approach specific problems, and giving them real world applicable solutions to those problems.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to run a local instance of the GTN website:
>
> 1. TOC
> {:toc}
>
{: .agenda}

"Live coding", as espoused by the Carpentries, is a fantastic strategy to communicate material to students and ensure they get a hands-on experience simultaneously.
Showing what happens live on the screen is received well by students, if
they can manage to watch what we type and try to type it themselves
simultaneously. We know at least that our examples give the correct result,
*but* students never see anything other than correct, working code, and never
have to formulate an internal model for how to write code. They end up
copying and pasting and not understanding *why*.

Predicting code behaviour without running it is a key component of work as a
programmer, and a lot of the time we spend debugging relies on us emulating
the computer in our head. Without a solid mental model of code behaviour
one cannot predict how it will function in one situation, much less other or
non-standard situations. Planning for code to handle both good and bad
inputs requires some creativity and mentally planning around expected values
at various points throughout the execution.

This situation leaves students unprepared for incorrect or buggy code,
either (un)intentionally included in homework assignments, or, generated by
themselves, if they cannot identify where code will fail without executing
it.

Augmenting lessons with:

- Pair programming
- Tracing - Stepping through the internal state
- Faded examples
- Compounding examples
- Debugging intentionally broken examples

Will give students enough tools to respond dynamically to failure
states with informed experience to resolve issues they encounter as programmers.

The student's mental model of the code underlies everything they do as a programmer, from conception to implementation to debugging to their self efficacy:

> This study shows that a well-developed and accurate mental model directly
> affects course performance and also increases self efficacy, the other key
> element in course performance. Given this double impact, helping students
> develop good mental models should remain a goal in introductory programming
> courses.
{: .quote cite="{% cite_url Ramalingam_2004 %}"}

This is a foundational skill to be able to *think* through a program, step by step, and understand how the code executes and which variables exist when, and what their values should be. This mental modelling allows students to predict the behaviour of a system, and when it diverges from their prediction, recognise any potential bugs.

## Course Management Strategies

### Pair Programming

Complementary to the other strategies, [Pair Programming](https://en.wikipedia.org/wiki/Pair_programming) or "pairing" provides a reinforcement activity where they utilise similar skills. As one person writes and executes code, the other person 'drives' the experience, telling them what to write ({% cite Williams %}, {% cite Williams_2001 %}). It has become a common learning model in introductory courses due to its benefits to students ({% cite mendes2005investigating %}, {% cite mendes2006replicated %}, {% cite Hannay_2007 %}). Specifically this technique has also been shown to be beneficial for women in computer science and gives them better chances for success in future programming endeavours ({% cite werner2004pair %}). Adopting this technique is promising, provided you adhere to principles outlined by {% cite Mentz_2008 %}.

These can often be implemented as breakout rooms wherein students are assigned a handful of problems to complete. After the breakout rooms end, you can have students summarize solutions, pick on individual ones for their ideas, etc.

## Problem Strategies

### Tracing Code Execution

> > <code-in-title>Code</code-in-title>
> > ```python
> > # Initialise our accumulator
> > x = 1 + 1
> > # Loop over our input data
> > for i in range(10): # 0..9
> >     # In-loop temporary variable
> >     tmp = x * 2 + i
> >     # Update our accumulator
> >     x = tmp + 1
> > # Output our result
> > print(f'The final value is {x}')
> > ```
> {: .code-in}
>
> > <code-out-title>Trace</code-out-title>
> >
> > Line | `i` | `x` | `tmp`
> > ---- | --- | --- | -----
> > 2    | n/a | 2   |  n/a
> > 4    | 0   | 2   |  n/a
> > 6    | 0   | 2   | 4
> > 8    | 0   | 5   | 4
> > 4    | 1   | 5   |  n/a
> > 6    | 1   | 5   | 11
> > 8    | 1   | 12  | 11
> > 4    | 2   | 12  | n/a
> > 6    | 2   | 12  | 26
> > 8    | 2   | 27  | 26
> >
> {: .code-out}
{: .code-2col}

While there is no bug in the above, when there *is* a bug present, having students produce a table like that significantly improves their understanding of code flow and execution {% cite Hertz_2013 %}. "Tracing" is a valuable and easy to complete exercise, and the results can even be checked automatically leading to good scalability of the exercise across larger classes.

Here students can also use a Debugger like [pudb](https://github.com/inducer/pudb) which can follow the execution of a bit of code, and show exactly how it's working.

<script id="asciicast" src="https://asciinema.org/a/530572.js" data-theme="monokai" async></script>

Here teaching liberal use of the `print()` command, as opposed to more complicated tools like the above, can give students the tools they need to solve problems.
This was generated by [hexylena/auto-python](https://github.com/hexylena/auto-python/) which can be reused or contributed to if new examples are needed.

### Faded Examples

When teaching programming one must constantly be cognisant of the student's cognitive load. It is a complicated task that demands a lot of students, requiring types of explicit logic analysis that they may not have engaged in before. Both learning based on problem-solving and worked examples may cause high cognitive loads for different audiences, and exploring alternatives is important ({% cite Retnowati_2017 %}). Faded examples such as what is seen below are exactly such an alternative, starting with a fully worked example and removing successive components until we reach a problem description requiring a full solution. This leads to fewer unproductive learning events ({% cite Renkl_2004 %}).

```python
### Write a function that multiplies two numbers
def multiply(a, b):
    c = a * b
    return c
```

The initial problem shows the entire solution to students

```python
### Write a function that adds two numbers
def add(___):
    ____
    return c
```

Increased fading, here we call out blanks students should fill in specifically with syntactically incorrect underscores.

```python
### Write a function that subtracts two numbers
```
Final fading, the entire problem is gone except for the description of what they need to do.

Faded examples however, do come at a higher cost of implementation than worked out examples ({% cite Zamary_2018 %}). They require writing the correct worked out example and then determining which components to remove, which presents an additional cost during course updates that if examples are changed they need to be double checked to ensure they are still valid, whereas worked examples can be checked more automatically.

### Compounding Problems

Compounding problems are a good strategy for homework problems, as you can ask multiple things of students and provide a gentle ramp up to increased complexity. Start by designing a small but complex problem like "write a fastq trimmer", where here they need to implement several different subtasks:

- file processing
- several utility functions
- multiple filter stages
- a single main function which combines all of the above

If done correctly, the students have freedom to move around individual functions that aren't dependent on each other, making sure they're correct, before building them up into a final function.

There are two alternative ways to further design the problem:

- Provide it broken down, precise small functions they should implement.
- Describe the problem and let students determine the optimal way to break it down into small, manageable components.

Which option is prefgerrable depends strongly on how advanced your students are. See the example homework [FastQ_trimmer.html](homework/FastQ_trimmer.html) and it's [associated ipynb file](homework/FastQ_trimmer.ipynb).

### Debugging

Debugging is the act of identify and resolving "bugs" or defects within code, a term popularly attributed to my personal hero Admiral Grace Hopper:

> While she was working on a Mark II computer at Harvard University, her associates discovered a moth stuck in a relay and thereby impeding operation, whereupon she remarked that they were "debugging" the system
{: .quote cite="{% cite_url enwiki:1069955193 %}"}

Debugging also functions as a reinforcement method we can use once students have an ok mental model of code execution, a necessary pre-requisite for this activity, which can be further developed through debugging ({% cite Ramalingam_2004 %}) alongside their self-efficacy ({% cite Michaeli_2019 %}). Debugging activities can take many forms but most commonly the task is to correct incorrect code, an activity that works best if they are primed with a number of methods of debugging ({% cite Murphy_2008 %}) such as the "Wolf Fence" ({% cite Gauss_1982 %}), commenting out code, or breakpoints.

```python
### Fix me!
for number in range(10):
    # use a if the number is a multiple of 3, otherwise use b
    if Number \% 3 == 0:
        message = message + a
    else:
        message = message + "b"
print(message)
```

The above debugging exercise featuring code with numerous issues from type confusion, variable typos, and failure to initialise a variable.
Students can run this example iteratively to figure out where it fails and attempt to fix it.

Use of more complex debugging tools is not always indicated, as the cognitive complexity may be too much for students. 

## Comparisons to K-12 methodologies

In K-12 teaching ({% cite Sentance_2017 %}), this intervention is used to good
results. Their model, PRIMM ({% cite Sentance_2017 %}), starts with a good
mental model which is required to predict, tracing during investigation, and
debugging to modify code, all building towards students making things
themselves.

[Here are some examples](https://primmportal.com/) of how to implement the PRIMM methodology in exercises.


topic: contributing
tutorial: create-new-topic
title: Including a new topic
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: aafe68848924f2601184f8e6a129cea8
last_modified: 2026-02-26T21:12:14.701808
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-topic/tutorial.html

content:
Each training material is related to a topic. All training materials (slides, tutorials, ...) related to a topic are found in a dedicated directory (*e.g.* `transcriptomics` directory contains the material related to exome sequencing analysis).

## Directory structure

Each topic has the following structure:

```
├── README.md
├── metadata.yaml
├── images
├── docker
│   ├── Dockerfile
├── slides
│   ├── index.html
├── tutorials
│   ├── tutorial1
│   │   ├── tutorial.md
│   │   ├── slides.html
│   │   ├── data-library.yaml
│   │   ├── workflows
│   │   │   ├── index.md
│   │   │   ├── workflow.ga
│   │   ├── tours
│   │   │   ├── tour.yaml
```


### `images` directory

The `images` directory collects all images/pictures needed for the training materials related to the topic, *i.e* pictures for the slides or the tutorials.

Images shared between several topics are in the `shared/images` directory at the root.

All images for the slides must be in `images` directory. The images must be in good quality. The sources (`svg` or other) of the images must also be added to the `images` directory. We encourage you to use [yEd](https://www.yworks.com/products/yed) to easily generate diagrams and [Inkscape](https://inkscape.org/en/) for any other images.


### `tutorials` directory

This directory collects the materials related to the topic, one material per subdirectory, though it may consist of slides, tutorials, or both.

The tutorials are hands-on built for workshop and self-training, with description of the whole infrastructure needed to run the tutorial on any Galaxy instance (tools, data library, etc).

The templates for the tutorials are different from the other pages to help users to focus on the content of the tutorial. To improve the output of the tutorial, several metadata are mandatory for every tutorials, such as the requirements or the objectives of the tutorials. Boxes are also used to highlight some key points as the hands-on or the tips.

The content of each tutorial is generated with [Jekyll](https://jekyllrb.com/) from a Markdown file and some metadata (e.g. the requirements, the Zenodo link, the questions) defined inside the metadata of the related topic.

> <comment-title>Contributing</comment-title>
> Want to contribute to a tutorial? Check out [our training materials about that]({% link topics/contributing/index.md %}).
{: .comment}

Sometimes, an hands-on tutorial is not the most appropriate format for a tutorial and slides are better.

### `docker` directory


For each topic, a flavored Docker image must integrate the tools needed for
the tutorials. The corresponding image must be based on official Galaxy Docker
images.

The `docker` image will also integrate the Galaxy tours available for each topics and the workflows.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

There are two kinds of topics we will cover: topics with their own materials, and synthetic topics that are based solely on a tag. For large collections of materials around a single central theme a topic with materials is a good choice. However, sometimes you will want to highlight materials spread across numerous or disparate topics like e.g. Cancer or SARS-CoV-2 tutorials where you might have individual tutorials on assembly or transcriptomics, but you'd like a single page listing all of them.

# Creating a new topic with it's own materials

## Defining the topic

When we structured the repository, we decided to use as topics the categories that are used in the [ToolShed](https://toolshed.g2.bx.psu.edu/). The ToolShed assigns a category to each tool. Therefore, to decide where to put your tutorial, have a look at which ToolShed's category the main tools in your tutorial belong. For example, this tutorial will rely on the NCBI Blast+ tool.

> <hands-on-title>Defining the topic for the tutorial</hands-on-title>
>
> 1. Search for NCBI Blast+ on the [ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. Check in which category it has been placed
>
>    > <question-title></question-title>
>    >
>    > In which topic will you put the new tutorial?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > If we search for [NCBI Blast+ in the ToolShed](https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/7538e2bfcd41), it is placed in 2 categories (bottom): "Next Gen Mappers", and "Sequence Analysis".
>    > > We decided to put it in "Sequence analysis" because this is the most general one for this tutorial.
>    > {: .solution}
>    {: .question}
{: .hands_on}

In this tutorial, we want to add a new topic called about "my-favorite-topic".

## Creating the skeleton for the topic

Once the topic name has been chosen, we can create it.

> <hands-on-title>Create all the required files and folders structures automatically</hands-on-title>
>
> 1. Open a terminal
> 2. Run (by adapting the information between the quotes)
>
>    ```
>    $ planemo training_init \
>             --topic_name "my-favorite-topic" \
>             --topic_title "Test" \
>             --topic_target "use" \
>             --topic_summary "Summary of the topic"
>    ```
>
> 3. Check that a new directory has been generated in `topics`
> 4. Check that a YAML file with your topic name has been generated in `metadata` folder
> 5. Make sure that Jekyll is running
>
>    > <comment-title>Jekyll</comment-title>
>    > Want to learn how to start Jekyll? [Check out our tutorial to serve the website locally]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
>    {: .comment}
>
> 6. Check if the topic has been correctly added at [http://localhost:4000/training-material/](http://localhost:4000/training-material/)
>
{: .hands_on}

## Adapt the metadata for your topic

Several metadata are defined in `metadata.yaml` file in your topic folder to :

{% assign kid_key = "Topic Schema" %}
{% assign kid_val = site.data['schema-topic'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

> <hands-on-title>Update the new topic to the website</hands-on-title>
>
> 1. Open the `metadata.yaml` file in your topic folder
> 2. Fill the correct metadata of the topic
> 3. Make sure that Jekyll is running
>
>    > <comment-title>Jekyll</comment-title>
>    > Want to learn how to start Jekyll? [Check out our tutorial to serve the website locally]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
>    {: .comment}
>
> 4. Check how it changes the local website
>
{: .hands_on}

### Subtopics

For large topics, we can define subtopics and create multiple tutorial lists, within a topic's `metadata.yaml`

```yaml
name: transcriptomics
type: use
title: Transcriptomics
summary: Training material for all kinds of transcriptomics analysis.
docker_image: "quay.io/galaxy/transcriptomics-training"

[...]

subtopics:
- id: introduction
  title: "Introduction"
  description: "Start here if you are new to RNA-Seq analysis in Galaxy"
- id: end-to-end
  title: "End-to-End Analysis"
  description: "These tutorials take you from raw sequencing reads to pathway analysis"
- id: visualisation
  title: "Visualisation"
  description: "Tutorials covering data visualisation"
  crosstopic_tutorials:  ## optional
    - topic: microbiome
      tutorial: beer-data-analysis
      priority: 2  ## optional
```

Each subtopic has:
- an ID, used as a reference in the tutorial
- a short descriptive title
- a longer description discussion what is contained in that subtopic

Each subtopic **optionally** may define:
- tutorials from other topics to show, via the `crosstopic_tutorials` metadata

Tutorials can be assigned to subtopics by adding e.g. `subtopic: singlecell` to the tutorial metadata. An example of this subtopic division can be found in the [admin section]({{site.baseurl}}/topics/admin/ )


# Creating a tag based topic

For tag based topics, first ensure that all of the relevant tutorials share a single tag, across all materials that should be included in this view.

## Defining the topic

Compare with other topic level metadata files in the `metadata/` directory of the training material. Then create a file of your own naming the topic and providing editorial board members and so on. We will use the Covid-19 synthetic topic as an example:

```yaml
---
name: "covid19"
type: "use"
title: "SARS-CoV-2"
summary: "Tutorials covering analysis of SARS-CoV-2 (COVID 19)"
tag_based: true

editorial_board:
  - wm75

gitter: galaxyproject/sars-cov-2
```

As you can see it is very short, and there are only a handful of important points:

- `tag_based` must be set to `true`
- `name` must be equal to the tag name, that's being used across tutorials.
- `subtopics` must not be set.

With this done, all materials tagged `covid19` will be aggregated and available under this synthetic topic. They are organised by the "parent" topic, so e.g. assembly tutorials are collected together and transcriptomics tutorials are also in a section together, similar to how other tutorials define subtopics.

## More manual tag based topics

If you wish to more manually curate the tutorials listed in your tag-based topic, you can provide subtopics. These must have IDs that match tags that are expected to be found within your tag-based topic.

```yaml
---
name: gmod
type: use
topic_type: technology
title: GMOD
edam_ontology: ["topic_0362"]

tag_based: true

subtopics:
  - id: jbrowse1
    title: "JBrowse 1"
    description: Learn to use JBrowse v1 here.
  - id: apollo2
    title: Apollo v2
    description: Apollo is a collaborative, real time genome annotation editor.
```

Here anything with a tag `jbrowse1` will appear under the JBrowse section, same for `apollo2` and the Apollo section. Tutorials may appear multiple times, but this is still useful as it gives you a way to provide additional organisation within your topic.


# Conclusion

We just created a new topic. We can now fill it by [creating new tutorials]({% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %})


topic: contributing
tutorial: create-new-tutorial-quiz
title: Adding Quizzes to your Tutorial
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 001f8a6acdd0d536bafae961602dd304
last_modified: 2026-02-26T21:12:14.703296
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-quiz/tutorial.html

content:
Interactive quizzes can be used either alone, or with a classroom of students, to check student's knowledge.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## How it Works

We developed a small Kahoot-like interface, where a teacher can initiate a quiz, students can join the teacher's session (using PeerJS), and do a collaborative quiz.

## Quiz Format

Each quiz starts a lot like the tutorials!

```yaml
title: SQL Advanced Recap
contributors:
- hexylena
```

Within the questions section we have a list of question

```yaml
questions:
- title: How do you find the number of records in a query result?
  answers:
    - count(name)
    - count(*)
    - sum(id)
    - max(id)
  correct:
    - count(name)
    - count(*)
  timeout: 20
  type: choose-many
```

There are a few different types of questions:

- `choose-1`, one correct answer
- `choose-many`, potentially multiple correct answers
- `poll`, there's no right answer, just ask the students how they're feeling!


You can include images in the main area, if you need some context for a question

```yaml
- title: Which of these joins is NOT valid
  image: /training-material/topics/data-science/images/carpentries-sql/sql-join-structure.svg
  answers:
    - Select * From P as P1 Join P as P2 on P1.id = P2.id
    - SELECT * From P Join Q Join V on P.id = Q.person and Q.taken = V.id
    - SELECT * From S join Q on S.name=V.dated and V.site = Q.quant
    - SELECT * From S Join V on S.name = V.site
  correct: SELECT * From S join Q on S.name=V.dated and V.site = Q.quant
  timeout: 60
  type: choose-1
```

Poll's let you check in with students, how they're feeling, or take their opinion on what they think.

```yaml
- title: How are you feeling?
  answers:
    - Great
    - Horrid
  timeout: 20
  type: poll
  live: true
```

The `live` key there allows polls to show the results "live" and let students change answers while the time runs.

## Folder Structure

You place quizzes in a `quiz` subdirectory of your tutorial

```console
.
├── quiz
│   └── a.yaml
└── tutorial.md
```

## Inserting quizzes throughout a tutorial

If you want to show a quiz at a specific place within a tutorial

```markdown
{% raw %}
{% include _includes/quiz.html id="a.yaml" %}
{% endraw %}
```

{% include _includes/quiz.html id="a.yaml" %}

And then it's available!

## Default Location

By default all quizzes are found at the bottom of the tutorial.


topic: contributing
tutorial: create-new-tutorial
title: Creating a new tutorial
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 060bfe2908e33ae383ed099b8a77854e
last_modified: 2026-02-26T21:12:14.704605
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial/tutorial.html

content:
Galaxy is a great solution to train bioinformatics concepts:

- numerous bioinformatics tools are available (over 8,000 in the ToolShed)
- it can be used by people without any computer science skills
- it trains to use technology, outlining available resources and efforts that have made them accessible to researchers
- it is scalable

In 2016, the Galaxy Training Network decided to set up a new infrastructure for easily delivering Galaxy-related training material. The idea was to develop something open, online, based on a community effort, and on top of the Galaxy platform.

We took inspiration from [Software Carpentry](https://software-carpentry.org) and collected everything on a GitHub repository: [https://github.com/galaxyproject/training-material](https://github.com/galaxyproject/training-material).
We decided on a structure focusing on tutorials with hands-on activities; fitting both for online self-training but also for workshops. Each tutorial follows the same structure and comes with a virtualised instance, allowing you to run the training anywhere you have resources available.

Here you will learn how to create a new tutorial by developing a small tutorial explaining how to retrieve climate data from Copernicus (using **Copernicus Climate Data Store** tool).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <comment-title></comment-title>
> This tutorial explains the different steps to create a tutorial for the Galaxy Training Material.
> It may require some knowledge that you may not have or do not have the time to learn. If this is the case, you can create a skeleton of a tutorial with whatever existing materials you have, using your prefered text editor, and then share it with us by opening [issue on GitHub]({{ site.github_repository }}/issues/new), writing us on [Gitter]({{ site.gitter_url }}), or sending us an [email](mailto:{{ site.email }}).
{: .comment}

# Define the topic

The first step we need to do is to identify in which topic to place our new tutorial. This can be tricky. When we structured the GTN, we decided that each training material should be related to a topic.

We decided to use the categories from the [ToolShed](https://toolshed.g2.bx.psu.edu/) as our initial list of topics. Since every tool uploaded to the ToolShed must be in at least one category, you can look at the main tools in your tutorial and see which categories they are placed in within the ToolShed. This can provide a guide for where you might put your new tutorial. For example, if your tutorial will rely on the **Copernicus** tool:

> <hands-on-title>Defining the topic for the tutorial</hands-on-title>
>
> 1. Search for **Copernicus** on the [ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. Check in which category it has been placed
>
> > <solution-title></solution-title>
> > There are a couple of steps to reaching the answer:
> > 1. Search for `Copernicus`
> > 2. Press the <kbd>Enter</kbd> key to search
> > 3. Click on the result named `c3s`
> > 4. At the bottom of this page, there is a box labelled "Categories"
> >
> > It is placed in one category: "Climate Analysis"
> {: .solution}
>
> 3. Look if the categories fit to existing topics in the Galaxy Training Material
{: .hands_on}

> <comment-title>No fitting topic for the tools in your tutorial?</comment-title>
> If the categories in ToolShed do not fit to any existing topics, we recommend to use your better judgment to identify in which topic your tutorial should go. You can also ask us on [Gitter]({{ site.gitter_url }}) or raise an issue on [GitHub]({{ site.github_repository }}) explaining the aim of the tutorial. We will be happy to help you there.
{: .comment}

> <comment-title>Creating a new topic</comment-title>
> Want to create a new topic? [Check out our tutorial to create a new topic]({% link topics/contributing/tutorials/create-new-topic/tutorial.md %})
{: .comment}

# Store a tutorial

All training materials (slides, tutorials, ...) related to a topic are found in a dedicated directory (*e.g.* `climate` directory contains the material related to Climate analyses). Each topic has the following structure:

```
├── README.md
├── metadata.yaml
├── images
├── docker
│   ├── Dockerfile
├── slides
│   ├── index.html
├── tutorials
│   ├── tutorial1
│   │   ├── tutorial.md
│   │   ├── slides.html
│   │   ├── data-library.yaml
│   │   ├── workflows
│   │   │   ├── workflow.ga
│   │   ├── tours
│   │   │   ├── tour.yaml
```

Once the topic has been chosen, we can create the tutorial. An ideal tutorial in the Galaxy Training Network contains:
- a tutorial file `tutorial.md` written in Markdown with hands-on
- an optional slides file `slides.html` in Markdown with slides to support the tutorial
- a directory `workflows` with workflows extracted from the tutorial
- a YAML file `data-library.yaml`  with the links to the input data needed for the tutorial

The most important file is the `tutorial.md`, where the content of the tutorial is. The other files are not mandatory. They are there to support the tutorial and make it robust and usable across many environments. But they can help a lot in the development of a new tutorial.
The `data-library.yaml` is used by Galaxy servers to create a *Shared Data Library* containing your tutorial data, allowing you quicker access during training courses. If you have defined a `zenodo_link` in your tutorial metadata, then the `data-library.yaml` file will be automatically added by one of the GTN GitHub Actions if it is missing, so there is no need to add it yourself. 

# Create the workflow

Our tutorials try to follow the "learn by doing" approach; they combine both theoretical and practical sections done on Galaxy.

Most tutorials explain step-by-step a data analysis by running the tools. The steps taken in the tutorial can be represented as a workflow.

Before writing the tutorial, it is a good practice to get a workflow with the different steps that will be run during the tutorial. The workflow does not have to be the final one, but at least the major steps should be included. It helps to get a direction for the tutorial, but also to generate a skeleton of the tutorial, as we will see later.

> <comment-title>Use tools that are available on the ToolShed</comment-title>
> We recommend using tools in your workflows, especially for training, that are available on the Galaxy ToolShed.
{: .comment}


> <hands-on-title>Prepare the workflow for the training</hands-on-title>
>
> 1. Go to your favorite Galaxy server
> 2. Create a workflow with the different steps (tools) of your tutorial either from scratch or from an existing history
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
>    {% snippet faqs/galaxy/workflows_extract_from_history.md %}
>
>    For the Copernicus tutorial:
>    1. Add an **Input Dataset**
>    2. Add **Copernicus Climate Data Store**
>    3. Link the **Input Dataset** to **Copernicus Climate Data Store** on `API Request filename`
>    4. Rename **Input Dataset** to `API Request file`
>    5. Save the workflow
>
> 3. Add the topic name as a Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor
>
>    {% snippet faqs/galaxy/workflows_annotate.md %}
>
>    For the Copernicus tutorial:
>    - **Annotation**: `Retrieve climate data from Copernicus`
>    - **Tag**: `climate`
>
> 4. Make the workflow accessible (publishing is not necessary)
>
>    {% snippet faqs/galaxy/workflows_publish.md %}
>
{: .hands_on}


# Get a toy dataset

To run the different steps, the tutorial needs some data. The selected data must be informative enough to illustrate the meaning of using a tool or a given technique, but not too big to require long waiting times for processing during a workshop. Upload and download of files into and out of Galaxy is usually quick, but the time taken for a tool to run can be long. Tool run times of no more than 10-15 mins are recommended. Typically, the selected data should be the informative subset of a full real-life dataset.

Below we describe two examples of how toy datasets were generated for tutorials:

- **Example 1**: creating a toy dataset from scratch
  - Take one 16S sequence (for example found in the test case of a Galaxy tool):
  - Generate a reference database
      - Blast it on the NR database on [NCBI Blast](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastn&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome)
      - Extracting one similar sequence found with Blast
      - Search and extract 2 other sequences of the same species using the [NCBI Nucleotide database](https://www.ncbi.nlm.nih.gov/nuccore)

- **Example 2**: creating a toy dataset from an existing larger one
  - When the experiment takes a FASTQ as input and a few reads are sufficient:
    - Use **seqtk_sample** {% icon tool %} to extract randomly reads from your input fastq.
  - However, when it requires a lot of reads to be meaningful, you can use the following strategy (used for the ATAC-seq tutorial using [this workflow](./workflows/Galaxy-Workflow-MakeAFakeInput.ga)):
    - Run the workflow until the mapping step on the full dataset (or big enough to have good results).
    - Select IDs of reads which map on the smallest chromosome (for example chr22 for human data).
    - In order to keep in the toy dataset enough diversity, you can also take randomly 1% of the reads IDs.
    - Concatenate the two lists and remove the duplicated IDs.
    - Use **seqtk_subseq** {% icon tool %} to sample your original FASTQ with the list of IDs.

We would then develop the tutorial and test it on this toy dataset. Once we were ready to share it, we would upload the datasets to [Zenodo](https://zenodo.org/) to store them on long-term and obtain a dedicated DOI in the [Galaxy training network community](https://zenodo.org/communities/galaxy-training/?page=1&size=20).

> <hands-on-title>Upload the dataset to Zenodo</hands-on-title>
>
> 1. Go to [Zenodo](https://zenodo.org/)
> 2. Log in using your GitHub credentials
>
>    You may need to authorize Zenodo to access your GitHub account (only to read your information)
>
> 3. Click on **Upload** (top panel)
> 4. Start a new upload
> 5. Upload the files corresponding to your datasets
>
>     > <comment-title>No possible changes in the files after publication</comment-title>
>     > File addition, removal or modification is not allowed after you have published your upload.
>     > So be careful when you start your upload that all the files you need are ready.
>     >
>     > The metadata can be changed after publication.
>     {: .comment}
>
> 6. Search for and Select *Galaxy Training Network* in **Communities**
> 7. Select *Dataset* in **Upload type**
> 8. Use the title of your tutorial and mention also Galaxy Training Material
> 9. Add all the persons who contributed to the tutorial as authors
> 10. Add a short description of the tutorial and a link to the training material website
> 11. Keep *Open Access* as **Access right** and *Creative Commons Attribution 4.0* as **License**
> 12. Fill out any remaining information
> 13. Click on **Save**
> 14. Click on **Publish**
> 15. Copy the DOI link in the new page
{: .hands_on}

# Create the skeleton of the tutorial

Once we have the workflows and toy dataset (optionally already on Zenodo), we can generate the tutorial folder, including the `tutorial.md`.

Writing the tutorial while adding the different tools and their parameters, and following good formatting, can be quite cumbersome. To help in that process, we developed a training development kit available with [Planemo](https://planemo.readthedocs.io/en/latest/index.html). The tool has several commands. But the main one can be used to create the skeleton of a tutorial (folder, well-located, with good structure). It can also take a workflow as input and add it to the `workflows` folder. But more interestingly, the tool can extract from the workflow and add in the `tutorial.md` file the different steps with which tools to run and which parameters to select. If a Zenodo URL is provided, it also creates the `data-library.yaml` file.

This tool can be used via the command-line but also via a [webserver](https://ptdk.apps.galaxyproject.eu/). The webserver can take a public workflow that is available on [usegalaxy.org](https://usegalaxy.org), [usegalaxy.eu](https://usegalaxy.eu) or [usegalaxy.org.au](https://usegalaxy.org.au).

> <hands-on-title>Create the skeleton of a tutorial using the webserver</hands-on-title>
>
> 1. Make the workflow public
>
>    {% snippet faqs/galaxy/workflows_publish.md %}
>
> 2. Copy the workflow ID that can be found in the URL of the current page (after `?id=`)
> 3. Open the [PTDK webserver](https://ptdk.apps.galaxyproject.eu/)
> 4. Fill in the information
>    - Tutorial name (the name will be the name of the folder of the tutorial)
>    - Tutorial title
>    - Galaxy instance with the public workflow
>    - Id of the workflow
>    - (Not mandatory) Zenodo URL with the input data
>
>    For the Copernicus tutorial:
>    - **Tutorial name** (the name will be the name of the folder of the tutorial): `climate-data-retrieval-copernicus`
>    - **Tutorial title**: `Retrieve climate data from Copernicus`
>    - **Galaxy instance with the public workflow**: `usegalaxy.eu`
>    - **Id of the workflow**: `ac5b66c42681e7a8`
>
> 5. Click on **Submit**
> 6. Download the generated archive
>
>    This archive contains the tutorial skeleton, including:
>    - tutorial content, `tutorial.md` file, filled with all steps from the workflow and their parameters
>    - its workflow (`workflow` folder)
>    - a `data_library.yaml` file if Zenodo link was provided
>
> 7. Add the new material to Galaxy Training Material by unzipping the downloaded archive in the tutorials folder of the topic for the new tutorial
>
>    > <comment-title>Using the GitHub interface</comment-title>
>    >
>    > Prefer to use the GitHub interface?
>    >
>    > 1. Unzip the downloaded archive
>    > 2. Edit the content of the `tutorial.md` (as explained below)
>    > 3. Go to the GitHub repository of the Training Material
>    > 4. Fork the GitHub repository
>    > 5. Click on `topics`
>    > 6. Select the topic for the new tutorial
>    > 7. Go to `tutorials`
>    > 8. Click on **Create new file**
>    > 9. Type `name/tutorial.md`, replacing "name" by the name of your tutorial (not the title)
>    > 10. Copy the content of the downloaded and edited `tutorial.md` file there
>    > 11. Fill the **Commit new file** form
>    > 12. Create a new branch using the name of the tutorial
>    > 13. Click on **Propose new file**
>    > 14. Open a Pull Request, as explained in our tutorial
>    > 15. Add the workflow file (in `workflow` folder) by updating the Pull Request
>    {: .comment}
>
{: .hands_on}

If the workflow is not available on one of the previously listed Galaxy servers, we recommend that you run the tool via the command line:

> <hands-on-title>Create the skeleton of a tutorial via the command line</hands-on-title>
>
> 1. Get the workflow ID
>
>    The ID can be found on the URL when running, editing or sharing the workflow (after `?id=`)
>
> 2. Get your API key on the Galaxy instance
>
>    {% snippet faqs/galaxy/preferences_admin_api_key.md %}
>
> 3. (If not done yet) Get the Galaxy Training Material repository locally and move into it
>
>     1. (If not done yet) Clone the training material GitHub repository: `git clone https://github.com/galaxyproject/training-material.git`
>     2. Navigate to the `training-material/` folder with `cd`
>
> 4. Get planemo
>
>     - Option 1: Using Conda:
>       1. Set up the conda environment
>
>          It will install some needed tools (ruby, nodejs, etc) in a protected environment, without interfering with the existing tools or versions.
>
>          1. Install conda (if not already installed): `make install-conda`
>          2. (You may need to exit the terminal and re-open for conda to be recognised. Navigate back to the same place.)
>          3. Create the `galaxy_training_material` conda environment: `make create-env`
>
>       2. Activate the conda environment with `conda activate galaxy_training_material`
>
>     -  Option 2: Using pip by running `pip install planemo`
>
> 6. Generate the skeleton of your tutorial (by adapting the information between the quotes)
>
>      ```
>      $ planemo training_init \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --tutorial_title "Title of the tutorial" \
>             --galaxy_url "URL to Galaxy instance in which you created the workflow" \
>             --galaxy_api_key "Your API key on the Galaxy instance" \
>             --workflow_id "ID of the workflow on the Galaxy instance" \
>             --zenodo_link "URL to the Zenodo record"
>      ```
>
> 4. Check that a new directory (with your tutorial name) has been generated in the topic folder
{: .hands_on}

# Write the tutorial

Now that you have the structure in place, you can then fill the tutorial, per se.

> <hands-on-title>Write the tutorial</hands-on-title>
>
> 1. Open the `tutorial.md` file with your favorite text editor
> 2. Fill out the tutorial by following the [dedicated tutorial]({% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md %})
>
>    1. Add metadata at the top of the tutorial
>    2. Add a proper introduction
>    3. Organize the tutorial into different sections
>    4. Introduce the different concepts and the different steps of the analysis
>    5. Check the hands-on boxes
>    6. Comment on the outputs of the different steps
>    7. Add some question/solution boxes for assessment
>    8. Add a conclusion
>
> 3. (Optional) Build the website locally and check that the tutorial is there by following the [Jekyll tutorial]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
{: .hands_on}

# Keep track of the changes

The material is stored in a [GitHub repository]({{ site.github_repository }}), a code hosting platform for version control and collaboration. So to develop training material, we are following the [GitHub flow](https://guides.github.com/introduction/flow/), which is based on fork, branches, and pull requests.

This can be done online via the GitHub interface or locally on your computer via the command line.

> <comment-title>Learning how to contribute</comment-title>
> Want to learn how to contribute? Check our tutorials:
> - [Contributing with GitHub via its interface]({% link topics/contributing/tutorials/github-interface-contribution/tutorial.md %})
> - [Contributing with GitHub via command-line]({% link topics/contributing/tutorials/github-command-line-contribution/tutorial.md %})
{: .comment}

# Add slides (optional)

Sometimes, you may want to have slides to support a tutorial and introduce it during a workshop. Sometimes, a set of slides is better than a tutorial to cover a specific topic.

> <hands-on-title>Add slides</hands-on-title>
>
> 1. Create a slide deck in `slides.html` following the [Slide tutorial]({% link topics/contributing/tutorials/create-new-tutorial-slides/slides.html %})
{: .hands_on}

# Conclusion


To develop a new tutorial:

1. Determine the topic
2. Create a workflow
3. Find a good toy dataset and upload it to Zenodo
4. Create the skeleton for the tutorial
5. Add the skeleton to the training material
6. Write the tutorial
7. Keep track of the changes
8. Add slides (optional)
9. Submit as a Pull Request to GitHub

For the next time, you can make it quicker.

> <hands-on-title>Generation of a tutorial</hands-on-title>
>
> 1. Determine the topic
> 2. Create your workflow on a running Galaxy instance
> 3. Add the topic name as a Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor
> 4. Create a Zenodo record with the input data
> 5. Generate the skeleton of your tutorial
>
>    - Option 1: from the [PTDK webserver](https://ptdk.apps.galaxyproject.eu/) and get the skeleton in the training material
>    - Option 2: from a workflow located on a Galaxy
>
>      ```
>      $ planemo training_init \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --tutorial_title "Title of the tutorial" \
>             --galaxy_url "URL to Galaxy instance in which you created the workflow" \
>             --galaxy_api_key "Your API key on the Galaxy instance" \
>             --workflow_id "ID of the workflow on the Galaxy instance" \
>             --zenodo_link "URL to the Zenodo record"
>      ```
>
> 6. Fill the remaining metadata in the `tutorial.md`
> 7. Fill the content of the `tutorial.md`
> 8. Check it by serving the website locally
>
>    > <comment-title>Serving the website locally</comment-title>
>    > Want to learn how to see the change on the website locally? [Check out our dedicated tutorial]({% link topics/contributing/tutorials/rendering_gtn/tutorial.md %})
>    {: .comment}
>
> 9. Submit it to GitHub
{: .hands_on}


topic: contributing
tutorial: generating-pdf
title: Generating PDF artefacts of the website
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: []
tags_json: []
content_hash: b15be4a691bd45c9d26ca62ec45bec8e
last_modified: 2026-02-26T21:12:14.708731
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/generating-pdf/tutorial.html

content:
The website with the training material can be run locally. Sometimes, it is also interesting to freeze the tutorials or to get PDFs of the tutorials.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to run a local instance of the GTN website:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Generate PDFs artifact

To generate the PDFs, a command `make pdf` is given. This command:

- Launches a detached Jekyll server to serve the website
- Generates the PDFs of the tutorials by calling Chrome via command line
- Generates the PDFs of the slide decks by calling decktape

> <hands-on-title>Checking the website generation locally</hands-on-title>
>
> 1. Install a browser
> 2. Generate the PDFs: `make pdf`
> 3. Check the generated PDFs in `_pdf` folder
{: .hands_on}


topic: contributing
tutorial: slides-with-video
title: Adding auto-generated video to your slides
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 4885eb013706d585a4450553dc371e45
last_modified: 2026-02-26T21:12:14.723860
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/slides-with-video/tutorial.html

content:
# Video Lectures


Based on the work by Delphine Larivière and James Taylor with their [COVID-19 Lectures](https://github.com/galaxyproject/video-lectures/) we have implemented a similar feature in the Galaxy Training Network.

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## How it Works

We wrote a short script which does the following:

*Locally and in production*:

- Extracts a 'script' from the slides. We extract every presenter comment in the slidedeck, and turn this into a text file.
- Every line of this text file is then narrated by [Amazon Polly](https://aws.amazon.com/polly/) (if you have money) or [MozillaTTS](https://github.com/synesthesiam/docker-mozillatts) (free).
- The slide deck is converted to a PDF, and then each slide is extracted as a PNG.
- Captions are extracted from the audio components.
- The narration is stitched together into an mp3
- The images are stitched together into an mp4 file
- The video, audio, and captions are muxed together into a final mp4 file

*In production*

- We use Amazon Polly, paid for by the Galaxyproject
- The result is uploaded to an S3 bucket

# Enabling Video

We have attempted to simplify this process as much as possible, but making good slides which work well is up to you.

## Writing Good Captions

Every slide must have some narration in the presenter notes. It does not make sense for students to see a slide without commentary. For each slide, you'll need to write presenter notes in full, but short sentences.

### Sentence Structure

Use simple and uncomplex sentences whenever possible. Break up ideas into easy to digest bits. Students will be listening to this spoken and possibly reading the captions.

*2021-05-01* There used to be a limit of ~120 characters per sentence, but this is no longer an issue. We now break up sentences which are too long in the captions and show them over multiple timepoints. So if you need to write a really long sentence, you can, but we still advise to simplify sentences where possible.

### Captions per Slide

Every slide must have some speaker notes in this system, **NO exceptions**.

### Punctuation

Sentences should end with punctuation like `.` or `?` or even `!` if you're feeling excited.

### Abbreviations

These are generally fine as-is. (e.g. `e.g.`/`i.e.` is fine as-is, `RNA` is fine, etc.) Make sure abbreviations are all caps though.

> **Good**
> This role deploys CVMFS.
{: .code-in}

### "Weird" Names

In the captions you will want to teach the GTN how to pronounce these words by editing `bin/ari-map.yml` to provide your definition.

E.g.

Word       | Pronunciation
---------- | ---
SQLAlchemy | SQL alchemy
FastQC     | fast QC
nginx      | engine X
gxadmin    | GX admin
/etc       | / E T C

The same applies to the many terms we read differently from how they are written, e.g. 'src' vs 'source'. Most of us would pronounce it like the latter, even though it isn't spelt that way. Our speaking robot doesn't know what we mean, so we need to spell it out properly.

So we write the definition in the [`bin/ari-map.yml`](https://github.com/galaxyproject/training-material/blob/master/bin/ari-map.yml) file.

### Other Considerations

(*Written 2020-12-16, things may have changed since.*)

Be sure to check the pronunciation of the slides. There are known issues with [heteronyms](https://en.wikipedia.org/wiki/Heteronym_(linguistics)), words spelt the same but having different pronunciation and meaning. Consider "read" for a classic example, or ["analyses"](https://en.wiktionary.org/wiki/analyses#English) for one that comes up often in the GTN. "She analyses data" and "Multiple analyses" are pronounced quite differently based on their usage in sentences. See the [wiktionary](https://en.wiktionary.org/wiki/analyses#English) page for more information, or the [list of English heteronyms](https://en.wiktionary.org/wiki/Category:English_heteronyms) you might want to be aware of.

This becomes an issue for AWS Polly and Mozilla's TTS which both don't have sufficient context sometimes to choose between the two pronunciations. You'll find that "many analyses" is pronounced correctly while "multiple analyses" isn't.

Oftentimes the services don't understand part of speech, so by adding adjectives to analyses, you confuse the engine in to thinking it should be the third person singular pronunciation. This is probably because it only has one or two words of context ahead of the word to be pronounced.

## Enable the Video

Lastly, we need to tell the GTN framework we would like videos to be generated.

> <hands-on-title>Enable video</hands-on-title>
>
> 1. Edit the `slides.html` for your tutorial
> 2. Add `video: true` to the top
{: .hands_on}

That's it! With this, videos can be automatically generated.


# Voices

There are multiple voices available, see the following list:

Name    | Region    | Neural | Audio Clip
---     | ------    | ----   | ----
Amy     | en-GB     | True   | <audio controls src="{% link assets/audio/amy.mp3 %}"></audio>
Aria    | en-NZ     | True   | <audio controls src="{% link assets/audio/aria.mp3 %}"></audio>
Brian   | en-GB     | True   | <audio controls src="{% link assets/audio/brian.mp3 %}"></audio>
Emma    | en-GB     | True   | <audio controls src="{% link assets/audio/emma.mp3 %}"></audio>
Joanna  | en-US     | True   | <audio controls src="{% link assets/audio/joanna.mp3 %}"></audio>
Joey    | en-US     | True   | <audio controls src="{% link assets/audio/joey.mp3 %}"></audio>
Kendra  | en-US     | True   | <audio controls src="{% link assets/audio/kendra.mp3 %}"></audio>
Matthew | en-US     | True   | <audio controls src="{% link assets/audio/matthew.mp3 %}"></audio>
Nicole  | en-AU     | False  | <audio controls src="{% link assets/audio/nicole.mp3 %}"></audio>
Olivia  | en-AU     | True   | <audio controls src="{% link assets/audio/olivia.mp3 %}"></audio>
Raveena | en-IN     | False  | <audio controls src="{% link assets/audio/raveena.mp3 %}"></audio>
Salli   | en-US     | True   | <audio controls src="{% link assets/audio/salli.mp3 %}"></audio>
Ayanda  | en-ZA     | True   | <audio controls src="{% link assets/audio/ayanda.mp3 %}"></audio>
Geraint | en-GB-WLS | True   | <audio controls src="{% link assets/audio/geraint.mp3 %}"></audio>

By default a random voice is chosen every time the video is rebuilt (only
whenever a change is made to that slide deck.) We do this to ensure a good
diversity of genders and nationalities in the audio samples.

However, if you have a preferred voice, you can set that permanently for that
video, add the following metadata to the top of your slide deck:

```
voice:
  id: Lupe
  lang: es-US
  neural: true
```

The above voice example is specific to Spanish language content, hence not being represented in the first list.

# How it works: In Detail

1. We take our markdown slides, e.g. [`topics/introduction/tutorials/galaxy-intro-short/slides.html`](https://github.com/galaxyproject/training-material/blob/main/topics/introduction/tutorials/galaxy-intro-short/slides.html)
2. In order for them to be processed, slides must have an annotation saying `video: true` in the header metadata, and then 'speaker notes' (everything after the ??? before the ---)
3. This is turned into our 'plain text slides' which just renders the markdown a bit more nicely ([example]({{site.baseurl}}/topics/introduction/tutorials/galaxy-intro-short/slides-plain.html))
4. Then we run ari.sh which does the following:

	- `make video` is run which runs [`bin/ari-make.sh`](https://github.com/galaxyproject/training-material/blob/main/bin/ari-make.sh)
	- This builds PDFs for any slides which have changed
	- And runs `./bin/ari.sh` with the PDF, the original Slides, and where the mp4 should be saved.
		- In [`./bin/ari.sh`](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh)
		- It [extracts metadata](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L38) from the tutorial (title, authors, etc.)
		- It [builds a 'script'](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L51), a json document with blocks for every line of the speaker notes that were in the slides.
		- [Those get converted into mp3 files](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L55) by AWS Polly (or MozillaTTS), one per slide.
		- The PDFs get turned into [a series of PNG images](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L60)
		- We take the timings of the mp3 files together with the json 'script' to [write out webvtt / srt subtitles](https://github.com/galaxyproject/training-material/blob/main/bin/ari.sh#L69) which get embedded into the video, and supplied next to it.
		- [editly is used](https://github.com/mifi/editly) to knit together the PNGs + mp3s with appropriate delay


All of this is run on cron by [`.github/workflows/video.yml`](https://github.com/galaxyproject/training-material/blob/main/.github/workflows/video.yml) which handles building all of these videos and then later uploading them to s3.

Many of the scripts internally are prefixed with `ari`, we named our internal version after [github.com/jhudsl/ari/](https://github.com/jhudsl/ari/) which inspired it, but we wanted a version that would be more closely tied to the GTN and integrate with our infrastructure nicely, so we ended up writing our own.

# Conclusion


topic: contributing
tutorial: updating_tutorial
title: Updating tool versions in a tutorial
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: []
content_hash: 8fc57b06bd7f92f8c9c844fdc2c14db6
last_modified: 2026-02-26T21:12:14.724080
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/updating_tutorial/tutorial.html

content:
Here, we provide a clear set of instructions for updating a GTN tutorial that uses the Galaxy interface for performing analysis. The GTN, and Galaxy, have a number of features to make it as reproducible and shareable as possible, so navigating what needs to be done for an overall update may feel daunting - but not anymore!

We encourage you to pick a tutorial to try this on, so you can use this tutorial side-by-side.

# Phase 1: Find a tutorial with an outdated workflow

{% include _includes/cyoa-choices.html option1="No" option2="Yes" default="No"
       text="Do you already have a tutorial and workflow you want to update?" %}

<div class="No" markdown="1">

> <hands-on-title>Check for an outdated workflow</hands-on-title>
>
> 1. **Find a {% icon hands_on %} Hands-on tutorial** in your training topic of interest (for example, Single-cell)
> 2. **Select the {% icon workflow %} workflow** from the header
> 3. **Import the workflow** to your favourite Galaxy server
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 4. Go to the **Workflow** menu and select {% icon galaxy-wf-edit %} **Edit workflow**
> 5. **Click through the tools** in the workflow and check the {% icon tool-versions %} to see if the tools are outdated.
{: .hands_on}

If the workflow has tools that are up to date (or very close!), great! That tutorial does not need updating! Try another one!

</div>

<div class="Yes" markdown="1">

> <hands-on-title>Import the workflow</hands-on-title>
>
> 1. **Navigate to your target {% icon hands_on %} Hands-on tutorial**
> 2. **Select the {% icon workflow %} workflow** from the header
> 3. **Import the workflow** to your favourite Galaxy server
>
>    {% snippet faqs/galaxy/workflows_import.md %}
>
> 4. Go to the **Workflow** menu and select {% icon galaxy-wf-edit %} **Edit workflow**
{: .hands_on}

</div>

# Phase 2: Check that nobody else is working on this

It's always a good idea to check, just in case!

You can:
 - Send a message to the [GTN Matrix Channel](https://matrix.to/#/#Galaxy-Training-Network_Lobby:gitter.im) is your quickest way forward.
 - Search through the [GTN Github Repository](https://github.com/galaxyproject/training-material) for existing draft Pull Requests.
 - Check with [individual communities](https://galaxyproject.org/community/sig/), who may have their own method of tracking. For example, the [🖖🏾Single-cell & sPatial Omics Community](https://galaxyproject.org/community/sig/singlecell/) have a shared [Click-Up board](https://sharing.clickup.com/9015477668/b/h/5-90152810734-2/557452707486fef) at the time of writing.

# Phase 3: Update the workflow

Now, you will update the workflow to using the latest {% icon tool-versions %} tool versions.

> <hands-on-title>Update the workflow</hands-on-title>
>
> 1. Select {% icon galaxy-wf-options %} **Workflow Options** from the **Workflow Editor**
> 2. Select {% icon upgrade_workflow %} **Upgrade workflow** to automatically update the tools.
> 3. Address any issues that may arise.
> 4. **Save** the workflow.
{: .hands_on}


# Phase 4: Test & fix the workflow

It's crucial to test the workflow, as often times the outputs will be different due to the new tool versions. It can also transpire that the newer tool versions lead to errors, either because they don't work or because you need to change parameter settings that were previously unavailable or not required. Testing is key!

> <hands-on-title>Test the workflow</hands-on-title>
>
> 1. Import the input datasets from the tutorial you are upgrading (follow the instructions on the tutorial itself for this).
> 2. Run your updated workflow on the input datasets.
> 3. Address any issues that may arise, and **NOTE DOWN** all changes.
> 4. Ensure the workflow meets workflow best practices using the {% icon galaxy-wf-best-practices %} **Best Practices** button and add your name as a contributor, if not there already.
>
>    {% snippet faqs/galaxy/workflows_best_practices.md %}
>
> 5. Add yourself as an author of the workflow.
> 6. **Save** the `updated workflow`.
> 7. Run your `updated workflow` on the input datasets in a fresh history.
> 7. **Save** this history as an `answer key history` & make your history publicly available in Published Histories.
>
>    {% snippet faqs/galaxy/histories_sharing.md %}
>
{: .hands_on}

# Phase 5: Update the tutorial

> <warning-title>If you run out of time</warning-title>
> You have already completed a large chunk of work (`updated workflow`+ `answer key history`) to get here, and we don't want to lose it!
> Updating the tutorial to match the `updated workflow` can be a separate contribution. So if you get to this point and run out of time, please:
> 1. Create an **issue** on the GTN Github Repository and include shareable links to your `updated workflow` & `answer key history`
> 2. Message on the GTN Matrix channel with a link to your issue and explaining which tutorial you updated.
> If, however, you are able to finish the task yourself, please read on!
{: .warning}

Note that you will need to have done either the [Contributing to the GTN Github tutorial using command line]({% link topics/contributing/tutorials/github-command-line-contribution/tutorial.md %}) or the [Contributing to the GTN Github tutorial using Github Desktop]({% link topics/contributing/tutorials/github-interface-contribution/tutorial.md %}). It's helpful as well to understand the folder structure in the GTN, particularly how images go in an image folder either in a tutorial or in the parent folder. Each `topic` has roughly the following structure:

Each topic has the following structure:

```
├── README.md
├── metadata.yaml
├── images
├── docker
│   ├── Dockerfile
├── slides
│   ├── index.html
├── tutorials
│   ├── tutorial1
│   │   ├── tutorial.md
│   │   ├── images
│   │   ├── slides.html
│   │   ├── data-library.yaml
│   │   ├── workflows
│   │   │   ├── workflow.ga
```

The `tutorial.md` is what you'll be editing, however you will also at the end upload a `workflow.ga` file and likely some image files.

> <hands-on-title>Update the tools in the tutorial</hands-on-title>
>
> 1. **Update tool versions**: Each {% icon hands_on %} *Hands-on* step in the tutorial will likely have tool (s) with versions in the text. Update these versions / links to be equivalent to your `updated workflow`.
> 2. **Update tool instructions**: (Simultaneously) update the text to address any differences arising during the update, i.e. new parameters to set or other changes.
> 3. **Update images**: Wherever images are used to show tool outputs, these will need updating. Use your final `answer key history` for this.
> 4. **Update text**: Wherever results are referenced in the tutorial text, update these numbers (and possibly interpretation) to reflect the new `answer key history`.
> 5. **Update header**: In the metadata at the beginning of the tutorial, there are likely `answer key history` links. Update this by adding the link to your `answer key history`.
> 6. **Update workflow file**: In the tutorial folder in the `training-material` repository, there is a subfolder titled `workflows`. Download your `updated workflow` as a file and deposit that file there.
> 7. Check if there are any links to workflows/histories in the tutorial text, and if so, update them.
> 7. Add your name as a *contributor* to the tutorial as an **editor** in the metadata.
{: .hands_on}

You may find this tutorial to be a helpful reference in the Markdown content of a GTN tutorial: [Creating content in Markdown]({% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md %})

# Phase 6: Make a pull request and (optional) update workflow testing

At this point, you're welcome to make your Pull Request for the updated tutorial. However, it is likely to fail linting if the workflow testing is not also updated. This can be tricky, so we'd rather you make the Pull Request with all the work you did than let this stop you.

But if you can make the workflow tests, that would be amazing!
Here's the tutorial: [Creating workflow tests using Planemo]({% link faqs/gtn/gtn_workflow_testing.md %})


{% icon congratulations %} Congratulations! You've made it to the end! Thank you so much for contributing to the sustainability of Galaxy tutorials!


topic: contributing
tutorial: schemas
title: GTN Metadata
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: []
tags_json: []
content_hash: f7e16da7a5691380af2360dd508317a6
last_modified: 2026-02-26T21:12:14.723574
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/schemas/tutorial.html

content:
## Training Materials

{% assign kid_key = "Tutorial Schema" %}
{% assign kid_val = site.data['schema-tutorial'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Slides Schema" %}
{% assign kid_val = site.data['schema-slides'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

## GTN Resources

{% assign kid_key = "FAQ Schema" %}
{% assign kid_val = site.data['schema-faq'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Topic Schema" %}
{% assign kid_val = site.data['schema-topic'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Quiz Schema" %}
{% assign kid_val = site.data['schema-quiz'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Learning Pathway Schema" %}
{% assign kid_val = site.data['schema-learning-pathway'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "News" %}
{% assign kid_val = site.data['schema-news'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Events" %}
{% assign kid_val = site.data['schema-event'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Events (External)" %}
{% assign kid_val = site.data['schema-event-external'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

## Contributors

{% assign kid_key = "Contributor Schema" %}
{% assign kid_val = site.data['schema-contributors'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Organisation Schema" %}
{% assign kid_val = site.data['schema-organisations'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "Grant Schema" %}
{% assign kid_val = site.data['schema-grants'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}


topic: contributing
tutorial: create-new-tutorial-content
title: Creating content in Markdown
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: fbf0d5a678fc9c75e0addc241a568eab
last_modified: 2026-02-26T21:12:14.702996
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html

content:
Once we have set up the infrastructure, we are ready to write the tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

The tutorial's content should be placed in the file `tutorial.md`. Its syntax and structure are simple, and will have the following structure:

```markdown
---
layout: tutorial_hands_on

title: Title of the tutorial
zenodo_link: ''
questions:
- Which biological questions are addressed by the tutorial?
- Which bioinformatics techniques are important to know for this type of data?
objectives:
- The learning objectives are the goals of the tutorial
- They will be informed by your audience and will communicate to them and to yourself
  what you should focus on during the course
- They are single sentences describing what a learner should be able to do once they
  have done completed tutorial
- You can use Bloom's Taxonomy to write effective learning objectives
time_estimation: ''
key_points:
- The take-home messages
- They will appear at the end of the tutorial
contributors:
- contributor1
- contributor2
---

introduction text

> <agenda​-title></agenda​-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}

# Section 1

blabla

## Subsection 1

blabla

# Section 2

blabla

## Subsection 2

blabla

# Conclusion

blabla
```

# Metadata

The `tutorial.md` needs to start with some metadata at the top:

- `layout: tutorial_hands_on`: keep the default
- `title`: title of the tutorial (it will appear on the tutorial page and the topic page)
- `level`: `Introductory`, `Intermediate` or `Advanced`
- `zenodo_link`: link on Zenodo to the input data for the tutorial
- `contributions`: eveybody who has contributed to this tutorial (usernames must match those in `CONTRIBUTORS.yaml` file)

> <hands-on-title>Fill the basic metadata</hands-on-title>
>
> 1. Update the tutorial information in the header section of your tutorial:
>
>     ```
>     title: "Similarity search with BLAST"
>     ```
> 2. (Optional) Add the Zenodo link (if created)
>
{: .hands_on}

This information is used to display the data from the topic and tutorial page. They are also used to check which information is missing for the tutorials.

We also define metadata related to the pedagogical content of the tutorial, which will appear at the top ("Overview" box) and bottom of the online tutorial:

{% assign kid_key = "Tutorial Schema" %}
{% assign kid_val = site.data['schema-tutorial'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

For this category of metadata, we have taken inspiration from what Software Carpentry has done and particularly what they described in their [Instructor training](https://swcarpentry.github.io/instructor-training/).

> <hands-on-title>Fill out the pedagogical metadata</hands-on-title>
>
> 1. Define 2 questions that will be addressed during the tutorial and add them to the metadata
> 2. Define 2 learning objectives for the tutorial and add them to the metadata
{: .hands_on}

> <comment-title>When filling the pedagogical metadata</comment-title>
> We recommend that you fill out the *questions* and the *learning objectives* before starting writing the tutorial content. You can still refine them afterwards, but it will help to guide you in developing your tutorial, and gives you some time to think beforehand on what topics are worth being covered.
>
> For the take-home messages, it is easier to define them once the tutorial is written and you identified the issues.
{: .comment}


## Listing contributors

All tutorials and slides must give credit to all contributors. This can be any type of contribution, adding them in GitHub, creating images for it, etc.

1. Make sure all contributors are listed in the [`CONTRIBUTORS.yaml`](https://github.com/galaxyproject/training-material/blob/main/CONTRIBUTORS.yaml) file.
   Each contributor is defined in this file like:

   ```yaml
   contributor-username:                 # GitHub username (if the contributor has one)
     name: Full Name                     # mandatory
     joined: 2020-06                     # mandatory
     email: saskia.hiltemann@gmail.com   # optional
     twitter: shiltemann                 # optional
     linkedin: shiltemann                # optional
     gitter: shiltemann                  # optional
     orcid: 0000-0003-3803-468X          # optional
     bio: Researcher at EMC              # optional
   ```

2. Add all contributors to the metadata of the tutorial or slide deck:

   ```yaml
   contributors:
     - contributor-username
     - shiltemann
     - hexylena
   ```

   Make sure these names match the usernames you used in the `CONTRIBUTORS.yaml` file.

3. **Optional:** Specifying types of contributions. If you want to give more detailed credit for conributions, you can do the following (instead of step 2 above)

   ```yaml
   contributions:
     authorship:
       - shiltemann
     editing:
       - bebatut
       - hexylena
     funding:
       - carpentries
     testing:
       - userX
     ux:
       - userY
     infrastructure:
       - userZ
   ```

   To define a funding body in the `CONTRIBUTORS.yaml` there are a few extra fields available:

   ```yaml
   gallantries:
     name: Gallantries Project
     joined: 2020-09
     avatar: "https://www.erasmusplus.nl/assets/images/logo.png"
     github: false
     funder: true
     funding_id: 2020-1-NL01-KA203-064717
     funding_statement: |
        This project ([`2020-1-NL01-KA203-064717`](https://erasmus-plus.ec.europa.eu/projects/search/details/2020-1-NL01-KA203-064717)) is funded with the support of the Erasmus+ programme of the European Union. Their funding has supported a large number of tutorials within the GTN across a wide array of topics.
        ![eu flag with the text: with the support of the erasmus programme of the european union](https://gallantries.github.io/assets/images/logosbeneficaireserasmusright_en.jpg)
   ```

   Funding bodies will be credited at the bottom of the tutorial with the appropriate funding statement, and will get a page in the hall of fame listing all tutorials that list them as a funder.

   For an example of how this all looks, see the [R basics tutorial]({% link topics/data-science/tutorials/r-basics/tutorial.md %}) (top and bottom of the tutorial).

{% assign kid_key = "Contributions Schema" %}
{% assign kid_val = site.data['schema-tutorial']['mapping']['contributions'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% assign kid_key = "CONTRIBUTORS Schema" %}
{% assign kid_val = site.data['schema-contributors'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

# Content

The tutorial's content is written directly after the section of metadata. This is written in Markdown, a simple markup language.

> <comment-title>Markdown</comment-title>
>
> Check [this cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to learn more how to use Markdown.
{: .comment}

The Markdown content is then transformed into a user friendly webpage through a templating system. With this approach, there is no need to add the name of every tutorial each time, since they are automatically added based on the tutorial's metadata.

To help developing the tutorial, we recommend to create a workflow of the different steps of the tutorial inside Galaxy first, and then you can create the structure of the tutorial automatically from that:

> <hands-on-title>Create the structure of the tutorial from a workflow</hands-on-title>
>
> 1. Create a small workflow with one or two steps on a running Galaxy instance
> 2. Add the topic name as Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor.
> 3. Get the workflow id
>    1. Go the "Share" page of the workflow
>    2. Copy the information after `id=` in the URL of the page
> 4. Get your API key for this Galaxy instance
>    1. Click on **User** --> **Preferences**
>    2. Click on **Manage API key**
>    3. Click on **Create a new key** (if none is available)
>    4. Copy the API key
> 5. Generate the skeleton of the tutorial locally
>
>    ```
>    $ planemo training_generate_from_wf \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --galaxy_url "URL to Galaxy instance in which you created the workflow" \
>             --galaxy_api_key "Your API key on the Galaxy instance" \
>             --workflow_id "ID of the workflow on the Galaxy instance" \
>             --zenodo_link "URL to the Zenodo record (Optional)"
>    ```
>
>    > <comment-title>Using a local workflow</comment-title>
>    > It is also possible to download the workflow locally (with the `.ga` extension), and then run a slightly different command:
>    >
>    > ```
>    > $ planemo training_generate_from_wf \
>    >          --topic_name "my-topic" \
>    >          --tutorial_name "my-new-tutorial" \
>    >          -- workflow PATH/to/the/file.ga \
>    >          --zenodo_link "URL to the Zenodo record (Optional)"
>    > ```
>    {: .comment}
>
> 6. Inspect the generated `tutorial.md`
{: .hands_on}

The generated tutorial is structured with:

- An introduction, to give an overview of the tutorial with its use cases, data, and methods
- Multiple sections representing the steps of the analysis, complete with automatically generated hands-on blocks, as practicing is a vital part of the learning process
- A conclusion to summarize what has been done in the tutorial (with a graphic)

> <hands-on-title>Filling out the structure of the tutorial</hands-on-title>
>
> 1. Fill out the "Introduction" with a general introduction of the tutorial and a small description of the dataset (goals)
> 2. Rename/restructure the sections with several levels and more explication
> 3. Add some theory about the tool used to introduce each section
> 4. Add a small conclusion and relate the results to the original question
>
{: .hands_on}

## Self Study Tutorials

By following all of the guidelines in this file you can be sure that your tutorial will be optimised for self-study.
The GTN framework encourages the use of snippets, and PTDK ensures tutorials have fully detailed parameters learners should configure.
If you make use of snippets in appropriate places, learners can easily follow a tutorial despite different skill levels with Galaxy.

Additionally use of [Question and Solution boxes](#questions-and-solutions-boxes) can ensure that students can self-check their understanding and progress as they progress through the tutorial.
Generally we recommend one or more questions (and associated correct solutions!) after every hands-on box (which might have a one or more steps to follow.)
These questions will let students be sure their work is correct before they proceed too far.
As such you should design your questions carefully in order to catch common and likely failure modes that learners may encounter.
If a student might forget to select a less-common or deeply-nested parameter, be sure that your question following that hands on tests that properly, and if possible explain that in the solutions.

## Adding images with captions

To add an image in Markdown file, we need to use the markdown syntax for this: {% raw %}`!​[proper alt text describing the image for visually impaired learners](../../images/image.png)`{% endraw %}.

We have also added a small plugin to handle captions for each image:

![A textual description of the image](../../images/image_caption_screenshot.png "Example of an image with a caption ")<!-- Adding a space to the caption to not trigger figurigy skip_titles -->

The prefix "Figure 1." is automatically added before its caption. This is done with the following Markdown syntax:

{% raw %}
```markdown
!​[A textual description of the image](../images/image.png "Example of an image with a caption")
```
{% endraw %}

We can also cross-reference images inside our Markdown with an anchor. For example, we can link to [the previous figure](#figure-1) using `[the display text]​(#figure-nb)` (changing `nb` with the image's number).

### Guidelines on Alt vs Figcaption Text

> While both the alt attribute and the figcaption element provide a way to
> describe images, the way we write for them is different. **`alt` descriptions
> should be functional; `figcaption` descriptions should be editorial or
> illustrative.**
>
{: .quote cite="https://thoughtbot.com/blog/alt-vs-figcaption" author="thoughtbot.com"}

As an example for this image:

![alt text]({{site.baseurl}}/topics/microbiome/images/plasmid-metagenomics-nanopore/sequence_method.jpg "Example of an image with a caption ")

{% raw %}
```markdown
!​[Alt text (shown when image cannot be displayed)](path/to/image.png "Example of an image with a caption")
```
{% endraw %}

Field          | Appropriate Contents
----           | -----
alt text       | Image of cell membrance with an embedded protein with central pore. DNA is shown splitting and entering the pore, an electrical signal comes out reading A C T or G.
figure caption | Using nanopore sequencing, a single molecule of DNA or RNA can be sequenced without the need for PCR amplification or chemical labeling of the sample. (Image from: <a href="https://nanoporetech.com/sites/default/files/s3/white-papers/WGS_Assembly_white_paper.pdf?submissionGuid=40a7546b-9e51-42e7-bde9-b5ddef3c3512">Nanopore sequencing: The advantages of long reads for genome assembly</a>)

## Writing mathematical expressions

Mathematical expressions can be written in LaTeX, and are automatically rendered with [MathJax](https://www.mathjax.org/).

Surround your math expression with two `$` signs on each side (like in LaTeX math blocks):

- inline expressions, *e.g.* `$$ 5 + 5 $$` will be rendered as $$ 5 + 5 $$
- block expressions, *e.g.* `$$ 5 + 5 $$` will be rendered in its own line block as

   $$ 5 + 5 $$

- Note: if inline mode is not working correctly, you can force it by using the following delimiters instead of dollar signs: `\\( 5 +5 \\)`

Dollar signs are therefore *reserved characters* for instructing the templating system to open/close LaTeX math blocks. If you want to use a `$` within your expression, you will need to *escape* it: `$$ a + 3\$ = 5\$ $$` will be rendered as: $$ a + 3\$ = 5\$ $$


LaTeX code that uses the pipe symbol `|` in inline math statements may lead to a line being recognized as a table line by the templating system.
This can be avoided by using the `\vert` command instead of `|`

## Tables and Matrices

Tables can be generated using markdown by using the `|` symbol to indicate column dividers, and `--` for table headers:

{% raw %}
```markdown
|       | Obs1 | Obs2 | Obs3 |
|------ |--------------------|
| Feat1 | 0    | 1    | 2    |
| Feat2 | 1    | 2    | 3    |
| Feat3 | 2    | 3    | 4    |
```
{% endraw %}

When rendered, they will take the full width of the page:

|       | Obs1 | Obs2 | Obs3 |
|------ |--------------------|
| Feat1 | 0    | 1    | 2    |
| Feat2 | 1    | 2    | 3    |
| Feat3 | 2    | 3    | 4    |

This does not appear to be visually appealing when representing matrices, which is why a matrix box can be used instead:

{% raw %}
```markdown
> |       | Obs1 | Obs2 | Obs3 |
> | ----- |--------------------|
> | Feat1 | 0    | 1    | 2    |
> | Feat2 | 1    | 2    | 3    |
> | Feat3 | 2    | 3    | 4    |
{: .matrix}
```
{% endraw %}

The rendered table is then given as a minimum-width and centred matrix:

> |       | Obs1 | Obs2 | Obs3 |
> | ----- |--------------------|
> | Feat1 | 0    | 1    | 2    |
> | Feat2 | 1    | 2    | 3    |
> | Feat3 | 2    | 3    | 4    |
{: .matrix}

## Internally linking to other training material

If you want to link to other training material within your text, please use the {%raw%}`{​% link path/to/file.ext %​}`{%endraw%} tag:

{%raw%}
```markdown
[link text]( {% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %} )
```
{%endraw%}

(Note the `.md` extension, and not `.html`, always provide the file name here, it will automatically be converted to the correct link)

If you want to link to a specific section in a tutorial using an anchor (e.g. `#getting-started`), place it outside of the {%raw%}`{​% link %​}`{%endraw%} tag:

{%raw%}
```markdown
[link text]({% link topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md %}#section-name)
```
{%endraw%}


# Improving the learning experience with Boxes

To improve the learning experience in our tutorial, we define some boxes to highlight content.

These boxes are defined always with the same structure:

{% raw %}
```markdown
> <type​-title>Name of the box</type​-title>
> text goes here!
{: .type}
```
{% endraw %}

where type is something like `tip`, so `<tip-title>` and `{: .tip}`. You must follow this structure **exactly** for it to be rendered correctly.

## **Overview** box

This box at the top of each tutorial is automatically generated using the metadata we defined in the topic's metadata file:

> <overview-title>Overview</overview-title>
>
> **{% icon question %} Questions**
> - Which biological questions are addressed by the tutorial?
> - Which bioinformatics techniques are important to know for this type of data?
>
> **{% icon objectives %} Objectives**
> - The learning objectives are the goals of the tutorial
> - They will be informed by your audience and will communicate to them and to yourself what you should focus on during the course
> - They are single sentences describing what a learner should be able to do once they have completed the tutorial
> - You can use Bloom's Taxonomy to write effective learning objectives
>
> {% icon requirements %} Requirements
> - [Galaxy introduction]({% link topics/introduction/index.md %})
>
> {% icon time %} Time estimation: '1H'
>
{: .overview}

> <hands-on-title>Checking the metadata</hands-on-title>
>
> 1. Check that the metadata added previously are correctly filling the overview box
>
>    > <question-title></question-title>
>    >
>    > What metadata hasn't been added to this box?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > The take-home messages are not added to this box but into the last box of the tutorial
>    > {: .solution}
>    {: .question}
>
{: .hands_on}

## **Agenda** box

In most tutorials, the second box is the agenda box, placed at the end of the introduction. It shows the table of contents for the tutorial

{% raw %}
```markdown
> <agenda​-title></agenda​-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}
```
{% endraw %}

There is no need to fill out the list; this will be done automatically based off of your tutorial's section titles. First and second level headings (lines starting with `#` and `##` in markdown) will be shown in the agenda.

If a section is not important enough to be shown in the agenda, consider making it a third-level heading (`###`) instead.

The agenda will be rendered as follows:

> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## **Hands-on** box

We find that having users walk through the tutorial, doing all of the steps is important for learning the concepts taught. We therefore emphasize this by regularly adding hands-on sections, where trainees are encouraged to do the analysis by themselves. We have designed some special boxes to make these sections easier to find.

{% raw %}
```markdown
> <hands​-on-title>Spliced mapping</hands​-on-title>
>
> 1. **RNA STAR** {% icon tool %}: Map your reads on the reference genome with
>    - *"Single-end or paired-end reads"*:  `Paired-end (as individual datasets)`
>    - *"RNA-Seq FASTQ/FASTA file, forward reads"*: the generated `trimmed reads pair 1` files (multiple datasets)
>    - *"RNA-Seq FASTQ/FASTA file, reverse reads"*: the generated `trimmed reads pair 2` files (multiple datasets)
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>    - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>    - *"Select reference genome"*: `Drosophila Melanogaster (dm6)`
>    - *"Gene model (gff3,gtf) file for splice junctions"*: the imported `Drosophila_melanogaster.BDGP6.87.gtf`
>    - *"Length of the genomic sequence around annotated junctions"*: `36`
>
>        This parameter should be length of reads - 1
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}: Aggregate the STAR logs with
>      - *"Which tool was used generate logs?"*: `STAR`
>      - *"Type of FastQC output?"*: `Log`
>      - *"STAR log output"*: the generated `log` files (multiple datasets)
{: .hands​_on}
```
{% endraw %}

For consistency please use:

- {% raw %}`{% icon hands_on %}`{% endraw %} icon to define that is an hands-on
- Short imperative sentences to make it easy to identify the tasks
- Name of the tool in bold followed by {% raw %}`{% icon tool %}`{% endraw %} icon to make it easy to identify a Galaxy tool
- Parameters for the tool as a sublist

This will be rendered like:


> <hands-on-title>Spliced mapping</hands-on-title>
>
> 1. **RNA STAR** {% icon tool %}: Map your reads on the reference genome with
>    - *"Single-end or paired-end reads"*:  `Paired-end (as individual datasets)`
>    - *"RNA-Seq FASTQ/FASTA file, forward reads"*: the generated `trimmed reads pair 1` files (multiple datasets)
>    - *"RNA-Seq FASTQ/FASTA file, reverse reads"*: the generated `trimmed reads pair 2` files (multiple datasets)
>    - *"Custom or built-in reference genome"*: `Use a built-in index`
>    - *"Reference genome with or without an annotation"*: `use genome reference without builtin gene-model`
>    - *"Select reference genome"*: `Drosophila Melanogaster (dm6)`
>    - *"Gene model (gff3,gtf) file for splice junctions"*: the imported `Drosophila_melanogaster.BDGP6.87.gtf`
>    - *"Length of the genomic sequence around annotated junctions"*: `36`
>
>        This parameter should be length of reads - 1
>
> 2. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}: Aggregate the STAR logs with
>      - *"Which tool was used generate logs?"*: `STAR`
>      - *"Type of FastQC output?"*: `Log`
>      - *"STAR log output"*: the generated `log` files (multiple datasets)
{: .hands_on}

There are also some predefined **parameter icons** available which can be used to indicate the *type* of parameter. These are optional,
but may be helpful in some cases (for example to distinguish between single file inputs and collection inputs).

The available icons are:

{% raw %}
```markdown
> <hands​-on-title>My Step</hands​-on-title>
>
> 1. **My Tool** {% icon tool %} with the following parameters
>    - {% icon param-text %} *"My text parameter"*: `my value`
>    - {% icon param-file %} *"My input file"*: `my file`
>    - {% icon param-files %} *"My multiple file input or collection"*: `my collection`
>    - {% icon param-select %} *"My select menu"*: `my choice`
>    - {% icon param-check %} *"My check box"*: `yes`
>    - {% icon param-toggle %} *"My toggle button"*: `Yes`
>    - {% icon param-repeat %} **My repeat parameter**
>      - *"param1"*: `42`
{: .hands​_on}
```
{% endraw %}

which, when rendered, look like:

> <hands-on-title>My Step</hands-on-title>
>
> 1. **My Tool** {% icon tool %} with the following parameters
>    - {% icon param-text %} *"My text parameter"*: `my value`
>    - {% icon param-file %} *"My input file"*: `my file`
>    - {% icon param-files %} *"My multiple file input or collection"*: `my collection`
>    - {% icon param-select %} *"My select menu"*: `my choice`
>    - {% icon param-check %} *"My check box"*: `yes`
>    - {% icon param-toggle %} *"My toggle button"*: `Yes`
>    - {% icon param-repeat %} **My repeat parameter**
>      - *"param1"*: `42`
{: .hands_on}

## **Questions** and **Solutions** boxes

Questions can be added to force trainees to think about what they are currently doing, and to put things in perspective.
They can also help the instructors by exposing and clarifying common scenarios, errors, or applications.

{% raw %}
```markdown
> <question​-title></question​-title>
>
> 1. Why are some tests filtered?
> 2. Does it improve the *p*-value distribution?
>
> > <solution​-title></solution​-title>
> >
> > 1. Sol for the first question
> > 2. Sol for the second question
> >
> {: .solution}
{: .question}
```
{% endraw %}

Which will be rendered as:

> <question-title></question-title>
>
> 1. Why are some tests filtered?
> 2. Does it improve the *p*-value distribution?
>
> > <solution-title></solution-title>
> >
> > 1. Sol for the first question
> > 2. Sol for the second question
> >
> {: .solution}
{: .question}

Questions should be quick to answer. You can directly ask a question and expect an answer, or you can provide some answers and create multiple choices questions (MCQs).
With well chosen wrong answers, MCQs can do much more than just measure how much someone knows, such as exposing common misconceptions and mistakes.

In the box below, initially hidden, we add the correct answer and possibly any additional explanation. Self-trainees can then check the solution and its explanation.

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching-selfstudy.md %}


## **Tips** box

Tips boxes are really just for 'tips', usually hints regarding Galaxy operations that users may or may not be familiar with. If you want to provide extended discussion or links to external materials then consider the comment and detail boxes instead.

{% raw %}
```markdown
> <tip​-title>Importing data via links</tip​-title>
>
> * Copy the link location
> * Open the Galaxy Upload Manager
> * Select **Paste/Fetch Data**
> * Paste the link into the text field
> * Press **Start**
{: .tip​}
```
{% endraw %}

Rendered:

> <tip-title>Importing data via links</tip-title>
>
> * Copy the link location
> * Open the Galaxy Upload Manager
> * Select **Paste/Fetch Data**
> * Paste the link into the text field
> * Press **Start**
{: .tip}

## **Comments** boxes

{% raw %}
```markdown
> <comment​-title></comment​-title>
> - Edit the "Database/Build" to select "dm3"
> - Rename the datasets according to the samples
{: .comment}
```
{% endraw %}

Rendered:

> <comment-title></comment-title>
> - Edit the "Database/Build" to select "dm3"
> - Rename the datasets according to the samples
{: .comment}

## **Details** box

The detail box is used to give more background explanation on the subject. By default the box is collapsed, trainees can expand it if they wish to know extra information about a topic.

{% raw %}
```markdown
> <details​-title>More details on the ....</details​-title>
>
> Add more details in Markdown...
>
{: .details​}
```
{% endraw %}

Rendered:

> <details-title>More details on the ....</details-title>
>
> Add more details in Markdown...
>
{: .details}

## **Key points** box

This last box of the tutorial is automatically created with the take-home messages defined in the topic's metadata

To render the boxes correctly, the syntax needs to be correct. If it doesn't work, have a look at similar tutorials and get inspiration.

## **Warning** box

{% raw %}
```markdown
> <warning​-title>Danger: You can lose data!</warning​-title>
> Something really bad can happen here!
{: .warning}
```
{% endraw %}

Rendered:

> <warning-title>Danger: You can lose data!</warning-title>
> Something really bad can happen here!
{: .warning}


## Nested boxes

Boxes can be nested, *e.g.* for having tips inside a hands-on:

{% raw %}
```markdown
> <hands​-on-title>Defining the topic for the tutorial</hands​-on-title>
>
> 1. Search for NCBI Blast+ on the [ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. Check in which category it is
>
>    > <question​-title></question​-title>
>    >
>    > In which topic will you put the tutorial?
>    >
>    > > <solution​-title></solution​-title>
>    > >
>    > > If we search for [NCBI Blast+ in the ToolShed](https://toolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/7538e2bfcd41), it is attributed to 2 categories (bottom): "Next Gen Mappers" and "Sequence Analysis".
>    > > We decided to put it in "Sequence analysis" because this is the most general one for this tutorial.
>    > {: .solution​}
>    {: .question​}
{: .hands​_on}
```
{% endraw %}

## **Code** box

We have added code in/out boxes to help you show commands, and their effects, when running command line commands.

Normally a single column, with the boxes above one another, it will automatically split side-by-side over a given width (1200px);

{% raw %}
```markdown
> > <code​-in-title>Bash</code​-in-title>
> > ```bash
> > cat /tmp/test.ini
> > ```
> {: .code-in}
>
> > <code​-out-title></code​-out-title>
> > The file should look like:
> >
> > ```ini
> > [example]
> > server_name = Dogs!
> > listen = 192.168.0.2
> > apikey = super-secret-api-key-wow!
> > ```
> {: .code-out}
{: .code-2col}
```
{% endraw %}

Rendered (try it! resize your browser)

> > <code-in-title>Bash</code-in-title>
> > ```bash
> > cat /tmp/test.ini
> > ```
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > The file should look like:
> >
> > ```ini
> > [example]
> > server_name = Dogs!
> > listen = 192.168.0.2
> > apikey = super-secret-api-key-wow!
> > ```
> {: .code-out}
{: .code-2col}

If you leave off the `{: .code-2col}`, it will render as a single column always.

{% raw %}
```markdown
> <code​-in-title>Bash</code​-in-title>
> ```bash
> cat /tmp/test.ini
> ```
{: .code-in}

> <code​-out-title></code​-out-title>
> The file should look like:
>
> ```ini
> [example]
> server_name = Dogs!
> listen = 192.168.0.2
> apikey = super-secret-api-key-wow!
> ```
{: .code-out}
```
{% endraw %}

Rendered:

> <code-in-title>Bash</code-in-title>
> ```bash
> cat /tmp/test.ini
> ```
{: .code-in}

> <code-out-title></code-out-title>
> The file should look like:
>
> ```ini
> [example]
> server_name = Dogs!
> listen = 192.168.0.2
> apikey = super-secret-api-key-wow!
> ```
{: .code-out}

## **Quote** boxes

{% raw %}
```markdown
> If you don't know where you're going, you might not get there.
{: .quote cite="https://en.m.wikiquote.org/wiki/Yogi_Berra" author="Yogi Berra"}
```
{% endraw %}

Rendered:

> If you don't know where you're going, you might not get there.
{: .quote cite="https://en.m.wikiquote.org/wiki/Yogi_Berra" author="Yogi Berra"}

The citation and author parameters are both optional. If provided the `cite` key must be a URL.

> If you don't know where you're going, you might not get there.
{: .quote cite="https://en.m.wikiquote.org/wiki/Yogi_Berra"}

> If you don't know where you're going, you might not get there.
{: .quote author="Yogi Berra"}

# Additional Features to Improve Learning

Here we cover additional features you can use throughout your tutorials to improve the learning experience.

## Tool Links

With the new [GTN in Galaxy Webhook](https://github.com/galaxyproject/galaxy/pull/10024), trainees can view training directly within Galaxy. As part of this, we enable those trainees to click on tools, and have those tools directly activated in Galaxy, enabling for a seamless training experience for trainees.

![GIF of a user using the GTN in Galaxy webhook.](../../images/88277962-ddda4a80-cce1-11ea-92cd-41b1df063db0.gif "A gif showing how the GTN in Galaxy webhook works. A student clicks the learning hat icon in the masthead of a Galaxy server, and an overlay is activated showing the GTN website. Within the GTN they can browse around and their place in tutorials is saved. While following a tutorial the student reches a step which instructs them to run a specific tool. Instead of the normal experience searching for a tool (quite difficult on large servers), they click a blue button and the tool is activated in Galaxy, and the overlay is closed. The student can reactivate the overlay at any time and return to their place in the tutorial.")

To enable these in your tutorial you can use the following syntax:

{% raw %}
```
- {% tool MultiQC %}
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}
- {% tool [Import some data](upload1) %}
```
{% endraw %}

Which will be rendered as:

- {% tool MultiQC %}
- {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.8+galaxy0) %}
- {% tool [Import some data](upload1) %}

When viewed through Galaxy, students will see:

<span data-tool="upload1" title="Tested with upload1" class="tool galaxy-proxy-active"><strong>Import some data</strong> <i class="fas fa-wrench" aria-hidden="true"></i><i aria-hidden="true" class="fas fa-cog"></i><span class="visually-hidden">Tool: upload1</span></span>

### How to find these IDs?

The easiest way is to use planemo to generate the training from a workflow. In recent versions of planemo, this is managed automatically.

The alternative is to figure out the ID for the tool you want to use:

1. Find your tool in Galaxy, and click to access the tool form.
2. Click on Options at the top right
3. Click on Share
4. The URL shown will be something like `https://usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/galaxyp/mz_to_sqlite/mz_to_sqlite/2.0.4+galaxy1`
5. Keep only the part after the `=`, so `toolshed.g2.bx.psu.edu/repos/galaxyp/mz_to_sqlite/mz_to_sqlite/2.0.4+galaxy1` in this example

![Finding the tool ID](../../images/tool-id.png)

## Example Histories

If you have example input histories for your tutorial, perhaps for specific servers where trainees will often follow a tutorial but want to skip a slow input step, then you can provide example histories as part of your tutorial.

> <code-in-title>Tutorial Frontmatter</code-in-title>
> ```yaml
> answer_histories:
>   - label: "UseGalaxy.eu"
>     history: https://humancellatlas.usegalaxy.eu/u/j.jakiela/h/generating-a-single-cell-matrix-using-alevin-3
>   - label: "Older Alevin version"
>     history: https://humancellatlas.usegalaxy.eu/u/wendi.bacon.training/h/cs1pre-processing-with-alevin---answer-key
>     date: 2024-01-01
> input_histories:
>   - label: "UseGalaxy.eu"
>     history: https://humancellatlas.usegalaxy.eu/u/wendi.bacon.training/h/cs1pre-processing-with-alevin---input-1
> ```
{: .code-in}

> <code-out-title>Rendered Tutorial</code-out-title>
> ![a screenshot of the GTN metadata box showing a dropdown for input histories and answer histories. The answer histories features two examples one on UseGalaxy.eu and an older alevin one, each with a date. In the dropdown is also a link to an FAQ titled how to use this](images/example-histories.png)
{: .code-out}

## Workflows

In some tutorials you aren't as interested in teaching users the individual steps for analysing data, but rather want to focus on some downstream aspects of analysis, or to showcase the best practice workflows that are already available for a user to use! In those cases it can be useful to have a nicer way of inviting the user to execute those steps.

If you are accessing these in tutorial mode, they should function as button that, when clicked, launch the workflow directly. Outside of tutorial mode, they will link to our redirection service which will let you supply which Galaxy you plan to use.

Note that if for some reason the 'fancy' method doesn't work, there are fallback tip boxes to help a user execute a similar procedure manually.

### GTN Workflows

If you've included the workflow in the GTN but haven't uploaded to a repository yet:

{% raw %}
```markdown
{% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Data_QC.ga" title="Galaxy Workflow Data QC" %}
```
{% endraw %}

Rendered:

{% snippet faqs/galaxy/workflows_run_trs.md path="topics/assembly/tutorials/largegenome/workflows/Galaxy-Workflow-Data_QC.ga" title="Galaxy Workflow Data QC" %}


### WorkflowHub

You can use a dedicated snippet to invite users to run a WorkflowHub workflow:

{% raw %}
```markdown
{% snippet faqs/galaxy/workflows_run_wfh.md title="mRNA-Seq BY-COVID Pipeline" wfhub_id="685" version="1" %}
```
{% endraw %}

Rendered:

{% snippet faqs/galaxy/workflows_run_wfh.md title="mRNA-Seq BY-COVID Pipeline" wfhub_id="685" version="1" %}

Note that it links to a specific workflow, on any Galaxy server. When this tutorial is opened from within the Tutorial Mode, that link will change to one on the current server, removing the intermediate step.

### Dockstore

Please note that the dockstore ID should be provided without the `#workflow/` prefix, so starting from `github.com`.

{% raw %}
```markdown
{% snippet faqs/galaxy/workflows_run_ds.md title="K-mer Profiling HiFi" dockstore_id="github.com/iwc-workflows/kmer-profiling-hifi-VGP1/main" version="v0.1.5" %}
```
{% endraw %}

Rendered:

{% snippet faqs/galaxy/workflows_run_ds.md title="K-mer Profiling HiFi" dockstore_id="github.com/iwc-workflows/kmer-profiling-hifi-VGP1/main" version="v0.1.5" %}

This snippet has the same behaviour, it will use my.galaxy.training links to make them server independent, but in Tutorial Mode it will open on the current server.


## FAQs (snippets)

Many common questions or instructions may be useful to share between different tutorials. For example instructions on how to start a new history or importing data. To make these types of snippets easier to re-use and avoid duplication, they are available in the form of *snippets*.

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching.md %}

### Finding snippets
These are available in folders named `faqs`, either at the project level, topic level, or tutorial level.

- **Project-level FAQs:** `faqs/`
  - `faqs/galaxy/` for general Galaxy questions
  - `faqs/gtn/` for questions regarding the GTN website itself

- **Topic-level FAQs:** `topics/<topic>/faqs/`
  - for questions pertaining to that specific topic

- **Tutorial-level FAQs:** `topics/<topic>/tutorials/<tutorial>/faqs/`
  - for questions pertaining to that specific tutorial
  - if this is present, it is linked to from the tutorial overview box at the top, and from the end of the tutorial


### Including FAQs/snippets in your tutorials

To include one of these snippets in your tutorial, you can use the following syntax:

```
{% raw %}{% snippet faqs/galaxy/histories_create_new.md %}{% endraw %}
```

Which will be rendered as:

{% snippet faqs/galaxy/histories_create_new.md %}

The advantage of this approach is that when the Galaxy interface updates, we only have to update the snippet, rather than every tutorial. Please try to use snippets whenever you can!

You could also specify the box type you want the snippet to be rendered in:

```
{% raw %}{% snippet faqs/galaxy/histories_create_new.md box_type="hands_on" %}{% endraw %}
```

{% snippet faqs/galaxy/histories_create_new.md box_type="hands_on" %}

or without a box altogether:

```
{% raw %}{% snippet faqs/galaxy/histories_create_new.md box_type="none" %}{% endraw %}
```

{% snippet faqs/galaxy/histories_create_new.md box_type="none" %}



### Creating new FAQs/snippets

Do you want to include something in your tutorial that you think might be useful in other tutorials as well? Or are you answering a frequently asked question? Consider creating a snippet for it

Each snippet (question) is a separate file, with some metadata, residing in one of the `faqs` folders:

```yaml
---
title: How do I run a workflow?
area: workflows      # FAQs will be grouped by these areas on the FAQ page
box_type: tip        # tip/comment/hands_on/question; optional, if you want the content to be in a box
layout: faq          # if you set this the snippet will get its own page and be included in the FAQs page
contributors: [annefou,nomadscientist] # anybody who has contributed to the FAQ over time
---

Here you can write the snippet / answer to the FAQ in Markdown

- Go to `Workflows` on the top menu bar
- Click on ..
- ..

```

{% assign kid_key = "FAQ Schema" %}
{% assign kid_val = site.data['schema-faq'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching-faqs.md %}

### FAQ pages

All FAQs will also be collected on their own page, this makes it easy for and teachers to prepare the session, and for participants to quickly find the answers to common questions.


To do this, create a file named `index.md` inside the faq folder:

```yaml
---
layout: faq-page
---
```

If you would like to enforce an order of the faq areas, you can do so:

```yaml
---
layout: faq-page
area_order: [introduction, learners, instructors, contributors, other]
---
```

(just make sure you list all existing areas in the folder)

If a tutorial-level FAQ page exists (`topics/<topic>/tutorials/<tutorial>/faqs/index.md`) it will be automatically linked to from the overview box at the top of the tutorial, and at the end of the tutorial. Have a look at this tutorial to see it in action.

## Footnotes

> > <code-in-title>Markdown</code-in-title>
> >
> > ```
> > Footnotes[^test] can be used to insert more content or explanation as reference material to your content. You can use the same footnote reference multiple time, and the footnote will include backlinks to return to the correct place in the text.[^test]
> > ```
> > {: .pre-break-lines}
> {: .code-in}
>
> > <code-out-title></code-out-title>
> > Footnotes[^test] can be used to insert more content or explanation as reference material to your content. You can use the same footnote reference multiple time, and the footnote will include backlinks to return to the correct place in the text.[^test]
> {: .code-out}
{: .code-2col}

[^test]: The wikipedia definition of a footnote is: "A note is a string of text placed at the bottom of a page in a book or document or at the end of a chapter, volume or the whole text. The note can provide an author's comments on the main text or citations of a reference work in support of the text. Footnotes are notes at the foot of the page while endnotes are collected under a separate heading at the end of a chapter, volume, or entire work. Unlike footnotes, endnotes have the advantage of not affecting the layout of the main text, but may cause inconvenience to readers who have to move back and forth between the main text and the endnotes."

## Icons

To use these icons, take the name of the icon, 'details' in this example, and write something like this in your tutorial:

```markdown
{% raw %}{% icon details %}{% endraw %}
```

Some icons have multiple aliases, any may be used, but we'd suggest trying to choose the most semantically appropriate one in case Galaxy later decides to change the icon.

<div class="row">
{% assign icon_groups = site['icon-tag'] | group_icons %}
{% for icon in icon_groups %}
	<div class="col-md-2 col-sm-3" style="text-align: center">
		<div style="font-size: 400%">{% icon_var icon[0][0] %}</div>
		<div>{% for z in icon[0] %}{{ z }}{%unless forloop.last%},{%endunless%} {% endfor %}</div>
	</div>
{% endfor %}
</div>

## Abbreviations

Oftentimes there are terms you'll use over and over again where there is an acronym or abbreviation you'll use consistently. However for learners new to the material, they might need to scroll up to the first definition every time to remember what it meant. It would be annoying as an author to have to re-define it every time, so we've implemented some very simple syntax to allow you to create a list of definitions and then use those in the text.

In your tutorial metadata you can add an abbreviations section like:

```yaml
---
title: My awesome tutorial
...
abbreviations:
  API: Application Programming Interface
  JSON: JavaScript Object Notation
---
```

And in your text you can use braces to refer to the term

> > <code-in-title>Markdown</code-in-title>
> > <code>
> > The `/jobs` &lbrace;API&rbrace; will return &lbrace;JSON&rbrace;. When we call the &lbrace;API&rbrace; we'll get back this result &lbrace;JSON&rbrace;.
> > </code>
> {: .code-in}
>
> > <code-out-title></code-out-title>
> >
> > The `/jobs` {API} will return {JSON}. When we call the {API} we'll get back this result {JSON}.
> >
> {: .code-out}
{: .code-2col}

{% assign kid_key = "Abbreviations Schema" %}
{% assign kid_val = site.data['schema-tutorial']['mapping']['abbreviations'] %}
{% include _includes/schema-render.html key=kid_key value=kid_val %}

## Choose Your Own Tutorial

Sometimes you're writing a large tutorial and at one small step there are multiple paths or multiple ways to get the data you want, and you'd like to showcase them all! You could write them all out in order, Option 1...2...etc, however maybe you want it to be a bit more interactive and focus only on one option at a time, so a user doesn't get distracted by the other options.

Include this markdown where you want your user to choose between the multiple paths:

<!-- GTN:IGNORE:041 we cannot tell code samples from text so we get dupe warnings here. -->

> <code-in-title>Markdown</code-in-title>
> {% raw %}
> ```
> {% include _includes/cyoa-choices.html option1="Ananas of course" option2="Avocados" default="Avocados"
>        text="Here is why some people choose Ananas. Other times you want Avocados as they fit the menu better." %}{% endraw %}
> ```
{: .code-in}

{% include _includes/cyoa-choices.html option1="Ananas of course" option2="Avocados" default="Avocados" text="Here is why some people choose Ananas. Other times you want Avocados as they fit the menu better." %}

And then they can wrap the relevant sections with a `div` block with the relevant class (if you have space in your option, use `-` in the class). You **must** set `markdown="1"` as well to have the inner contents rendered corretly.

**NB**: If you do not set a default, then on the very first page load, both options will be shown in their entirety. As soon as the user selects one of the options by clicking the relevant button, then the list is filtered. The user's browser generally will remember which button was selected across navigation and page reloads.

> > <code-in-title>Markdown</code-in-title>
> > ```
> > <div class="Ananas-of-course" markdown="1">
> > - 🍍 are fantastic
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> > <div class="Avocados" markdown="1">
> > - 🥑 are amazing
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> > ```
> >
> {: .code-in}
>
> > <code-out-title></code-out-title>
> >
> > <div class="Ananas-of-course" markdown="1">
> > - 🍍 are fantastic
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> > <div class="Avocados" markdown="1">
> > - 🥑 are amazing
> > - hands on!
> > - questions!
> > - solutions!
> > </div>
> >
> {: .code-out}
{: .code-2col}

> <tip-title>Why ananas-of-course? Name munging</tip-title>
> CYOAs work by filtering the content based on the class name. Class names cannot contain spaces, so, we need to replace all whitespace with `-`.
> The exact algorithm we use to create 'safe' IDs is:
>
> 1. Remove the following character `"`, `'`, `/`, `;`, `:`, `,`, `.`, `!`, `@`, `#`, `$`, `%`, `^`, `&`, `*`, `(`, `)` from the text.
> 2. Replace all whitespace with `-`
> 3. Replace multiple `-`s with a single `-`
>
> This results in retaining capitalisation. Hence, `Ananas of course` becomes `Ananas-of-course`.
> For a more complicated example `(RNA) inhibitors` would become `RNA-inhibitors`.
{: .tip}

This can also be used inline: My favourite fruit is an <span class="Ananas-of-course">🍍</span><span class="Avocados">🥑</span>.

> <tip-title>Multiple, Disconnected CYOAs</tip-title>
> If you wish to have multiple CYOAs in a single tutorial, you are free to do that! However you must:
>
> 1. Ensure that all options are disjoint, there should not be any shared terms! (I.e if the both CYOAs need to use "STAR", please find a different way to phrase it, or even use "STAR ", it just needs to be different.)
> 2. Provide a disambiguation term for them, passed as a parameter to all, or all but one, includes.
>
> This disambiguation term will affect the URL parameter, which will become `?gtn-cyoa{term}={value}`
>
> E.g.:
>
> ```
> {% raw %}
> {% include _includes/cyoa-choices.html option1="Oui" option2="Non" default="Oui" text="Vos données ESTAMP sont prêtes ?" %}
> {% include _includes/cyoa-choices.html option1="Yes" option2="No" default="Yes" text="Do the thing?" disambiguation="english" %}
> {% endraw %}
> ```
{: .tip}

> <tip-title>Using every branch</tip-title>
> We, the GTN, don't always know if you *intend* to have not used a specific branch. As such we recommend that if you have a branch that has no steps, you should include a `div` block with the class name of that branch, just so we know for sure you meant it to be empty.
>
> E.g. 
>
> ```
> {% raw %}
> {% include _includes/cyoa-choices.html option1="Yes, they don't pass QC" option2="No" default="Yes, they don't pass QC" text="Do you think your samples need to be filtered?" %}
> 
> <div class="Yes-they-dont-pass-QC" markdown="1">
> Some content here!
> </div>
> 
> <div class="No" markdown="1">
> <!-- intentionally empty. This comment itself isn't necessary, but we recommend adding the div. -->
> </div>
> 
> {% endraw %}
> ```
{: .tip}

### URL Parameter

The branch can be selected via URL parameter e.g. for courses, to prevent users selecting the wrong path. Just supply `?gtn-cyoa=Ananas` (or your preferred value) on the tutorial URL.

- [See this page with Ananas](?gtn-cyoa=Ananas#choose-your-own-tutorial)
- [See this page with Avocados](?gtn-cyoa=Avocados#choose-your-own-tutorial)

{% snippet topics/contributing/tutorials/create-new-tutorial-content/faqs/remote-teaching-cyoa.md %}

# Citations
If you would like to cite any articles, books or websites in your tutorial, you can do so by adding a file called `tutorial.bib` next to your `tutorial.md` file. In this file you may enter [bibtex](http://www.bibtex.org/Using/) formatted citations. An e


topic: contributing
tutorial: admin-knitting
title: Updating diffs in admin training
difficulty: intermediate
time_estimation: 5m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 0465728964d9ef87b105a21cf3b9f90b
last_modified: 2026-02-26T21:12:14.701427
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/admin-knitting/tutorial.html

content:
# Updating Diffs


The admin training was recently converted completely to use diffs. This was
done so we could build a git repository, and time travel to any point within
admin training, in support of students and testing.

Here's some common editing scenarios to help you out.

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

## How it Works

Tutorials are written with diffs that look like this:

    ```diff
    --- /dev/null
    +++ b/ansible.cfg
    @@ -0,0 +1,4 @@
    +[defaults]
    +interpreter_python = /usr/bin/python3
    +inventory = hosts
    +retry_files_enabled = false
    ```
    {: data-commit="Add ansible.cfg"}

The `data-commit` will be turned into the commit message, so, you can write something nice here. Otherwise it's a pretty standard diff. For tutorials which are written in this way, we can extract all of the diffs.

### Frogging

In knitting you sometimes need to rip out the stitches, a process sometimes called frogging.

> If they are not secured, the loops of a knitted course will come undone when their yarn is pulled; this is known as ripping out, unravelling knitting, or humorously, frogging (because you 'rip it', this sounds like a frog croaking: 'rib-bit').
{: .quote cite="https://en.wikipedia.org/wiki/Knitting"}

```console
$ python bin/knit-frog.py topics/admin/tutorials/singularity/tutorial.md /tmp/03-singularity
/tmp/03-singularity-commit-0000-add-golang-and-singulary-ansible-roles.patch
/tmp/03-singularity-commit-0001-configure-golang-and-singularity.patch
/tmp/03-singularity-commit-0002-add-the-roles-to-the-playbook.patch
/tmp/03-singularity-commit-0003-configure-the-container-and-dependency-resolvers.patch
/tmp/03-singularity-commit-0004-configure-the-dependency-resolvers.patch
/tmp/03-singularity-commit-0005-configure-the-container-resolver.patch
/tmp/03-singularity-commit-0006-update-the-job-conf-xml-with-singularity-destination.patch
```

This rips out the diffs and writes them into patch files.

### Knitting

With a series of patch files, we have a similar script to knit them back together.

```console
$ python bin/knit.py topics/admin/tutorials/singularity/tutorial.md --patches /tmp/03-singularity*
```

This takes the patch files and lines them up in the tutorial. It is **not intelligent**. You should have the same number of patch files as you do diffs with `data-commit`. Up to you to manage that.

## Automation

But mostly you don't need to care about the `knit.py` or `knit-frog.py`, we have `knit-automated.sh` which takes care of calling those properly. You can call it with `export` to write the patches into `/tmp/git-gat/`

```
$ ./bin/knit-automated.sh export
/tmp/git-gat/0-ansible-galaxy-commit-0000-add-requirements.patch
/tmp/git-gat/0-ansible-galaxy-commit-0001-add-ansible-cfg.patch
/tmp/git-gat/0-ansible-galaxy-commit-0002-add-hosts.patch
/tmp/git-gat/0-ansible-galaxy-commit-0003-add-initial-group-variables-file.patch
/tmp/git-gat/0-ansible-galaxy-commit-0004-add-initial-galaxy-playbook.patch
/tmp/git-gat/0-ansible-galaxy-commit-0005-add-pip--miniconda--galaxy-to-playbook.patch
/tmp/git-gat/0-ansible-galaxy-commit-0006-configure-miniconda-and-galaxy.patch
[...]
/tmp/git-gat/6-pulsar-commit-0000-add-requirements.patch
/tmp/git-gat/6-pulsar-commit-0001-configure-rabbitmq.patch
/tmp/git-gat/6-pulsar-commit-0002-add-role.patch
/tmp/git-gat/6-pulsar-commit-0003-add-pulsar-group-variables.patch
/tmp/git-gat/6-pulsar-commit-0004-add-pulsar-host.patch
/tmp/git-gat/6-pulsar-commit-0005-add-pulsar-playbook.patch
/tmp/git-gat/6-pulsar-commit-0006-add-pulsar-plugin.patch
/tmp/git-gat/6-pulsar-commit-0007-add-pulsar-destination.patch
/tmp/git-gat/6-pulsar-commit-0008-send-bwa-and-bwa-mem-to-pulsar.patch
```

And there you can build a repository from them!

```
[hxr@mk:/tmp/git-gat]$ git init
[hxr@mk:/tmp/git-gat]$ git am -3 *
Applying: admin/ansible-galaxy/0000: Add requirements
applying to an empty history
Applying: admin/ansible-galaxy/0001: Add ansible.cfg
Applying: admin/ansible-galaxy/0002: Add hosts
Applying: admin/ansible-galaxy/0003: Add initial group variables file
Applying: admin/ansible-galaxy/0004: Add initial galaxy playbook
Applying: admin/ansible-galaxy/0005: Add pip, miniconda, galaxy to playbook
[...]
Applying: admin/pulsar/0004: Add pulsar host
Applying: admin/pulsar/0005: Add pulsar playbook
Applying: admin/pulsar/0006: Add pulsar plugin
Applying: admin/pulsar/0007: Add pulsar destination
Applying: admin/pulsar/0008: Send bwa and bwa-mem to pulsar
```

Et voila!

```console
$ rm *.patch
$ tree
.
├── ansible.cfg
├── files
│   └── galaxy
│       ├── config
│       │   ├── dependency_resolvers_conf.xml
│       │   └── tool_destinations.yml
│       ├── dynamic_job_rules
│       │   ├── map_resources.py
│       │   └── my_rules.py
│       └── tools
│           └── testing.xml
├── galaxy.yml
├── group_vars
│   ├── all.yml
│   ├── galaxyservers.yml
│   └── pulsarservers.yml
├── hosts
├── pulsar.yml
├── requirements.yml
└── templates
    ├── galaxy
    │   └── config
    │       ├── container_resolvers_conf.xml.j2
    │       ├── job_conf.yml.j2
    │       └── job_resource_params_conf.xml.j2
    └── nginx
        ├── galaxy.j2
        └── redirect-ssl.j2

10 directories, 18 files
```

When you're done editing, you can run the `./bin/knit-automated.sh import`, it will

- extract the history as patches (using `git format-patch`)
- knit them into the tutorials

> <warning-title>Danger: Don't mess with the commit names at this step</warning-title>
> They're used for figuring out which commit a patch file belongs to. Sorry!
{: .warning}


# Git-Gat Knick-Knacks: Some Common Scenarios

## I want to update the text of a specific commit

> <question-title>Example problem</question-title>
> I want to edit the commit titled "Add production facing vars" to add an addtional variable in admin/ansible-galaxy.
{: .question}

> <hands-on-title>Make the change you want to see in the GAT</hands-on-title>
> 1. Start by exporting
>
>    ```
>    $ bash bin/knit-automated.sh export
>    ```
>
> 2. Build the repository, and apply the patches to get the current state.
>
>    ```
>    [hxr@mk:/tmp/git-gat]$ git init
>    [hxr@mk:/tmp/git-gat]$ git am -3 *.patch
>    Applying: admin/ansible-galaxy/0000: Add requirements
>    applying to an empty history
>    Applying: admin/ansible-galaxy/0001: Add ansible.cfg
>    Applying: admin/ansible-galaxy/0002: Add hosts
>    Applying: admin/ansible-galaxy/0003: Add initial group variables file
>    Applying: admin/ansible-galaxy/0004: Add initial galaxy playbook
>    Applying: admin/ansible-galaxy/0005: Add pip, miniconda, galaxy to playbook
>    Applying: admin/ansible-galaxy/0006: Configure miniconda and galaxy
>    Applying: admin/ansible-galaxy/0007: Configure galaxy config
>    [...]
>    [hxr@mk:/tmp/git-gat]$ rm *.patch
>    ```
>
> 3. Find the commit:
>
>    ```
>    $ git log --oneline | grep "Add production facing vars"
>    2f5b416 admin/ansible-galaxy/0022: Add production facing vars
>    ```
> 4. Edit rebase one upstream of that commit
>
>    ```
>    $ git rebase -i 2f5b416~
>    ```
>
>    Mark it for editing
>
>    ```
>    edit 2f5b416 admin/ansible-galaxy/0022: Add production facing vars
>    pick a352bec admin/ansible-galaxy/0023: Add nginx x-accel-redir and g-i-g webhook config to nginx
>    pick 59f9ff0 admin/singularity/0000: Add golang and singulary ansible roles
>    pick f38ddbf admin/singularity/0001: Configure golang and singularity
>    ```
>
>    It should stop there
>
>    ```
>    Stopped at 2f5b416...  admin/ansible-galaxy/0022: Add production facing vars
>    You can amend the commit now, with
>
>      git commit --amend
>
>    Once you are satisfied with your changes, run
>
>      git rebase --continue
>    ```
>
> 5. Open your file, make your change. Here I'm showing the change I made with `git diff`
>
>    ```
>    $ git diff | cat
>    diff --git a/group_vars/galaxyservers.yml b/group_vars/galaxyservers.yml
>    index 15fbf1e..8fe4343 100644
>    --- a/group_vars/galaxyservers.yml
>    +++ b/group_vars/galaxyservers.yml
>    @@ -53,6 +53,8 @@ galaxy_config:
>         expose_user_name: true
>         expose_dataset_path: true
>         expose_potentially_sensitive_job_metrics: true
>    +    # NFS workarounds
>    +    retry_job_output_collection: 3
>         # Debugging
>         cleanup_job: onsuccess
>         allow_user_impersonation: true
>    ```
>
>
> 6. The status probably looks like this, the file that was changed in that commit is marked as modified
>
>    ```
>    $ git status
>    interactive rebase in progress; onto 7284036
>    Last command done (1 command done):
>       edit 2f5b416 admin/ansible-galaxy/0022: Add production facing vars
>    Next commands to do (46 remaining commands):
>       pick 3c6a72e admin/ansible-galaxy/0023: Add nginx x-accel-redir and g-i-g webhook config to nginx
>       pick a893955 admin/singularity/0000: Add golang and singulary ansible roles
>      (use "git rebase --edit-todo" to view and edit)
>    You are currently editing a commit while rebasing branch 'main' on '7284036'.
>      (use "git commit --amend" to amend the current commit)
>      (use "git rebase --continue" once you are satisfied with your changes)
>
>    Changes to be committed:
>      (use "git restore --staged <file>..." to unstage)
>            modified:   group_vars/galaxyservers.yml
>    ```
>
> 7. Ammend the commit, and continue the rebase
>
>    ```
>    $ git commit --amend
>    [detached HEAD ed290de] admin/ansible-galaxy/0022: Add production facing vars
>     Author: The Galaxy Training Network <galaxytrainingnetwork@gmail.com>
>     Date: Mon Feb 15 14:06:56 2021 +0100
>     1 file changed, 22 insertions(+)
>    $ git rebase --continue
>    Successfully rebased and updated refs/heads/main.
>    ```
>
> 8. Import the changes back in the GTN repository
>
>    ```
>    ./bin/knit-automated.sh import
>    ```
>
{: .hands_on}


The above shows commands with outputs, the rest will be more abbreviated

## Upgrading an existing tutorial

> <question-title>Example problem</question-title>
> I have a new tutorial in the schedule and I want to upgrade it to use this feature
{: .question}

> <hands-on-title>Make the change you want to see in the GAT</hands-on-title>
>
> 1. Start by exporting
>
>    ```
>    $ bash bin/knit-automated.sh export
>    ```
>
> 2. Build the repository... **up to the point where your tuto is**
>
>    ```
>    [hxr@mk:/tmp/git-gat]$ git init
>    [hxr@mk:/tmp/git-gat]$ git am -3 0* 1* 2*
>    Applying: admin/ansible-galaxy/0000: Add requirements
>    applying to an empty history
>    Applying: admin/ansible-galaxy/0001: Add ansible.cfg
>    Applying: admin/ansible-galaxy/0002: Add hosts
>    Applying: admin/ansible-galaxy/0003: Add initial group variables file
>    Applying: admin/ansible-galaxy/0004: Add initial galaxy playbook
>    Applying: admin/ansible-galaxy/0005: Add pip, miniconda, galaxy to playbook
>    Applying: admin/ansible-galaxy/0006: Configure miniconda and galaxy
>    Applying: admin/ansible-galaxy/0007: Configure galaxy config
>    [...]
>    ```
>
> 3. Go diff-by-diff in your tutorial, making the change, and then making a new commit on top for each diff. For each of these, add the `{: data-commit="Something usful"}` below the commit.
>
>    Do they have a `{ % endraw % }`? then move that above the backticks.
>
> 4. Format them as patches
>
>    ```
>    git format-patch <the last commit before you started committing>
>    ```
>
> 5. Knit them together
>
>    ```
>    python bin/knit.py topics/admin/tutorials/<yourtuto>/tutorial.md --patches /tmp/<the-patches-you-exported>*
>    ```
>
> 6. That's part one done! Now we need to finish the rest of the patches.
>
> 7. Update `./bin/knit-automated.sh` to include your tutorial in the right order.
>
> 8. Go through an export-import cycle
>
>    ```
>    $ rm -rf /tmp/git-gat/
>    $ ./bin/knit-automated.sh export
>    $ cd /tmp/git-gat/
>    $ git am -3 *.patch # Resolve any conflicts
>    $ cd -
>    $ ./bin/knit-automated.sh import
>    ```
>
> 9. Done!
{: .hands_on}

## I want to add a new commit

> <hands-on-title>Make the change you want to see in the GAT</hands-on-title>
>
> 1. Add your commit where it belongs in the tutorial
>
>    <pre>
>    {% raw %}
>    { raw }
>    ```diff
>    --- a/requirements.yml
>    +++ b/requirements.yml
>    @@ -22,3 +22,7 @@
>       version: 0.0.2
>     - src: galaxyproject.slurm
>       version: 0.1.3
>    +- name: usegalaxy_eu.rabbitmq
>    +  version: 0.1.0
>    +- src: galaxyproject.pulsar
>    +  version: 1.0.7
>    { endraw }
>    ```
>    {: data-commit="Add requirements"}
>    {% endraw %}
>    </pre>
>
>    Use your imagination to pretend the raw/endraw tags are correct with the missing percent signs.
>
> 2. Now in the above diff we've written some --- and +++ and more metadata we don't really have. So: just make it up, the commit will fail to apply, and we'll go fix it. You can really write *anything* here you want, and the diff will fail which is what we want.
>
> 3. Now, go apply the commits
>
>    ```
>    $ ./bin/knit-automated.sh export
>    $ cd /tmp/git-gat/
>    $ git am -3 *.patch # Resolve any conflicts
>    ```
>
> 4. This will fail. Here's how to fix it:
>
>    1. Run `git am --show-current-patch` to see the above diff you've inserted (or the otherwise incorrect diff that's a downstream result of your new change).
>    2. Make the correct change to the file on disk.
>    3. `git add <path>`
>    3. `git am --continue`
>
> 5. You'll probably have to do this quite a few more times, diffs have a habit of affecting downstream diffs (e.g. changing a version number or being included in the diff context.)
>
> 6. Once you've finished all of them, you'll just `./bin/knit-automated.sh import` and commit the resulting changes!
{: .hands_on}

The alternative to the above approach is

1. Export
2. git rebase and make the corresponding change in the history
3. Let the rebase finish
4. Insert a 'dummy' diff (raw/open code+diff/endraw/close code/data-commit tag) without any content and then run `import` which should insert it in the correct place.

# Errors that can occur and how to fix them


## sha1 information is lacking or useless

This is because there's a mismatch in the commits you're trying to apply and what the merge commit is seeing. Usually you won't notice. **fix by rebasing, and roundtripping the commits.**

```
Applying: admin/pulsar/0012: Send bwa and bwa-mem to pulsar
Tagging step-17
Committing 18-gxadmin-commit-0000-add-requirement.patch
18-gxadmin-commit-0002-add-the-gxadmin-role.patch
Applying: admin/gxadmin/0000: Add requirement
error: sha1 information is lacking or useless (requirements.yml).
error: could not build fake ancestor
Patch failed at 0001 admin/gxadmin/0000: Add requirement
hint: Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
```

Here you grumble at how picky it is.

```
$ git am --show-current-patch
From: The Galaxy Training Network <galaxytrainingnetwork@gmail.com>
Date: Mon, 15 Feb 2021 14:06:56 +0100
Subject: admin/gxadmin/0000: Add requirement


--- a/requirements.yml
+++ b/requirements.yml
@@ -26,3 +26,5 @@
   version: 0.1.0
 - src: galaxyproject.pulsar
   version: 1.0.6
+- src: galaxyproject.gxadmin
+  version: 0.0.8
--
2.25.1
```

That looks right, doesn't it? But if we grep we'll see

```
$ ag -Q '1.0.6' topics/admin
topics/admin/tutorials/gxadmin/tutorial.md
55:>       version: 1.0.6

topics/admin/tutorials/monitoring/tutorial.md
80:>       version: 1.0.6

topics/admin/tutorials/pulsar/tutorial.md
149:>    +  version: 1.0.6
```

again right, but is it? No. Github is testing the merge commit. Rebase your commits, and now you'll see:


```
$ ag -Q '1.0.6' topics/admin
topics/admin/tutorials/gxadmin/tutorial.md
55:>       version: 1.0.6

topics/admin/tutorials/monitoring/tutorial.md
80:>       version: 1.0.6

$ ag -Q '1.0.7' topics/admin
topics/admin/tutorials/pulsar/tutorial.md
149:>    +  version: 1.0.7
```

Aha, two different version. This is what breaks the hashes (hence the error message.)

Roundtripping the commits and fixing up everywhere there was a single character difference in the diff context (UGH) will fix it.


topic: contributing
tutorial: meta-analysis-data
title: Single Cell Publication - Data Analysis
difficulty: advanced
time_estimation: 1H
tools_json: []
requirements_json: ["[]"]
tags_json: ["Ruby"]
content_hash: 840ed36fe0fe17feace2e80923f5a6eb
last_modified: 2026-02-26T21:12:14.715804
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/meta-analysis-data/tutorial.html

content:
Extracting data from the GTN's Git history isn't that difficult, but it requires some internal knowledge of how the GTN's Jekyll-based codebase works. Here we'll document what we've done!

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Our imports and metadata (all merged github PRs), and the list of all historical names of single-cell tutorial folders.

The author recommends running this code in a 'Jekyll Console' context. Jekyll does not natively have support for a console, but [there is an open Pull Request to add it](https://github.com/jekyll/jekyll/pull/8164). We recommend you install this yourself to most easily run the following code. You can do that by:

> <hands-on-title>Installing Jekyll's Console</hands-on-title>
> 1. View [the open Pull Request to add it](https://github.com/jekyll/jekyll/pull/8164)
> 2. Download `lib/jekyll/commands/console.rb` to somewhere on your computer.
> 3. Find out where jekyll is installed:
>    
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > gem which jekyll
>    > ```
>    {: .code-in}
>    
>    > <code-out-title></code-out-title>
>    > ```bash
>    > /home/user/galaxy/training-material/.direnv/ruby/gems/jekyll-4.3.3/lib/jekyll.rb
>    > ```
>    {: .code-out}
>    
>    That means the commands should be in:
>    
>    > <code-in-title>Bash</code-in-title>
>    > ```bash
>    > ls /home/user/galaxy/training-material/.direnv/ruby/gems/jekyll-4.3.3/lib/jekyll/commands/
>    > ```
>    {: .code-in}
>    
>    > <code-out-title></code-out-title>
>    > ```bash
>    > build.rb  clean.rb  doctor.rb  help.rb  new.rb  new_theme.rb  serve  serve.rb
>    > ```
>    {: .code-out}
> 
> 4. Copy the `console.rb` you downloaded to that folder
> 
>     ```bash
>     cp ~/Downloads/console.rb $(dirname $(gem which jekyll))/jekyll/commands/
>     ```
> 
>     This was written with commands in case folks want to copy paste it, to reduce error.
>
> 5. Launch the console with 
>    
>    ```bash
>    jekyll console
>    ```
> 
{: .hands_on}

## Which PRs are Single Cell RPs?

```ruby
require 'yaml'
data = YAML.load_file('metadata/github.yml')
```

<!-- GTN:IGNORE:003 -->

Let's fetch data from within the GTN's infrastructure. You can see documentation for some of our APIs in the [RDoc](https://training.galaxyproject.org/training-material/gtn_rdoc/). E.g. here is how we document [`TopicFilter.list_materials_structured`](https://training.galaxyproject.org/training-material/gtn_rdoc/TopicFilter.html#method-c-list_materials_structured).

```ruby
# Obtain all single cell materials
mats = TopicFilter
 .list_materials_structured(site, 'single-cell')
 .map { |k, v| v['materials'] }
 .flatten
 .uniq { |x| x['id'] }

# And flatten them into a useful list of old folder names
@sc = mats.map{|x| x['ref_tutorials'].map{|t| t['redirect_from']} + x['ref_slides'].map{|t| t['redirect_from']}}
    .flatten.uniq
    .reject{|x| x=~ /\/short\//} # /short/ is a folder of redirects
    .map{|x| x.split('/')[1..-2].join('/')} # Remove the filename.

```

> <tip-title>Jekyll console only needed for this one command</tip-title>
> The `site` object is currently required for calculating this list of structured materials. This is only available in the Jekyll console, so you'll need to run this command in a Jekyll console.
> All of the subsequent steps can be run in a normal Ruby environment, but you might as well keep running it in the Jekyll console anyway.
{: .tip}

Let's go ahead and patch array to let us calculate a mean, because laziness is great actually

```ruby
class Array
  def mean
    return self.sum / (0.0 + self.length)
  end
end
```

Here's how we'll define what is or isn't a single cell tutorial, based on URL:

```ruby
def is_sc(path)
  p = path.gsub(/-ES/, '_ES').gsub(/-CAT/, '_CAT'); 

  if p =~ /\/single-cell\// then
    return 1
  end

  # If anything in @sc is a prefix for p, then it's a single-cell file.
  if @sc.any?{|sc| p.start_with?(sc)} then
    return 1
  end
  
  return 0
end
```

Now let's obtain everything that IS a single cell PR. Here we define everything with 50% or more of the files being "single cell" files, as a single cell PR. How did we arrive at 50%? We made some plots and spot-checked individual results to see what made sense.

```ruby
sc_prs = data
  .reject{|num, pr| pr['author']['login'] == 'github-actions'} # Remove all automation
  .map{|num, pr| 
    [
      num,
      pr['files']
        .reject{|f| f['path'].split('/')[2] == 'images' && f['path'] !~ /scrna/ }
        .reject{|f| f['path'] =~ /^assets/}
        .map{|f| is_sc(f['path'])}
      .mean
    ]
  }
  .reject{|num, sc| sc < 0.5}
  # Reject NaN
  .select{|num, sc| sc == sc}
```

Output number 1 done!

```ruby
File.open('dist.txt', 'w') do |f|
  f.puts "num\tdist"
  sc_prs.each do |num, sc|
    f.puts "#{num}\t#{sc}"
  end
end
```

## Classifying PR Content

Let's write a classifier for each file type, to enhance our statistics:

```ruby
def classify(path)
  if path =~ /tutorial[A-Z_]*?\.md/ then
    return 'tutorial'
  elsif path =~ /slides.*\.html/ then
    return 'slides'
  elsif path =~ /faqs.*md/ || path =~ /^snippets/ then
    return 'faq'
  elsif path =~ /metadata.yaml/ || path == 'CONTRIBUTORS.yaml' || path =~ /index.md$/ || path =~ /README.md$/ then
    return 'metadata'
  elsif path =~ /\/workflows\// then
    return 'workflows'
  elsif path =~ /data-(library|manager)/ then
    return 'data-library'
  elsif path =~ /.bib$/ then
    return 'bibliography'
  elsif path =~ /\/images\// then
    return 'image'
  elsif path =~ /tutorials\/.*md/ then
    return 'tutorial'
  elsif path =~ /_plugins/ || path =~ /^bin/ || path =~ /_layouts/ || path =~ /_include/ || path == '_config.yml' || path =~ /assets/ || path =~ /Gemfile/ || path =~/shared/ then
    return 'framework'
  elsif path =~ /metadata\/.*.yaml/ then
    return 'metadata'
  elsif path =~ /metadata\/.*.csv/ || path =~ /Dockerfile/ then
    return 'ignore'
  elsif path =~ /^news/ then
    return 'news'
  end

  # This will raise an exception which will ensure we catch the case where we
  # haven't defined a classification rule for a file yet.
  1/ 0
end
```

And now we'll classify each of the single cell pull requests by their file type:

```ruby
results = []
results << [
  "num", "path", "class", "additions", "deletions", "createdAt", "mergedAt"
]
sc_prs.each do |num, _|
  data[num]['files'].reject{|f| f['path'] =~ /test-data/}.each do |f|
    results << [
      num, f['path'], 
      classify(f['path']),
      f['additions'], f['deletions'],
      data[num]['createdAt'], data[num]['mergedAt']
    ]
  end
end
```

Output number 2!

```ruby
# save to file.csv
File.open('sc.tsv', 'w') do |f|
  results.each do |r|
    f.puts r.join("\t")
  end
end
```

Preview of that data:

num | path | class | additions | deletions | createdAt | mergedAt
--- | ---- | ----- | --------- | --------- | --------- | --------
5484 | topics/single-cell/faqs/single_cell_omics.md | faq | 3 | 3 | 2024-10-29T12:24:51Z | 2024-10-29T12:47:23Z
5473 | topics/single-cell/tutorials/alevin-commandline/tutorial.md | tutorial | 48 | 47 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case-jupyter_basic-pipeline/tutorial.md | tutorial | 3 | 2 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_FilterPlotandExploreRStudio/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_FilterPlotandExplore_SeuratTools/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_JUPYTER-trajectories/tutorial.md | tutorial | 6 | 5 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_alevin-combine-datasets/tutorial.md | tutorial | 4 | 4 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_alevin/tutorial.md | tutorial | 3 | 2 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_basic-pipeline/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_monocle3-rstudio/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_monocle3-trajectories/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5473 | topics/single-cell/tutorials/scrna-case_trajectories/tutorial.md | tutorial | 1 | 0 | 2024-10-25T11:24:40Z | 2024-10-28T12:59:16Z
5447 | topics/single-cell/tutorials/GO-enrichment/tutorial.md | tutorial | 7 | 3 | 2024-10-11T11:22:34Z | 2024-10-11T13:36:22Z
5445 | topics/single-cell/tutorials/scrna-case-cell-annotation/slides.html | slides | 1 | 1 | 2024-10-11T10:52:08Z | 2024-10-12T15:44:00Z
5443 | topics/single-cell/faqs/single_cell_omics.md | faq | 4 | 2 | 2024-10-11T10:16:18Z | 2024-10-12T15:43:24Z
5416 | topics/single-cell/metadata.yaml | metadata | 4 | 2 | 2024-10-07T22:26:36Z | 2024-10-12T15:35:11Z
5416 | topics/single-cell/tutorials/GO-enrichment/tutorial.md | tutorial | 1 | 0 | 2024-10-07T22:26:36Z | 2024-10-12T15:35:11Z

## Contributors over time

Additionally we want to figure out how our contributions and contributors changed over time.

```ruby
require 'yaml'
require 'date'
require 'pp'
```

Here are the current classifications of contributors:

```ruby
KEYS = %w[authorship editing testing ux infrastructure translation data]
```

Let's get all the single cell tutorials:

```ruby
tutorials = Dir.glob("topics/single-cell/tutorials/*/tutorial.md")
```

We'll want to get all the timepoints from 2019 (when single cell was added) to
2025, so we'll setup an empty data structure for this.

**NB** This is NOT the best solution, this is just a simple brute force solution because the runtime is fine actually.

```ruby
timepoints = []
(2019..2025).each do |year|
  (1..12).each do |month|
    s = "#{year}-#{month}-01T00:00:00Z"
    timepoints << [
      s,
      DateTime.parse(s).to_time.to_i
    ]
  end
end
contribs_over_time = timepoints.map{|n, t| [t, KEYS.map{|k| [k, []]}.to_h]}.to_h
```

Ok, let's get the history of each tutorial

```ruby
tutorials.each do |tutorial|
  # if tutorial !~ /bulk/
  #   next
  # end

  git=`git log --follow --name-only --format="GTN_GTN %H %at" #{tutorial}`
  commits = git.split("GTN_GTN ")
  commits.reject!{|c| c.empty?}
  commits.map!{|c| 
    res = c.gsub(/\n+/, "\t").split(/\t/)
    if res.size > 2
      puts "ERROR: #{res}"
    end

    hash = res[0].split(' ')[0]
    time = res[0].split(' ')[1].to_i

    f = res[1]
    contents_at_time = `git show #{hash}:#{f}`
    begin
      contents_meta = YAML.load(contents_at_time)
    rescue
      next
    end

    if contents_meta.nil?
      next
    end

    if contents_meta.key?("contributors")
      c = {
        'authorship' => contents_meta["contributors"],
      }
    else
      c = contents_meta["contributions"]
    end

    squashed_i = DateTime.parse(Time.at(time).strftime("%Y-%m-01T00:00:00Z")).to_time.to_i

    {
      :hash => hash,
      :time => time,
      :date => Time.at(time),
      :sqsh => squashed_i, # The time rounded to the month
      :path => res[1],
      :role => c
    }
  }

  # For every commit
  commits.reverse.compact.each do |c|
    KEYS.each do |k|
      # For every role
      if c[:role].key?(k)
        # add to contribs now and at every time point in the future
        now_and_future_keys = contribs_over_time.keys.select{|t| t >= c[:sqsh] }
        now_and_future_keys.each do |t|
          contribs_over_time[t][k] << c[:role][k]
          contribs_over_time[t][k].flatten!
          contribs_over_time[t][k].uniq!
        end
      end
    end
  end
end
```

See, terrible, but it works!

```ruby
contribs_over_time.reject!{|k, v| v.values.all?{|vv| vv.empty?}}
pp contribs_over_time

File.open("sc-roles.tsv", "w") do |f|
  f.write("date\tarea\tcount\tcontributors\n")

  KEYS.each do |k|
    f.write(contribs_over_time.map{|date, roles| [date, roles[k]]}.map{|date, contribs| "#{Time.at(date).strftime("%Y-%m-01")}\t#{k}\t#{contribs.count}\t#{contribs.join(',')}"}.join("\n"))
  end
end
```

## All contributions over time

The plot of all new items over time was requested, and since the git log approach is so deeply ugly and slow, let's do something slightly different.

The date that Single Cell Tutorials, FAQ, Video, News, Events were added is requested.

The GTN produces RSS [feeds of all new content]({% link feeds/index.md %}), as well as feeds subsetted by their area (e.g. the [single cell feed](https://training.galaxyproject.org/training-material/feeds/single-cell-all.i.w.html) or the [single cell grouped by month](https://training.galaxyproject.org/training-material/feeds/single-cell-month.w.html)) so we can use that to get a list of all new single cell content over time.

```ruby
require 'net/http'
require 'date'
require './_plugins/util'
require 'uri'

# Ruby's stdlib for doing web requests is ... not great.
def request(url)
  uri = URI.parse(url)
  request = Net::HTTP::Get.new(uri)
  req_options = {
    use_ssl: uri.scheme == 'https',
  }
  Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
    http.request(request)
  end
end
```

Let's look though the data we get back:

```ruby
feed = request('https://training.galaxyproject.org/training-material/feeds/single-cell-month.xml').body
# The body of the feed has structure but it's slightly inconvenient.
# The URLs however...
# <a href="https://training.galaxyproject.org/training-material/topics/single-cell/tutorials/scrna-preprocessing/workflows/scrna_mp_celseq.html?utm_source=matrix&amp;utm_medium=newsbot&amp;utm_campaign=matrix-news">CelSeq2: Multi Batch (mm10) (February 22, 2019)</a>
#
# That has everything and a date, can just figure out what 'type' of thing it was by the folder/name.

def classify(url)
  if url =~ /tutorial(_[A-Z_]*)?\.html/
    return 'tutorial'
  elsif url.include? '/events/'
    return 'event'
  elsif url =~ /slides(_[A-Z_]*)?\.html/
    return 'slide'
  elsif url.include? '/faqs/'
    return 'faq'
  elsif url.include? '/workflows/'
    return 'workflow'
  elsif url.include? '/news/'
    return 'news'
  end

  p url

  1/0
end

out = []

feed.scan(/<a href="([^"]+)">([^<]+)<\/a>/).each do |url, title|
  next if url.include? 'gtn-standards-rss.html'
  date = title.match(/\(([^)]+)\)$/)[1]
  date = Date.parse(date).to_s
  out << [date, classify(url)]
end
```

And that's almost done! Let's add in the video data as well:

```ruby
# Let's add in videos, they're stored in the tutorials. Thankfully these have a
# recording date we can use.
tutos = Dir.glob("topics/single-cell/tutorials/*/tutorial.md")
tutos.each do |tuto|
  meta = safe_load_yaml(tuto)
  next if meta['recordings'].nil? || meta['recordings'].empty?

  meta['recordings'].each do |rec|
    out << [rec['date'], 'video']
  end
end
```

and write it out:

```ruby
File.open("single-cell-over-time.tsv", "w") do |f|
  f.puts "date\ttype"
  out.sort_by{|d, t| d}.each do |date, type|
    f.puts "#{date}\t#{type}"
  end
end
```

With that, I think we're done! Ready to plot.

> <tip-title>Do you want to re-do this analysis? For your community?</tip-title>
> [Let us know](https://github.com/galaxyproject/training-material/issues/new?title=GTN%20Metadata%20reuse) and we can perhaps generalise this dataset such that you could more easily download it for your own community.
{: .tip}


topic: contributing
tutorial: create-new-tutorial-technical
title: Tools, Data, and Workflows for tutorials
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 2e492bf7cda91b1e9e4fb841314d1f95
last_modified: 2026-02-26T21:12:14.704008
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/create-new-tutorial-technical/tutorial.html

content:
# Building a Galaxy instance specifically for your training


To be able to run the tutorial, we need a Galaxy instance where all of the needed tools and data are available. Thus we need to describe the needed technical infrastructure.

The files we define in this tutorial will be used to automatically build a Docker Galaxy flavour, and also to test if a public Galaxy instance is able to run the tool.

In this tutorial, you will learn how to create a virtualized Galaxy instance, based on Docker, to run your training - either on normal computers or cloud environments.

> <agenda-title></agenda-title>
>
> In this tutorial, we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Extracting workflows

Once the tutorial is ready, we need to develop a workflow that represents the steps taken in the tutorial, and then extract these workflow(s) and add them to the `workflows` directory in the tutorial. Additionally we will need to add some explanation about the workflow(s) in a `README.md` file

> <hands-on-title>Extract the workflow</hands-on-title>
>
> 1. Add the topic name as a Tag and the tutorial title as Annotation/Notes to the workflow using the workflow editor.
> 2. Download the workflow for the tutorial
> 3. Save it in the `workflow` directory of the tutorial
> 4. Check that your `workflow` directory has an `index.md` with the contents:
>
>    ```yaml
>    ---
>    layout: workflow-list
>    ---
>    ```
{: .hands_on}

## Testing the workflow (recommended)

Workflow testing is a great way to get feedback that your tutorial can be run successfully on a given server. When you're giving a training this can provide peace of mind, not only are the tools installed (as is indicated by the badges we provide) but they also work.

Given the workflow you created above and have included in the tutorial folder, you'll need to create a corresponding `-test.yml` file.

> <hands-on-title>Creating the `-test.yml` file for your workflow</hands-on-title>
>
> 1. Find the correct name for the file; if your workflow was `unicycler.ga`, then your test file should be `unicycler-test.yml`, they need to share the same prefix.
>
> 2. Create the following structure:
>
>    ```yaml
>    ---
>    - doc: Test sample data for the workflow
>      job:
>        an_input_file:
>          class: File
>          location: https://....
>          filetype: fasta
>      outputs:
>        ffn:
>          asserts:
>            - that: has_text
>              text: ">A"
>            - that: has_text
>              text: ">B"
>    ```
>
>
{: .hands_on}

You'll need to edit the `job` and `outputs` sections according to your workflow's inputs and outputs. Additionally you will need to edit the steps of your workflow `.ga` file appropriately.

### Inputs

Your workflow **must** use "Data Inputs" for each input dataset. For each of these input steps in the `.ga` file, you'll need to do the following:

1. Edit the `label`
2. Edit the `name`
3. Edit the `inputs[0].name`
4. Edit the `tool_state`

In a normal workflow you have exported from Galaxy, you'll see something like

```json
{
    "id": 0,
    "input_connections": {},
    "inputs": [
        {
            "description": "",
            "name": "patient1_ChIP_ER_good_outcome.bam"
        }
    ],
    "label": null,
    "name": "Input dataset",
    "outputs": [],
    "position": {
        "left": 10,
        "top": 10
    },
    "tool_id": null,
    "tool_state": "{\"name\": \"patient1_ChIP_ER_good_outcome.bam\"}",
    "tool_version": null
}
```

You should synchronize the aforementioned fields so it looks like this:

```json
{
    "id": 0,
    "input_connections": {},
    "inputs": [
        {
            "description": "",
            "name": "good_outcome"
        }
    ],
    "label": "good_outcome",
    "name": "good_outcome",
    "outputs": [],
    "position": {
        "left": 10,
        "top": 10
    },
    "tool_id": null,
    "tool_state": "{\"name\": \"good_outcome\"}",
    "tool_version": null
}
```

This will allow you to specify `good_outcome` in your job to load a file:

```
- doc: ...
  job:
    good_outcome:
      class: File
      location: ...
      filetype: ...
```

The filetype should be the Galaxy datatype of your file, for example `fastqsanger`, `tabular`, `bam`.

### Outputs

For the outputs the process is somewhat simpler:

1. Identify a step, the outputs of which you would like to test
2. Convert the relevant `outputs` to `workflow_outputs`

   In a normal workflow you see

   ```json
   {
       "outputs": [
           {
               "type": "txt",
               "name": "ofile"
           },
           {
               "type": "txt",
               "name": "ofile2"
           }
       ],
       "workflow_outputs": []
   }
   ```

   If you want to test the contents of `ofile`, you should change it to

   ```json
   {
       "outputs": [
           {
               "type": "txt",
               "name": "ofile"
           },
           {
               "type": "txt",
               "name": "ofile2"
           }
       ],
       "workflow_outputs": [
           {"output_name": "ofile", "label": "my_output"}
       ]
   }
   ```

3. You can now use the label you chose (here `my_output`) in your test case:

   ```yaml
   - doc:
     job: ...
     outputs:
       my_output:
         asserts:
           has_text:
             text: 'some-string'
   ```

### Running the Tests

You can test the file you've written with the following command and a recent version (>=0.56.0) of planemo:

```console
planemo test \
	--galaxy_url "$GALAXY_URL" \
	--galaxy_user_key "$GALAXY_USER_KEY" \
	--no_shed_install \
	--engine external_galaxy \
	workflow.ga
```

Planemo will autodetect that the `workflow-test.yml` file and load that for the testing.

### Skipping Testing in the GTN

If for some reason you want to skip this workflow being tested in the GTN,
please add a comment with `GTN_RUN_SKIP_REASON` in the `-test.yml` file stating
the reason it is skipped.

This will also exempt you from writing output tests.

A good use case for this is you want to provide a working test, but the workflow takes upwards of 6 hours to execute (e.g. large download jobs.)

# Creating the `data-library.yaml` (recommended)

The datasets needed for a tutorial can also be integrated in the Galaxy instance inside of data libraries. These allow the datasets to be easily shared with all users of a Galaxy instance. Additionally it lets trainees avoid each re-downloading the input data.

These datasets are described in the `data-library.yaml` files:

```yaml
---
destination:
  type: library
  name: GTN - Material
  description: Galaxy Training Network Material
  synopsis: Galaxy Training Network Material. See https://training.galaxyproject.org
items:
- name: Title of the topic
  description: Summary of the topic
  items:
  - name: Title of the tutorial
    items:
    - name: 'DOI: 10.5281/zenodo....'
      description: latest
      items:
      - info: https://doi.org/10.5281/zenodo....
        url: https://zenodo.org/records/URL/files/path/to/input
        ext: galaxy-datatype
        src: url
```

> <hands-on-title>Creating the `data-library.yaml`</hands-on-title>
>
> 1. Copy the Zenodo link
> 2. Generate the `data-library.yaml` file and update the tutorial metadata with the link:
>
>    ```
>    $ planemo training_fill_data_library \
>             --topic_name "my-topic" \
>             --tutorial_name "my-new-tutorial" \
>             --zenodo_link "URL to the Zenodo record"
>    ```
>
> 3. Check that the `data-library.yaml` has been generated (or updated)
> 4. Check that the Zenodo link is in the metadata at the top of the `tutorial.md`
{: .hands_on}

# Creating the `data-manager.yaml` (optional)

Some of the tools may require specific databases, specifically prepared for the tool. In this case, some Galaxy tools come with "data managers" to simplify this process.

If you need such data managers for your training, then you should describe how to run them in the `data-manager.yaml` file:

```yaml
data_managers:
    - id: url to data manager on ToolShed
      params:
        - 'param1': '{{ item }}'
        - 'param2': 'value'
      # Items refer to a list of variables you want to run this data manager. You can use them inside the param field with {{ item }}
      # In case of genome for example you can run this DM with multiple genomes, or you could give multiple URLs.
      items:
        - item1
        - item2
      # Name of the data-tables you want to reload after your DM are finished. This can be important for subsequent data managers
      data_table_reload:
        - all_fasta
        - __dbkeys__
```

# Creating the Galaxy Interactive Tour (optional)

A Galaxy Interactive Tour is a way to go through an entire analysis, step by step inside Galaxy in an interactive and explorative way.
It is a great way to help users run the tutorial directly inside Galaxy. To learn more about creating a Galaxy tour please have a look at our [dedicated tour training]({{site.baseurl}}/topics/contributing/tutorials/create-new-tutorial-tours/tutorial.html).

# Testing the technical infrastructure

Once we have defined all the requirements for running the tutorial, we can test these requirements, either in a locally running Galaxy or in a Docker container. Please see our tutorial about [Setting up Galaxy for Training]({{site.baseurl}}/topics/teaching/tutorials/setup-galaxy-for-training/tutorial.html) about how to test your tutorial requirements.


# Conclusion


topic: contributing
tutorial: rendering_gtn
title: Preview the GTN website as you edit your training material
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["Choose your own Adventure", "GitPod", "CodeSpaces", "Command-line"]
content_hash: b40893f6e2c530cfec14c456fb48e1f3
last_modified: 2026-02-26T21:12:14.723425
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/rendering_gtn/tutorial.html

content:
If you are working on training materials, you will likely want to preview your changes as you go! You have a few options on how to do this.

{% include _includes/cyoa-choices.html option1="Codespaces" option2="Gitpod" option3="Command-line" option4="Container-based" default="Codespaces"
       text="We recommend using Codespaces. We used to use Gitpod, however it loads a bit slower and the interface changed as of spring 2025, so the tutorial may be out of date. **CodeSpaces and GitPod are both online methods**; only the command-line option requires you to install things on your machine." %}

<div class="Codespaces" markdown="1">

If you are working on your own training materials and want preview them online without installing anything on your computer, you can do this using GitHub CodeSpaces! Everybody gets 60 free hours of CodeSpaces per month.



# Launching CodeSpaces


> <hands-on-title>Setting up Codespaces</hands-on-title>
>
> 1. **Create a fork** of the [GTN GitHub repository](https://github.com/galaxyproject/training-material)
>
>    {% snippet topics/contributing/faqs/github-fork-gtn.md %}
>
> 2. Already have a fork of the GTN? Make sure it is up to date.
>
>    {% snippet topics/contributing/faqs/github-fork-master-main.md %}
>
>    {% snippet topics/contributing/faqs/github-fork-sync.md %}
>
> 2. **Navigate to your fork** of the GTN
>
> 3. Click on the green **Code** button
>
> 4. At the top, switch to the **CodeSpaces** tab
>    ![the codespaces tab of the menu](images/codespaces-launch.png)
>
> 5. Click on **Create codespace on main**
>    - Note: if you switch to a specific branch in GitHub first, you can create a codespace for that branch
>
> 6. This will setup a [Visual Studio Code](https://code.visualstudio.com/) environment for you
>    - It may take a couple minutes to finish setting everything up
>    - In this environment you can also build the GTN website to preview your changes
>    - When everything is ready, you should see something like this:
>
>    ![screenshot of Codespaces just after startup](images/codespaces-home.png)
{: .hands_on}

# The VScode environment

Let's have a closer look at your CodeSpaces window:

- **Left:** Here you see all the files in the GTN repository
- **Top:** This is the main window where you can view and edit files
- **Bottom:** Terminal window. Here you can type commands (e.g. to build the website preview) and read output and error messages.

![VScode interface](images/codespaces-home.png)


# Build and preview the GTN website

Before we start making any changes, let's build the website and get a live preview.

> <hands-on-title>Setting up GitPod</hands-on-title>
>
> 1. In the terminal window (bottom), type the following command:
>    `make preview`
>
>    - This will take 2-3 minutes to complete
>
>    ![output in the terminal after issuing the make preview command](images/codespaces-make-preview.png)
>
> 3. When the build process is finished, a preview window will automatically open (at the top right)
>    - The preview will show the GTN 404 (codespace doesnt know what to show)
>    - Just click on **Return to homepage**.
>
>    > <tip-title>Window not opening? </tip-title>
>    > If the preview window doesn't open for you, or if you close it and want to reopen it, you can always do so as follows:
>    > 1. Go to the **Ports** tab of the bottom panel
>    >
>    >    ![screenshot of the tabs on the bottom panel](images/codespaces-ports-tab.png)
>    >
>    > 2. Hover over the link in the **Forwarded Address** column, 3 icons should appear
>    >
>    >    ![screenshot of the 3 icons](images/codespaces-ports-preview-buttons.png)
>    >
>    > 3. Click on:
>    >    - The **world/globe icon** to open the GTN preview in a new window, or
>    >    - Or, click on the **window icon** to the right of the globe icon to preview the GTN in a tab inside the codespaces environment
>    {: .tip}
>
> 4. If you opened the GTN preview inside the codespace, your window will now look something like this:
>
>    ![screenshot of the codespace with the preview editor opend to the GTN homepage](images/codespaces-preview-editor.png)
>
>    > <tip-title>Not opening?</tip-title>
>    > ![screenshot of firefox permissions dialog which shows blocked windows and an allow menu](../../images/gitpod_popup.png)
>    > Some browsers block popups by default, you may need to allow CodeSpaces to show popups in your browser.
>    {: .tip}
>
{: .hands_on}



# Editing Training Materials on CodeSpaces

Now that you have the codespace environment working and we have a live preview up, let's make some changes to the GTN materials and get an instant preview.


**Scenario:** You have spotted a typo in one of the tutorials, and would like to fix this and see the resulting GTN webpage.


> <hands-on-title>Make and view changes</hands-on-title>
>
> 1. In the preview of the GTN website, open the following tutorial:
>    - Topic: "Introduction to Galaxy Analyses" topic
>    - Tutorial: "A Short Introduction to Galaxy""
>    - We will edit this tutorial and watch the live preview window for the effects
>
>
> 2. On the file browser on the left, open the following file:
>
>    ```
>    topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>    ```
>
> 3. Change the title of the tutorial
>    - **From:** "A Short Introduction to Galaxy"
>    - **To:** "A Short and Cool Introduction to Galaxy"
>
>    ![we changed the title of the tutorial in the text editor window](images/codespaces-tutorial-edited.png)
>
> 4. Save the file
>    - **CTRL+S** to save the file
>    - You should immediately see a message in the terminal saying "regenerating". CodeSpaces has detected your changes and is rebuilding the website.
>
>    ![the terminal shows a message stating the website is being regenerated](images/codespaces-regenerating.png)
>
> 5. Move to the top right panel where the GTN is previewed and refresh the website
>    - {% icon galaxy-refresh %} Refresh button in front of the address bar of the preview panel
>    - You can also open the preview in it's own brower tab, using the {% icon galaxy_instance %} button at the top-right corner of the preview window. Then you can reload the page the regular way (e.g. <kbd>F5</kbd> or <kbd>ctrl + r</kbd> or the reload button in the browser)
>
>    > <tip-title> Reload not working? </tip-title>
>    > It is possible that this reload button gives you the 404 again, in that case there are 2 solutions
>    > 1. Right-click in the preview panel, and choose
>    >    - Chrome: "Reload Frame"
>    >    - Firefox: "This Frame -> Reload Frame"
>    > 2. Open the preview in it's own browser tab
>    >    - Click the {% icon galaxy_instance %} button at the top-right corner of the preview window
>    {: .tip}
>
> 6. You should see the change you made:
>
>    ![The updated preview with our changed tutorial title](images/codespaces-reloaded.png)
>
{: .hands_on}


In this way you can edit files in the text editor, and see the effects in the website preview.


# Saving your changes back to GitHub

When you have finished your changes, it all looks good in the preview, you want to save your changes back to GitHub so that you can either continue later, or make a Pull Request to the GTN to contribute your changes.


> <hands-on-title>Comitting changes</hands-on-title>
> Before you can commit your changes you have to create a branch. You have two options to preform this task:
>  1. **Option 1: via the terminal**
>     - Hit <kbd>ctrl+c</kbd> if your preview was still running to stop it
>     - Create a new branch, commit your changes, push changes:
>
>     ```bash
>     git checkout -b fix-title
>     git commit -m "update tutorial title" topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>     git push origin fix-title
>     ```
>
> 2. **Option 2: via the web interface**
>    - Create a new branch:
>      -  On the bottom-left, click on the branch logo (probably labelled "main")
>         ![button on the bottom of the page with the current branch](images/codespaces-branch-change1.png)
>      - Enter `fix-title` as the name for your new branch (at top of window)
>         ![Addint a new branch](images/codespaces-branch-change2.png)
>      - Choose "+ Create new branch..." from the dropdown
>    - Commit changes:
>      - On the left menu, click on the "changed files" tab
>        ![button on the left menu to show changed files](images/codespaces-commit-1.png)
>      - You should see your changed file (`tutorial.md`)
>      - Click on the "+" icon next to the file we edited to *stage changes*
>        ![stage changes button](images/codespaces-commit-plus.png)
>      - Enter a commit message (top of window)
>      - Hit the checkmark icon below the massage to commit the changes
>   - Publish changes
>     - Click the cloud button at bottom left to publish your changes
>       ![publish changes button](images/codespace-publlish.png)
>
{: .hands_on}

Next, we will see these changes to on your branch/fork. We will do this from outside of our codespace.

> <hands-on-title>Pushing changes to GitHub</hands-on-title>
>
> 1. In your browser (outside of codespaces), navigate to the [GTN GitHub page](https://github.com/galaxyproject/training-material)
> 2. GitHub will helpfully show you any recent branches you've pushed to your fork
>    - Click the **Compare & pull request** button to create a PR for your changes
>    ![compare and pull request button on the new branch](images/codespaces-compare-pr.png)
{: .hands_on}


# Closing your CodeSpace

Everybody gets 60 hours per month for free on CodeSpaces. Your codespace will automatically shut down after 30 minutes of inactivity,
but it is always a good idea to close your CodeSpace when you are finished with it, to conserve your quotum.


> <hands-on-title>Shutting down your CodeSpace</hands-on-title>
>
> 1. Return to the [GTN GitHub page](https://github.com/galaxyproject/training-material)
>
> 2. Click on the green **Code** button again
>
> 3. Under the Codespaces tab, you should see your running codespace
>
>    ![codespaces menu in github](images/codespaces-stop-1.png)
>
> 4. Click on the 3-dots menu to the right of your (randomly generated) codespace name
>
> 5. In this menu you can quit your codespace in two ways:
>    - **Stop codespace**: your changes will be kept and you can restart the codespace later
>    - **Delete** your codespace. Any changes you did not commit and push to GitHub are lost.
>      ![screenshot of the codespace options menu](images/codespaces-stop-2.png)
>
>    - In this menu you can also resume a stopped codespace by simply clicking **Open in Browswer**
>
{: .hands_on}


Congrats! You learned how to contribute to the GTN by using the CodeSpaces environment!

</div>

<div class="Gitpod" markdown="1">


If you are working on your own training materials and want preview them online without installing anything on your computer, you can do this using GitPod!



# Setting up Gitpod


> <hands-on-title>Setting up GitPod</hands-on-title>
>
> 1. **Create a fork** of the [GTN GitHub repository](https://github.com/galaxyproject/training-material)
>
>    {% snippet topics/contributing/faqs/github-fork-gtn.md %}
>
> 2. Already have a fork of the GTN? Make sure it is up to date.
>
>    {% snippet topics/contributing/faqs/github-fork-master-main.md %}
>
>    {% snippet topics/contributing/faqs/github-fork-sync.md %}
>
> 3. **Open** your browser and navigate to [gitpod.io/login](https://gitpod.io/login)
>    - Note: don't leave the `/login` part of the URL off, it will lead you to a different flavour of GitPod. We are using Gitpod classic
> 4. **Log in** with GitHub
>    ![GitPod sign-in menu with various options, including GitHub](../../images/gitpod/signin.png){: width="25%"}
> 5. Click on **Configure your own repository** under the Workspaces menu
>    ![GitPod homepage](../../images/gitpod/gitpod-home.png)
> 6. Under **Select a repository** choose your fork, e.g. `https://github.com/shiltemann/training-material`
>    ![screenshot of new workspace menu](../../images/gitpod/gitpod-workspace.png)
> 7. Click **continue**
>    - This will create an enviroment where you can make changes to the GTN and preview them
>    - **Note:** It can take quite some time to start up the first time (15-30 minutes)
>    - We can make future starts a lot faster using **prebuilds** (see tip box below), you can configure this now while you wait
>
>    ![progress indicator of GitPod first setup](../../images/gitpod/setup.png)
>
>    > <tip-title> Enabling Prebuilds for your GitPod workspace </tip-title>
>    > Prebuilds can speed up the launch of your GitPod workspace after the first time. When this is enabled,
>    > starting a workspace will check if there is a previous build to reuse. This should bring startup times
>    > down to 3-5 minutes.
>    >
>    > **Enable Prebuilds**
>    > 1. On the GitPod website, click on your name in the top left of the screen
>    > 2. Select **Repositories**
>    > 3. Add your fork as a repository
>    > 4. On the left, click **Prebuild settings**
>    > 5. Click the slider in front of **Prebuilds are disabled** to toggle the setting
>    > 6. Set **Branch Filter** to **Default Branch**
>    >
>    > ![screenshot of prebuild setting of our repository](../../images/gitpod/gitpod-prebuilds.png)
>    >
>    > **Run a prebuild now**
>    > 1. In GitPod, click on your name in the top left of the screen
>    > 2. Select **Prebuilds**
>    > 3. Click **Run a Prebuild**
>    > 4. **Select a Repository**: your fork
>    > 5. Click **Run Prebuild**
>    >    - This will also take a bit of time to complete
>    >    - In the future, it will automatically refresh this prebuild every 20 commits, so you only have to do this once
>    >
>    >
>    > **Note:** Prebuilds expire if you haven't used them in 7 days, so if you have taken a longer
>    > break from GTN you might have to wait longer again the first time. But otherwise it should
>    > bring the launch time down to around 3-5 minutes.
>    {: .tip}
>
> 8. Once the setup is finished, you should see a page something like this:
>    ![GitPod welcome screen](../../images/gitpod/welcomepage.png)
>
>    > <tip-title>Make this even easier with the GitPod browser extension!</tip-title>
>    >  1. To make this even easier, you can install the GitPod browser extension
>    >     - [Firefox](https://addons.mozilla.org/en-GB/firefox/addon/gitpod/)
>    >     - [Chrome or Edge](https://chrome.google.com/webstore/detail/gitpod-always-ready-to-co/dodmmooeoklaejobgleioelladacbeki)
>    >  2. Now it is just a single click to open any GitHub repository in GitPod
>    >     ![screenshot of the button the GitPod browser extension adds to GitHub repository ](../../images/gitpod/browser-extension.png)
>    {: .tip}
>
{: .hands_on}

# The GitPod Workspace

Let's have a closer look at the GitPod workspace:

- **Left:** Here you see all the files in the GTN repository
- **Top:** This is the main window where you can view and edit files
- **Bottom:** Terminal window. Here you can type commands (e.g. to build the website preview) and read output and error messages.

![GitPod welcome screen](../../images/gitpod/welcomepage.png)



# Build and preview the GTN website

Before we start making any changes, let's build the website and get a live preview.

> <hands-on-title>Setting up GitPod</hands-on-title>
>
> 1. In the terminal window (bottom), type the following command:
>    `make serve-gitpod`
>
>    ![output in the terminal after issuing the make serve-gitpod command](../../images/gitpod/command.png)
>
> 2. The first time you run this command, it will install some dependencies, so it may take a little bit of time (a few minutes)
>    - Next times will be a lot faster! (~30 seconds)
>
> 3. When the build process is finished, the terminal will show you a link, and a window should popup with a preview on the right.
>    - You may have to click the "Try Again" button if the preview was generated before the site building completed
>
>      ![gitpod with the GTN open in the right panel](../../images/gitpod/gitpod-launched.png)
>
>    - Click on the link in the terminal to see the GTN in full-screen: `Server address: http://127.0.0.1:4000/training-material/`
>
>    ![output in the terminal after the build process has completed ](../../images/gitpod/ready.png)
>
>    > <tip-title>Not opening?</tip-title>
>    > ![screenshot of firefox permissions dialog which shows blocked windows and an allow menu](../../images/gitpod_popup.png)
>    > Some browsers block popups by default, you may need to allow GitPod to show popups in order to click links.
>    {: .tip}
>
> 4. This will take you to a preview of the GTN website
>    - **Note:** It will take you the 404 page because GitPod doesn't know the location of the GTN homepage, but from here you can just click the homepage button.
>
{: .hands_on}


![screenshot of gitpod's code editor with a tutorial open, and the GTN preview on the right](../../images/gitpod.png)

# Editing Training Materials on GitPod

Now that you have the GitPod environment working and we have a live preview up, let's make some changes to the GTN materials and get an instant preview.


**Scenario:** You have spotted a typo in one of the tutorials, and would like to fix this and see the resulting GTN webpage.


> <hands-on-title>Make and view changes</hands-on-title>
>
> 1. In the preview of the GTN website, open the following tutorial:
>    - Topic: "Introduction to Galaxy Analyses" topic
>    - Tutorial: "A Short Introduction to Galaxy""
>    - We will edit this tutorial and watch the live preview window for the effects
>
>
> 2. On the file browser on the left, open the following file:
>
>    ```
>    topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>    ```
>
>    > <tip-title>Finding files faster?</tip-title>
>    > You can use the key combination <kbd>ctrl-p</kbd> to launch the file search dialog and simply enter the text `topics/introduction/tutorials/galaxy-intro-short/tutorial.md`
>    {: .tip}
>
> 3. Change the title of the tutorial
>    - **From:** "A Short Introduction to Galaxy"
>    - **To:** "A Short and Cool Introduction to Galaxy"
>
>    ![we changed the title of the tutorial in the text editor window](../../images/gitpod/changes.png)
>
>    - You should immediately see a message in the terminal saying "regenerating". GitPod has detected your changes and is rebuilding the website.
>
>    ![the terminal shows a message stating the website is being regenerated](../../images/gitpod/regenerating.png)
>
> 4. Move to the top right panel where the GTN is previewed and refresh the frame
>    - {% icon galaxy-refresh %} Refresh button in front of the address bar of the preview panel
>    - You can also open the preview in it's own brower tab, using the {% icon galaxy_instance %} button at the top-right corner of the preview window. Then you can reload the page the regular way (e.g. <kbd>F5</kbd> or <kbd>ctrl + r</kbd> or the reload button in the browser)
>
> 5. You should see the change you made:
>    ![The updated preview with our changed tutorial title](../../images/gitpod-changed.png)
>
{: .hands_on}


In this way you can edit files in the text editor, and see the effects in the website preview.


# Saving your changes back to GitHub

When you have finished your changes, it all looks good in the preview, you want to save your changes back to GitHub so that you can either continue later, or make a Pull Request to the GTN to contribute your changes.

> <hands-on-title>Save changes</hands-on-title>
>
> 1. **Option 1: via the terminal**
>    - Hit <kbd>ctrl+c</kbd> if your preview was still running to stop it
>    - Create a new branch, commit your changes, push changes:
>
>    ```bash
>    git checkout -b fix-title
>    git commit -m "update tutorial title" topics/introduction/tutorials/galaxy-intro-short/tutorial.md
>    git push origin fix-title
>    ```
>
> 2. **Option 2: via the web interface**
>    - Create a new branch:
>      -  On the bottom-left, click on the branch logo (probably labelled "main")
>         ![button on the bottom of the page with the current branch](../../images/gitpod/branch-change.png)
>      - Enter `fix-title` as the name for your new branch (at top of window)
>      - Choose "+ Create new branch..." from the dropdown
>    - Commit changes:
>      - On the left menu, click on the "changed files" tab
>        ![button on the left menu to show changed files](../../images/gitpod/changedfiles.png)
>      - Click on the "+" icon next to the file we edited to *stage changes*
>        ![stage changes button](../../images/gitpod/stagechanges.png)
>      - Hit the checkmark icon at the top to commit the changes
>      - Enter a commit message (top of window)
>   - Publish changes
>     - Click the cloud button at bottom left to publish your changes
>       ![publish changes button](../../images/gitpod/publlish.png)
>
>
> 3. Your changes are now saved to your fork, and you can make a PR via the GitHub interface.
>    - If you used option 1 and want to do that right away, you can click on the link in the terminal:
>      ![link in the terminal to open a PR](../../images/gitpod/push.png)
>
> > <tip-title>GitPod Permissions</tip-title>
> > If this is your first time using GitPod, you may need to grant permissions to let GitPod push to your GitHub fork
> > ![prompt to changes permissions](../../images/gitpod/access.png)
> > This will take you through a process to authorize GitPod, and set permissions
> > ![permission settings for GitPod](../../images/gitpod/permissions.png)
> > If it doesn't take you to GitPod's permission page because your browser blocks popups, you can set them [on this page](https://gitpod.io/integrations).
> {: .tip}
{: .hands_on}


# Closing your GitPod Workspace

Everybody gets 50 hours per month for free on GitPod, so it is wise to close your GitPod workspace when you are finished with it, to conserve your quotum.

> <hands-on-title>Shutting down your GitPod workspace</hands-on-title>
>
> 1. Click on the menu button at the top left corner of the screen
>    - Choose "Gitpod: Stop Workspace"
>
>    ![stop workspace option in the main menu](../../images/gitpod/stopworkspace.png)
>
{: .hands_on}

</div>

<div class="Command-line" markdown="1">

If you want to run the entire GTN material website locally or test your new training material you can do this!

Currently, the website is generated from the metadata and the tutorials using Jekyll, a simple static site builder.
We can use Jekyll to run a server to check if the tutorial is correctly added and rendered.


# Installation of the requirements

The first step is to install the needed tools inside a conda environment. A conda environment is a directory that contains a specific collection of packages. For example here to run the website, we need ruby, pandas, requests, etc. By creating a conda environment and installing the needed tools there, we do not affect your main installation.

This step has to be done once.

We also need to make sure that a couple of other utilities and build requirements are present: **git**, **curl** & **make**. The easiest way to install these is with your package manager of choice - i.e. yum, apt, brew etc.

> <hands-on-title>Install the requirements</hands-on-title>
>
> 1. Open a Terminal
> 2. Use your package manager to install `git`, `curl` and `make`
>
>    - For Debian/Ubuntu: `sudo apt update && sudo apt install git curl make`
>    - For Fedora/CentOs/RedHat: `sudo yum install git curl make`
>
> 3. Clone the training material GitHub repository:
>    - `git clone --depth 1 --branch main https://github.com/galaxyproject/training-material.git`
>    - Note: The GTN repo is quite large, therefore we recommend to use the `--depth 1 --branch main` flags here. This will clone only the main branch, and only the most recent revision, rather than the full history.
> 4. Navigate to the the cloned repository
>    - `cd training-material`
> 5. Set up the conda environment (see tip box below for an method without conda)
>
>     It will install some needed tools (ruby, nodejs, etc) in a protected environment, without interfering with the existing tools or versions.
>
>     1. Install conda (if not already installed): `make install-conda`
>     2. (You may need to exit the terminal and re-open for conda to be recognised. Navigate back to the same place.)
>     3. Create the `galaxy_training_material` conda environment: `make create-env`
>
> 6. Install Jekyll and related modules into the conda environment: `make install`
>
> > <details-title>Troubleshooting `libxml2` errors</details-title>
> > If you encounter an error about libxml2 on Linux, please try to install `libxml2-dev` (executing `sudo apt install libxml2-dev`) if on Debian/Ubuntu or `libxml2-devel` (executing `sudo yum install libxml2-devel`) if on Fedora/RedHat/CentOS, and re-run `make install` .
> {: .details}
>
{: .hands_on}




> <tip-title>[Advanced] Building GTN without Conda (using Ruby's RVM)</tip-title>
>
> If you do not want to use conda, you can also run jekyll directly using Ruby and RVM:
> 1. Install RVM (Ruby Version Manager) on your system ([instructions](https://rvm.io/rvm/install))
>    - For Ubuntu:
>      ```
>      sudo apt-get install software-properties-common
>      sudo apt-add-repository -y ppa:rael-gc/rvm
>      sudo apt-get update
>      sudo apt-get install rvm
>      sudo usermod -a -G rvm $USER
>      ```
> 2. Install a Ruby version
>    ```
>    rvm install ruby-3.2.4
>    ```
> 3. Create ruby GTN environment
>    ```
>    rvm use 3.2.4@gtn --create
>    ```
> 4. Install dependencies
>    ```
>    bundle install
>    ```
> 5. Run the GTN
>    - Always make sure your ruby environment is activated `rvm use 3.2.4@gtn` and up-to-date `bundle install`
>    - Get a GTN preview
>      ```
>      bundle exec jekyll serve --incremental
>      ```
>    - Quicker build preview (but certain features are disabled)
>      ```
>      bundle exec jekyll serve --strict_front_matter -d _site/training-material --incremental --config _config.yml,_config-dev.yml
>      ```
> 6. Changes now showing?
>    - Stop the preview, run `make clean`, `bundle install`, and rebuild it
{: .tip}



# Checking the website generation

Once Jekyll and its modules are installed in our conda environment, we can check the generation of the website locally:

> <hands-on-title>Checking the website generation locally</hands-on-title>
>
> 1. Run a local Jekyll server with `make serve-quick` (if you use a Macbook, try `make serve-quick HOST=127.0.0.1` for forcing Jekyll to use the standard local loopback address.)
> 2. Visualize at [http://localhost:4000/training-material/ ](http://localhost:4000/training-material/)
> 3. Edit one of the tutorials:
>    - For example, open `topics/introduction/tutorials/galaxy-intro-peaks2genes/tutorial.md` in a text editor of your choice.
>    - Make some changes to the *Introduction* paragraph, and save the file.
>    - Refresh the tutorial page in your browser until you can see the changes you made.
>        - this may take a little bit of time; in the terminal you can monitor when the regeneration is complete
>
{: .hands_on}

With `make serve-quick`, a local Jekyll server will run in background. It will check the changes and regenerate the website accordingly. You may need to reload the page to see the changes (and sometimes to wait 1-2 minutes).


> <tip-title>make serve vs make serve-quick</tip-title>
>
> 1. Use `make serve` instead of `make serve-quick` to get all plugins, but also configure the post, host and pass additional flags. This however can be quite slow.
>
> 2. Need to speed up the cloning step? You coud fetch only the latest commit of the main branch:
>
>    ```
>    $ git clone {{ site.github_repository }}.git --depth 1 --branch main
>    ```
>
> 3. Running on a VM or remote machine?
>
>    If you are not running this on your local machine, but e.g. on a VM, you may need to configure a webserver to serve the website.
>
>    Below is an example NGINX configuration (e.g. in `/etc/nginx/sites-enabled/default`)
>
>    ```
>    location /training-material/ {
>      root /home/ubuntu/training-material/_site/;
>    }
>    ```
>
>    Change the `root` path above to wherever you cloned the training material folder
>
{: .tip}


# Stopping the server

Once you are done, you can stop the server and clean your repository.

> <hands-on-title>Stopping the server</hands-on-title>
>
> 1. Stop the server with <kbd>CTRL</kbd>-<kbd>C</kbd>
> 2. Clean the repository: `make clean`
>
{: .hands_on}

# Conclusion

</div>

<div class="container-based" markdown="1">

Using the container-based method, you can edit and preview GTN Training Material locally on you computer.

# Prerequisites

You will need the GTN Training Material source code and a container platform. If you wish to contribute your changes, you will also need git. The exact installation steps vary a little depending on your operating system.

For this tutorial, Docker will be used as container platform, as it is easy to use and available for all major operating systems.

Other container platforms should work as well. Beware with rootless container platform installations, you have to pay special attention to the user namespace mode and thus to the UID and GID configuration in order to preserve correct file ownership information.

## For Linux users

#### Docker

I recommend to install only Docker Engine (**not** Docker Desktop) for Linux users. Just follow the [official setup documentation](https://docs.docker.com/engine/install/) for your specific distribution.

An common practise is to assign your current user to the `docker` group. **Please do not!** This implies inherent security concerns, as it basically gives your current user unrestricted root access. Learn more: [https://docs.docker.com/engine/security/#docker-daemon-attack-surface](https://docs.docker.com/engine/security/#docker-daemon-attack-surface).

#### Git

Use the package manager of your specific distribution in order to install git. All major linux distributions do have git in their official package sources. Please consult your distros documentation for help with package installation.

###### Examples

```bash
# For Debian based distros, using apt:
sudo apt install git

# For Fedora based distros, using yum:
sudo yum install git

# For Fedora based distros, using dnf:
sudo dnf install git

# For Arch based distros, using pacman
sudo pacman -S git
```

## For Windows users

On windows you have multiple virtualization options in order to run linux containers. You should use the WSL2 method for various reasons. This tutorial is only tested for the WSL2 method only.

Additionally, to work with source code located within WSL2, I recommend installing VS Code.

#### WSL2

Please follow the [official setup documentation](https://learn.microsoft.com/en-us/windows/wsl/install) for installing WSL2.

###### TL;DR

Open PowerShell by right-clicking on the windows logo in taskbar and run:

```bash
wsl --install
```

#### Docker

Please follow the [official setup documentation](https://docs.docker.com/desktop/setup/install/windows-install/) for installing Docker Desktop on Windows.

#### Git

With the installation of Docker, an Ubuntu image has been installed on your system, which already includes git. No additional steps needed.

#### VS Code

Please follow the [official setup documentation](https://code.visualstudio.com/download) for installing VS Code. This is useful for windows users as it is one of the simplest methods to work with files located in the WSL2 (linux) file system.

## For MacOS users

#### Docker

Please follow the [official setup documentation](https://docs.docker.com/desktop/setup/install/mac-install/) for installing Docker Desktop on MacOS.

#### Git

Run a terminal and type a git command, e.g. `git --version`. If git is not installed on your system yet, MacOS will automatically prompt you to install it through Xcode. Just follow the on-screen instructions.

# Preparation

## Command line utilities

All following commands are meant to be entered in a shell. For Linux and MacOS, e.g. you can use a terminal application.

Windows users **must not** use the PowerShell, but the Ubuntu app that was automatically installed alongside WSL2 and Docker. If you are running it the first time, create a username and a password. Beware that files created in the Ubuntu app are located on a separate file system. If you need to access them from windows, you may open the explorer and choose the "Linux" Bookmark in the left sidebar. However, usually you should not need to access any of these files through windows.

## Git basics

In order to be able to contribute, you will need to have a Github account and you will need to create a fork of the GTN Training Material Repository. This can be simply done via Browser on the [GTN Training Material Github page](https://github.com/galaxyproject/training-material).

You will need to create a SSH key pair or provide other authentication vectors in order to push changes to your fork. Read more about this in the [official documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/about-ssh).

If this is your first time using git, please read the [Github contribution guide](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project).
## Clone the GTN Training Material Repository

For Linux and MacOS users, open your favorite terminal application. For Windows users, open the Ubuntu app which provides you with a terminal, as well. Run:

```bash
# You may want to replace the address with your forks address
# You may want to use SSH based clone if you want to contribute
git clone --depth 1 --branch main https://github.com/galaxyproject/training-material.git
```

## Configure environment

Switch into the cloned repository:

```bash
# "training-material" is the default name. If you have changed your forks name or if you provided a directory to clone into, please edit accordingly.
cd training-material
```

For Linux and MacOS users, you may open the directory in your favorite IDE.

For Windows users, if you have installed VS Code prior, run following command to open the directory in VS Code:

```bash
# For windows users
code .
```

Create a `.env` file. In this file, you may add several environment variables:

| Key | Description | Default |
| --- | --- | --- |
| COMPOSE_PROJECT_NAME | The project name to be used for composition, should be unique | gtn-dev |
| GTN_RUBY_TAG | The Ruby Tag to be used for the container | 3-bookworm |
| GTN_UID | The effective UID to use within the container, should match your current UID | 1000 |
| GTN_GID | The effective GID to use within the container, should match your current GID | 1000 |
| GTN_PORT | The Port on which GTN will be accessible, should not be already bound | 4000 |
| GTN_RELOAD_PORT | The Port on which the reload server will listen, should not be already bound | 4001 |

If you don't know your current UID and GID you may run:

```bash
# To find out your UID
id -u

# To find out your GID
id -g
```

Your `.env` should have a form like:

```bash
# KEY=VALUE
# e.g.:
COMPOSE_PROJECT_NAME=galaxytraining
```

# Using the composition

I recommend using the command line, but you can also do lifecycle operations via docker desktop after you have run the composition the first time.

Beware, the first time the composition is starting, it may take some time until everything is setup entirely.

When the composition has started, you may access the GTN Training Material on the port you provided, per default this is 4000: [http://localhost:4000/training-material/](http://localhost:4000/training-material/).

###### Starting

If you are done configuring, you may start the composition

```bash
# To start it with attached terminal
sudo docker compose up

# To start it with detached terminal
sudo docker compose up -d
```

If you have started the composition detached, you may retrieve the logs via

```bash
# -f will follow the logs, if you want to disconnect, press CTRL+C
sudo docker compose logs -f
```

###### Stopping

If you have started the composition with attached terminal, just press `CTRL+C`.

If you have started the composition detached, you may run:

```bash
sudo docker compose stop
```

###### Removing

If you want to remove associated containers and resources, you may run:

```bash
sudo docker compose down
```

If you have volumes involved, you may run this command with the `-v` flag in order to remove the volumes as well.

</div>


topic: contributing
tutorial: learning-principles
title: Principles of learning and how they apply to training and teaching
difficulty: intermediate
time_estimation: 2h
tools_json: []
requirements_json: []
tags_json: ["elixir", "train-the-trainers"]
content_hash: 35ae7eb25f2b0152474786d9bf16265e
last_modified: 2026-02-26T21:12:14.714723
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/learning-principles/tutorial.html

content:
In this tutorial we will discuss the principles of learning

> *Learning results from what the student does and thinks and only from what the student does and thinks. The teacher can advance learning only by influencing what the student does to learn*
>
> [H.A. Simon](https://en.wikipedia.org/wiki/Herbert_A._Simon) (one of the founders of the field of [Cognitive Science](https://en.wikipedia.org/wiki/Cognitive_science) and Nobel Laureate)
{: .quote}

This quotation from Herbert A. Simon clearly indicates that we cannot talk about teaching, teaching practices or effective teaching techniques if we don't understand first how people learn.

In this lesson, we want to share with you what we have collectively learnt about how learning works. Our knowledge in the field of cognition and learning comes from diving into pedagogical and cognitive research results, reading books such as ["How Learning Works"](https://www.amazon.it/How-Learning-Works-Research-Based-Principles/dp/0470484101) ({% cite ambrose2010learning %}) or ["Small Teaching"](https://www.wiley.com/en-gb/Small+Teaching%3A+Everyday+Lessons+from+the+Science+of+Learning%2C+2nd+Edition-p-9781119755555) ({% cite smallTeaching2021 %}) and many others, studying research articles published in the field of cognitive science and educational psychology, attending instructor training courses (like this one), and - last but not least - from our own experience as both instructors/teachers, learners, and passionate observers of learning processes and teaching practices.

Since this material cannot cover the broad literature on the subject of learning we strongly invite you to commit to read at least one book on how learning works ({% cite ambrose2010learning %}, {% cite smallTeaching2021 %}), have a look at the [Software Carpentry Instructor training materials](http://swcarpentry.github.io/instructor-training/) and explore the material available through the [ELIXIR Train-the-Trainer course](https://elixir-europe.org/platforms/training/train-the-trainer).

> <comment-title>Sources</comment-title>
>
> This tutorial is significantly based on {% cite via2020course %}, session one
> of ELIXIR-GOBLET Train the Trainer curriculum.
{: .comment}

<!--## Instruction design in 5 Phases

So how can we now use the knowledge we have gathered during this time in the course? Here we will go through the Instructional design in 5 phases and explain why learning outcomes (to articulate and specify these) is so important for the curricula and the course design process.

![A flow chart of the five phases of the instructional design. The 1st phase is to select the Learning Outcomes, followed by the selection of the corresponding Learning Experiences. The third phase is to select the content that is relevant to the LO. The fourth phase is the development of the necessary assessments for the learners, with the final phase being the evaluation of the effectiveness of the LE and the LO, at which point the flow connects back to the first phase for the refinement of the process.](images/PhasesInstructionalDesign.png "5 phases of instructional design described by Nicholls et al in Developing teaching and learning in higher education. London, UK: Routledge. pp 51-75")

1. **Writing LOs**: Important to build the training
2. **Selection learning experiences**: Focus on LOs
3. **Selection of content relevant to LOs**: we usually (traditionally) starting by the contentm, but here only 3rd step
4. **Develop assessment**

    Example: formative assessment as the previous challenge

5. **Evaluation** of your training

    Example
    * Final exams at university
    * Final feedback
-->

# How do people learn?

So let's start with "How do people learn and how does learning work"

## General definition of learning

(Relatively) permanent changes in behaviour, skills, knowledge, or attitudes resulting from identifiable psychological or social experiences.

* A key feature of learning: permanence
* The change has to last
* Changes do not count as learning if they are temporary
* Example: learning a phone number

Why is it important to learn about how people learn?

* It is common to confuse our effort (i.e. our teaching) with what learners get from our efforts (i.e. their learning)
    * We usually assume learners learnt
    * Nevertheless, the distinction between learning and teaching is especially important for teachers to remember and it is essential that teachers become aware of the fact that the way they think about learning affects the way they teach and that the very circumstances of teaching can influence teachers’ perceptions of learning, and therefore also influence how they teach.
    * For example, imagine that a cognitive scientist convinced you that people don’t retain anything of what you tell them after 20 minutes of passively listening to you. Would you keep lecturing for more than 20 minutes?
    * And how would you teach if, in contrast, you assumed that the mind is a vessel to be filled?
    * Moreover, most teachers tend to think that what is taught is equivalent to what is learned, namely that what they teach is the same as what students understand or retain of what they teach  (even though most of them know that assuming this is a mistake and that teaching and learning can be quite different).
* Learning how learning works helps clarify this distinction
* The way we think about learning affects the way we teach
* Understanding  how people learn makes you a more aware and therefore better teacher
* You  can use such knowledge to improve your day-to-day teaching

According the **Meaningful Learning Theory**, developed by the psychologist David Ausubel, meaningful learning generally arises from a process of learning to learn. He considers that if meaningful learning is achieved, it transcends the rote repetition of unconnected content and allows for the construction of meaning, making sense of what has been learned, and understanding its scope of application and relevance in academic and everyday situations.


## Learning metaphors, theories and models

Learning - similarly to thinking - is a complex process that requires multiple approaches spanning many different disciplines, ideas, theories and models to be understood.

Many learning theories, models, concept and ideas from from the **Science of Learning**, **Educational Psychology**, and the **Cognitive Sciences** are relevant to classrooms, in that they describe at least some of what usually happens there and offer guidance for assisting learning.

This session will provide you with some understanding of how learning works but it is far from being exhaustive. In particular, we will present and discuss learning models, concepts and perspectives that are important for you to know as they suggest things that you might do in your classroom to make learners’ learning more productive and your teaching more effective and mindful.

A metaphor is a word from one thematic domain that has been embedded in another one, thus entering a network of new linguistic relations. This is, indeed, what metaphors are all about: they are transplants from one discourse to another. They are a source of new ways of seeing things. "Metaphors of learning" are a source of new ways of seeing learning.

We can use metaphors to shape and reveal our way of thinking about learning and, therefore, shape our actions as teachers.

---

> A mind is a fire to be kindled, not a vessel to be filled
>
{: .quote author="Plutard"}

---

How the mind-as-a-vessel-to-be-filled metaphor may affect your way of teaching? You are likely to spend your time in the class at the blackboard, trying to 'transmit' to the students your own knowledge.

<!--{% include _includes/quiz.html id="learning_metaphors.yaml" %}-->


> <hands-on-title>Learning metaphors - {% icon time %} 5 min</hands-on-title>
>
> 1. Think about the following metaphors "A mind is a fire to be kindled". How this metaphors may affect your way of teaching?
> 2. Think about the following metaphors "Teaching is growing a garden". How this metaphors may affect your way of teaching?
>
{: .hands_on}


## Metaphors define the role of the teacher and the learners

Metaphor | Teacher | Learner
--- | --- | ---
Learning is knowledge transfer | Defines, develops and prepares contents | Receivers and memorizes
Learning metaphors | Prepares the environment, guides the exploration | Explores, works on information, attributes meanings
Learning is participation in social practices | Community that shares activities and social practices | As teacher

# Learning theories

Learning theories describes how students receive, process, and retain knowledge during learning. However, there is no single one theory (as in physics for example). There is a massive catalogue of learning theories out there and there is no universal theory of learning.

![The figure shows a dendrogram of the main learning theories that are currently available. The information is organized across four main levels; the Scientific Discipline (such as Organization, Education, Philosophy, etc), the Learning Theorists (such as Montessori and Bloom), the Learning Paradigms (such as Behaviourism, Constructivism), and the key concepts.](images/LearningTheories.png "This is a diagram representing the landscape of many known learning theories available.")

Philosphers and researchers from multiple disciplnes have proposed a large number of learning theories: a flavour of the diversity of approaches and philosophies can be gained, for example, from the [HoTEL](http://cmapspublic3.ihmc.us/rid=1LNV3H2J9-HWSVMQ-13LH/Learning%20Theory.cmap) (Holistic approach to Technology-Enhanced Learning) website. But which should we rely on? Which should we learn, teach and apply?

There is no universal theory of learning, but there are interesting elements for classroom teaching in many different theories There are elements of truth in different theories, and **evidence-based** research results that support learning principles recognized and applied today. In this sense there are interesting elements for a teacher in many different theories.

In the following sections we will explore the most important learning theories.

> <comment-title>Principal learning theories</comment-title>
>
> We will briefly describe the key ideas from each of these perspectives and explain how you can use them in your classroom to support learners' learning.
>
> #### Behaviourist Learning Theory
>
> **Behaviorism** is a psychological movement that posits behavior should be studied as a natural science akin to fields like chemistry or physics. It conceives learning as changes in overt behavior. In this context, learning is acquiring new behaviours by conditioning, reflex response to stimuli, reward/punishment.
>
> An important concept from this theory is **operant conditioning**, a variety of behaviourism that focuses on how the consequences of a behaviour affect the behaviour over time.
>
> This theory is based on the idea that certain consequences (reinforcement or inhibition of a behaviour) tend to make a behaviour happen more or less frequently. According to the behaviorist perspective on learning, the instructor assumes a dominant role within the classroom, exerting complete control over the learning process. Evaluation of learning is carried out by the teacher, who determines what is considered correct or incorrect.
>  
> #### Cognitivist Learning Theory
>
>
> Diverse authors agree that, in order to implement adequate online teaching practices, it is necessary to take into account the characteristics of mental processes ({% cite Wu1998 %}, {% cite Chandler2004 %}).
>
> **Cognitivism** emerged in the 1950s as a response to Behaviorism, which was deemed inadequate in explaining cognition. **Cognitive Learning Theory** aims to describe the links between cognitive structures -defined as the mental representations of objects or ideas- and the learning process ({% cite Qin2009 %}, {% cite vanderVeer1988 %}), with a special focus on the processing of information. Two important theories derive from this one: **Information Processing Theory** and **Cognitive load theory**:
>
> The **Information Processing Theory** describes learning as the result of sequential processing of information, which involve three types of memory: immediate memory, working memory and long term memory ({% cite Gentner2014 %}). According to this theory, adequate learning resources require to assume two key elements: firstly, the fact that working memory is limited; and secondly, that the interaction between working and long term memory plays an important role in the construction and transferability of knowledge ({% cite Chandler2004 %}, {% cite Herring1993 %}, {% cite Yilmaz2011 %}).
>
> On the other hand, **Cognitive Load Theory** seeks specifically to address the efficiency with which information is processed. It states that the verbal and visual information is processed by independent cognitive structures, both of them with a limited capacity ({% cite Fori2012 %}, {% cite Via2013 %}, {% cite Kats2010 %}, {% cite Wright2010 %}). An interesting concept derived from this theory is the cognitive overload, which refers to those situations in which the information flow exceeds the learner’s processing capacity, resulting in an inhibition of the learning process ({% cite Aidi2009 %}).
>
>
> #### Constructivist Learning Theory
>
> **Constructivism** holds that learning is essentially active; a person who learns something new incorporates it into their previous experiences and their own mental structures. Each new piece of information is assimilated and deposited into a network of previously existing knowledge and experiences  ({% cite Abbot1999 %}). All these ideas have been taken from different nuances, and two authors stand out: Jean Piaget with **psychological constructivism** and Lev Vigotsky with **social constructivism**.
>
>From the perspective of **psychological constructivism**, learning is fundamentally a personal matter. It maintains the idea that the individual, both through cognitive and social aspects of knowledge as well as affective ones, is not a mere product of the environment or a simple result of his or her internal dispositions, but a construction of his or her own that is produced day by day as a result of the interaction between these two factors. The driving force of this activity would be cognitive conflict; in any constructivist activity there must be a circumstance that shakes the previous structures of knowledge and forces a rearrangement of the old knowledge in order to assimilate the new.
>
>According to the **social approach of Lev Vigotsky**, learning is that process in which the individual assimilates certain historical-cultural experience at the same time as he/she appropriates it, which requires an active subject who gives meaning to this experience, transforming it into his/her subjectivity ({% cite RodrguezRuiz2005 %}). 
>
> The aim of Vigotsky's theory is to discover and stimulate the **zone of potential development** or zone of proximal development in each student, which is defined as the distance between the actual level of development, determined by the ability to independently solve a problem, and the level of development potential, determined through solving a problem in collaboration with a more able peer ({% cite Moreira1997 %}).
>
> For Vigotsky, cognitive development cannot be understood without reference to the social, historical and cultural context in which it occurs. According to this author, higher mental processes (thought, language,voluntary behaviour) have their origin in social processes; cognitive development would be the conversion of social relations into mental functions ({% cite Moreira1997 %}).
>
> #### Critical Learning Theory
>
> The **Critical Learning Theory**, also known as Transformative Learning Theory, aims to define the mechanisms involved in the development of critical thinking during the learning process. It considers that thought must be capable of questioning even the logic of reasoning itself, that is, it must be capable of problematising what has hitherto been treated as self-evident, of turning into an object of reflection what had previously been simply a tool. According to {% cite Marcuse1964 %}, critical thinking aims to define the irrational character of established rationality.
>
> According this theory, learning occurs when the learning process induces a shift in the student's frame of reference. Although this goal can be achieved in multiple ways, a successful strategy is to stimulate reflection on uncritically accepted assumptions. It is the educator responsibility to cultivate students’ skills related to autonomous thinking by designing activities for that purpose.
>
> Critical Learning Theory emphasises two elements: the importance of the subjective role of each learner in finding meaning for him/herself, and how social and economic conditions constrain and distort the social construction of meaning ({% cite Giroux2018 %}).
>
{: .comment}


> <hands-on-title>How do you go about learning something new? / How do you approach learning new things? - {% icon time %} 5 min</hands-on-title>
>
> - Order the following approaches (from the most to the less effective one) for you when you want to learn something new
>	* Read about it
>   * Attend a training session !
>   * Have a go ?
>   * Do, reflect, process, further understand?
> - Write if you recognise elements of one of more learning perspectives (constructivism / behaviourism) in your way of learning.
>
{: .hands_on}


# Principal learning theories: practical examples

In this section we will introduce some examples of practical applications from the different learning theories:

Learning theory | Example
--- | ---
Behaviorist | Dreyfuss model for skill acquisition.
Cognitivist | Human Memory and Learning from cognitive sciences
Constructivist | Bloom's Taxonomy
Critical | The hidden curriculum

## Behaviorist Learning theory: Dreyfuss model for skill acquisition

The Dreyfus model of skill acquisition presents a framework illustrating the process by which learners acquire skills through a combination of formal instruction and dedicated practice. It was presented by Hubert and Stuart Dreyfus in 1980 in a report on their research at the University of California, Berkeley, Operations Research Center for the United States Air Force Office of Scientific Research. Hubert Lederer Dreyfus (1929-2017) was a prominent American philosopher and a professor of philosophy. His areas of expertise encompassed phenomenology, existentialism, and the philosophy of psychology, literature, and artificial intelligence, exploring their profound philosophical implications. Stuart E. Dreyfus holds the position of professor emeritus in the Industrial Engineering and Operations Research Department at the University of California.

The Dreyfus model for skills acquisition by brothers Stuart and Huber Dreyfus uses a 5 stage from novice to expert.

![A representation of the Dreyfuss model, as a gradient bar. Starting from the left, is the Novice which relies primarily on rules to organize their skills. The Novice, with experience, transitions to Advanced Beginner and then competent, increasing the use of their intuition along with the rules. The next two stages are Proficient and Expert (at the right end of the bar), which know relies almost exclusively on their intuition.](./images/DreyfussModel.png "Dreyfuss model")

The stages follow a progression from rigid adherence (novice) to rules to an intuitive mode of reasoning based on tacit knowledge "silent knowledge" (expert).

The fundamental idea is that when teaching a concept, you have to tailor the style of teaching to where the learner is in their understanding and that progression follows a common pattern. Early stages of learning focus on concrete steps to imitate, the focus then shifts to understanding principles and finally into self-directed innovation.

The skill level is tied to a mental function transitioning from recollection (non-situational or situational), recognition (decomposed or holistic), decision (analytical or intuitive), and awareness (monitoring or absorbed)

### Example of Dreyfus model: the Carpentries

The Carpentries (a global organization building global capacity in essential data and computational skills for research) uses the Dreyfus model as its global [approach](https://carpentries.github.io/instructor-training/02-practice-learning/index.html#the-acquisition-of-skill) for skill acquisition:

![A simplified version of the Dreyfus model, as built by the Carpentries. There are three levels; the Novice, the Competent Practicioner and the Expert.](./images/DreyfussModelCarpentries.jpg "Carpentries build on the Dreyfus model. Source: The Carpentries Instructor training")

- **Novice**: someone who does not know what they do not know, i.e., they do not yet know what the key ideas in the domain are or how they relate. Novices may have difficulty formulating questions, or may ask questions that seem irrelevant or off-topic as they rely on prior knowledge, without knowing what is or is not related yet.

    Example: A novice learner in a Carpentries workshop might never have heard of the bash shell, and therefore may have no understanding of how it relates to their file system or other programs on their computer.

- **Competent practitioner**: someone who has enough understanding for everyday purposes. They will not know all the details of how something works and their understanding may not be entirely accurate, but it is sufficient for completing normal tasks with normal effort under normal circumstances.

    Example: A competent practitioner in a Carpentries workshop might have used the shell before and understand how to move around directories and use individual programs, but they might not understand how they can fit these programs together to build scripts and automate large tasks.

- **Expert**: someone who can easily handle situations that are out of the ordinary.

    Example: An expert in a Carpentries workshop may have experience writing and running shell scripts and, when presented with a problem, immediately sees how these skills can be used to solve the problem.

> <details-title>The Carpentries</details-title>
>
> [The Carpentries](https://carpentries.org/) was started by Greg Wilson and Brent Gorda and they coined the term "The Carpentries" for enabling researchers to do "computational thinking" and use tools of computation effectively.
>
> The Style of the Carpentries are that teachers are where the learners are e.g. by doing live-coding together.
{: .details}


## Cognitivist Learning Theory: Memory and learning

In a simple model from cognitive science, learning happens when information moves from **working memory** to **long-term memory**. This process can be impacted by **environment**

![A figure showning a simple representation of how memory works. The main interaction is between the two central boxes, representing the working memory and the long-term memory. The working memory maintain the active consiousness, when paying attention to the envnironment, and is extremely fast but rather limited in capacity (experiments have shown that it can keep track of 7+/-2 items). The long-term memory is much larger in capacity, but it's significantly slower to access - using a process called "remembering". At the same time, the process of depositing information from the working memory into the long-term memory, is the learning process.)](./images/HowMemoryWorks.png "Simple model (brain is much more complex)")

<!--1. Environment
2. Working memory
3. Long-term memory
4. Working memory: fast
	* Ex: phone number
5. Long term: slow
	* How much effort to remember a full poem-->

 
> <hands-on-title>How short is your short term memory? - {% icon time %} 10 min</hands-on-title>
>
> Based on {% cite Willingham2009 %}.
>
> 1. Read the following letter combination:
>  
>    > <solution-title>Hide</solution-title>
>    >
>    >    ```APH D BDN A CKG B DCI A```
>    >
>    {: .solution}
>
>   1. Hide them after 10 seconds
>   2. Take a piece of paper and write what you remember
>   3. Unhide them after 2 minutes
>   4. Check how many consecutive letters you can remember
>
> 2. Read the following letter combination
>
>    > <solution-title>Hide</solution-title>
>    >
>    >    ```A   PHD   B   DNA   C   KGB   D   CIA```
>    >
>    {: .solution}
>
>   1. Hide them after 10 seconds
>   2. Take a piece of paper and write what you remember
>   3. Unhide them after 2 minutes
>   4. Check how many consecutive letters you can remember
>
{: .hands_on}

<!--{% include _includes/quiz.html id="short_term_memory.yaml" %}-->

The amount of space in working memory does not depend on the number of letters. It depends on the **number of meaningful objects**. In the 2nd round, the meaning of the words (long-term memory) or connecting words with a story helps to remember. It make room in working memory.

To facilitate learning, we as instructors have to make room in working memory.

### What can we do to make room in working memory?

<!--{% include _includes/quiz.html id="short_term_memory.yaml" %}-->

> <hands-on-title>How can you avoid to overload learners working memory?  - *{% icon details %} 5 min - Groups of 2/3 persons*</hands-on-title>
>
> - Discuss one thing a teacher could do to avoid overloading their students' working memory
> - Write your proposal in the shared notes
{: .hands_on} 

What can we do to make room in working memory?
* Chunking

    Example: Short theoretical sessions alternated with practical sessions to allow the processing of the material

* Avoid extraneous **cognitive load**

    Example: Avoid too crowded slides

### Cognitive load

There are 3 cognitive loads recognized by cognitive science:

* **Intrinsic cognitive load**: effort associated with a specific topic.
* **Germane cognitive load**: the (desirable) mental effort required to create linkages between new information and old.
* **Extraneous cognitive load:** everything else that distracts or gets in the way. It refers to the way information or tasks are presented to a learner.

### Attention split effect

Split-attention occurs when learners are required to **split their attention** between **at least two sources of information** that have been separated either spatially or temporally

> <hands-on-title>Extraneous cognitive load - {% icon time %} 10 min - Groups of 2/3 persons</hands-on-title>
>
> * Pick one task you teach in your lessons/courses
> * Think and discuss within the room about the following questions
>     * What would be the extraneous load in performing this tak?
>     * How can you avoid it?
{: .hands_on}


### Six strategies for effective learning from cognitive psychology

**Six strategies for effective learning** (from "Understanding how we learn" {% cite weinstein2018understanding %}) are based on evidence from cognitive research.

![Six circles representing the six Strategies for Effective Learning.](./images/StrategiesEffectiveLearning.png "Six Strategies for Effective Learning {% cite weinstein2018understanding %} Six Strategies for Effective Learning by [Yana Weinstein, Megan Smith, & Oliver Caviglioli](http://www.learningscientists.org/downloadable-materials/) / [Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](http://creativecommons.org/licenses/by-nc-nd/4.0/), based on a work at[http://www.learningscientists.org](http://www.learningscientists.org/), [teachinghow2s.com/cogsci](http://teachinghow2s.com/cogsci).")

- **Spaced practice**: Creating a study schedule that spreads study activities out over time

    Spaced practice is the exact opposite of cramming. When you cram, you study for a long, intense period of time close to an exam. When you space your learning, you take that same amount of study time, and spread it out across a much longer period of time. Doing it this way, that same amount of study time will produce more _long-lasting learning._

- **Interleaving**: Switching between topics while studying

    You shouldn't study one idea, topic, or type of problem for too long. Instead, you should change it up often. Especially good for problem-solving topics like bioinformatics. Then you should jump from one topic to another and go back to the previous one, with the aim of developing the connections among topics.

- **Elaboration**: Asking and explaining why and how things work

    It can be used to mean a lot of different things. It involves explaining and describing ideas with many details. Elaboration also involves making connections among ideas you are trying to learn and connecting the material to your own experiences, memories, and day-to-day life.

- **Concrete examples**: When studying / explaining abstract concepts, illustrating them with specific examples

    The idea is to take an abstract concept and use real world examples to increase understanding and 2 examples are better than 1.

- **Dual coding**: Combining worlds with visuals

    Dual coding is combining words and visuals such as pictures, diagrams, graphic organizers, and so on. The idea is to provide two different representations of the information, both visual and verbal. Adding visuals to a verbal description can make the presented ideas more concrete, and provides two ways of understanding the presented ideas. The visuals should be meaningful.

- **Retrieval practice**: Bringing learned information to mind from long term memory

    Concept maps are examples of a retrieval practice when you bring out knowledge/memories from long-term memory

> <hands-on-title>How do you understand the 6 strategies for effective learning? - {% icon time %} 15 min - Groups of 2 persons</hands-on-title>
>
> - Pick up one strategy per group
> - Discuss about the strategy
>   - Did you understand it the same way?
>   - Do you have any questions?
> - Find out an example on how you would implement it as a teacher/instructor (what would you do as a teacher in order to facilitate learning based on the strategy at hand)
{: .hands_on}

We highly recommend to look at the materials in a book and a website and blog as they are intended to teach about principles of learning and to provide teachers and students with flexible guiding principles to guide learning and studying. In ["Training techniques to enhance learner participation and engagement" tutorial]({% link topics/teaching/tutorials/learner_participation_engagement/tutorial.md %}), a number of teaching practices are described. You should try to reconnect them to these learning strategies, i.e. try to understand how the proposed teaching practices facilitate learning based on these six strategies.

### Seven learning principles from cognitive sciences

From ["How Learning works"](https://www.academia.edu/6743627/How_Learning_Works_Seven_Research_Based_Principles_for_Smart_Teaching_Susan_A_Ambrose_Michael_W_Bridges_Michele_DiPietro_Marsha_C_Lovett_Marie_K_Norman_with_a_Foreword_by_Richard_E_Mayer) {% cite ambrose2010learning %}

* **Principle P1**: Students' **prior knowledge** can help or hinder learning.

    Hindrances include prior knowledge which may be accurate but insufficient to support learning new material. Inappropriate for the specific class, thus distorting comprehension of new material; or, completely inaccurate.

* **Principle P2**: How students **organise knowledge** influences how they learn and apply what they know.

    Generally, people organize knowledge "as a function of their experience, the nature of their knowledge, and the role that that knowledge plays in their lives"

* **Principle P3**: Students **motivation** determines, directs and sustains what they do learn.

    {% cite ambrose2010learning %} define motivation as "the personal investment that an individual has in reaching a desired state or outcome" and identify three core concepts for understanding motivation
    - the subjective value of a goal
    - the expectancies for successfully reaching the goal
    - ??

* **Principle P4**: To develop mastery, students must acquire **component skills**, practice **integrating** them, and  know when to **apply** what they have learned

    Mastery refers to "the attainment of a high degree of competence with a particular area". It could be either beneficial or detrimental.

* **Principle P5**: **Goal-directed** practice coupled with **targeted feedback** enhances the quality of students' learning

    Learning and performance are advanced when students employ a methodology that
    - focuses on a specific performance goal
    - pinpoints a suitable level of challenge relative to students’ current performance,
    - is in an amount and frequency sufficient to meet the performance criteria.

* **Principle P6**: Students' current level of development interacts with the **social, emotional, and intellectual climate** of the course to impact learning

    {% cite ambrose2010learning %} maintain that "student-centered teaching requires us to teach students, not content".

* **Principle P7**: To become self-directed learners, students must learn to **monitor and adjust** their approaches to learning.

    This principle relates to metacognition, or "the process of reflecting on and directing one's own thinking". You can find more information on the connection of metacognition to student learning and performance here {% cite Stanton2021 %}.

    Students develop metacognitive skills by learning to
    - assess the task at hand,
    - evaluate their own strengths and weaknesses,
    - plan appropriate approaches to complete the task,
    - apply strategies to monitor their performance, while making adjustments to the selected approach.

<!-- 
> <hands-on-title>How do you understand the 7 principles of learning?  - {% icon time %} 15 min - Groups of 2/3 person</hands-on-title>
>
> * Read and reflect together the principles
> * Pick one principle that is not (fully) clear to you
> * Ask one question about this principle
> * Write here any insights/thoughts/comments to share from your breakout room
{: .hands_on}-->

> <details-title>Resources to learn more about the Cognitive Learning Theory</details-title>
>
> There is a great deal of literature on how learning works.
>
> The reason why we mention these three books is that they show how to integrate effective, research based strategies for learning into classroom practice.
>
> Educational practice usually does not rely on research findings. People tend to rely on their intuitions about what’s best for learning. But relying on intuition may be a bad idea for teachers and learners alike.
>
> - ["How Learning Works"](https://www.amazon.it/How-Learning-Works-Research-Based-Principles/dp/0470484101) ({% cite ambrose2010learning %})
>
>    This book presents a set of evidence-based principles for how to help people learn that is grounded in cognitive theory (that is, the science of instruction). It tries to help teachers understand how research in the science of learning can improve their reaching.
>
>    The book is organised around seven learning principles. Each principle is based on research evidence from the science of learning and the science of instruction.
>
> - ["Small Teaching"](https://www.wiley.com/en-gb/Small+Teaching%3A+Everyday+Lessons+from+the+Science+of+Learning%2C+2nd+Edition-p-9781119755555) ({% cite smallTeaching2021 %})
>
>    Cognitive psychologists, neuroscientists, and biologists all have produced a revealing body of research over the past several decades on how human beings learn, but often translating these findings into the classroom is overwhelming for busy instructors.
>
>    "Small Teaching" bridges the gap between research and practice  by providing a fully developed strategy for making deliberate, structured, and incremental steps towards tuning into how students are hardwired to learn.
>
>    The book introduces small activities that require minimal preparation and grading. The models presented are specifically designed to be used as both one-time experiences to innovate a course session or unit plan as well as a menu of options that can be combined into an entirely new teaching approach.
>
> - "Understanding how we learn" {% cite weinstein2018understanding %}
>
>    By the "learning scientists" Weinstein and Sumeracki (and illustrated by Oliver Caviglioli), the books examines the cognitive psychology's application to education
>
>   It is designed to convey the concepts of research to the reality of an instructor's classroom. The book proposes 6 strategies for effective learning (based on evidence from cognitive research). It is structured in four parts:
>   - Evidence-based education and the science of learning
>   - Basics of human cognitive processes
>   - Strategies for effective learning
>   - Tips for students, teachers, and parents
>
> - [The Carpentries Instructor Training materials](https://carpentries.github.io/instructor-training/)
>
>   The Carpentries is a community of volunteers teaching computational and data management skills worldwide. The community also developed training materials to teach pedagogical skills. Much of the GTN TtT was inspired by the pedagogical materials from the Carpentries and the ELIXIR TtT.
>
{: .details}


<!-- 
> <hands-on-title>What is learning in your opinion/experience? How does learning occur? - {% icon time %} 5 min</hands-on-title>
>
> _Break-out_ room (2 ppl per room)
>
> We follow the first challenge with another one. Now IN PAIRS in breakout rooms (Zoom). Here the reflection is on LEARNING and what learning is in your experience. Write your thoughts, one or more sentences, in the shared Google Doc.
> 	* Discuss your reflections on learning and what learning is in your experience
>	* Write here any insights/thoughts/comments to share from your breakout room
{: .hands_on} -->


## Constructivist Learning Theory: The Bloom's Taxonomy

And one of the most used "models" or concepts is the one based on the Bloom’s six categories of cognitive skills. This tool is important to develop training, but not only.

**Bloom's taxonomy** is a set of six categories of cognitive skills, that are accompanied by actionable verb.

![The pyramid of Blooms Taxonomy of verbs. Starting from the bottom layer, the corresponding aspects are: remembering, understanding, applying, analyzing, evaluating and creating.](./images/bloomsTaxonomyPyramid.png "Source: https://tips.uark.edu/blooms-taxonomy-verb-chart/")

From bottom to top:
* **Remembering (original Knowledge):** involves recognizing or remembering facts, terms, basic concepts, or answers without necessarily understanding what they mean.
* **Understand (original comprehension):** involves demonstrating an understanding of facts and ideas by organizing, summarizing, translating, generalizing, giving descriptions, and stating the main ideas.
* **Apply/Application:** involves using acquired knowledge—solving problems in new situations by applying acquired knowledge, facts, techniques and rules. Learners should be able to use prior knowledge to solve problems, identify connections and relationships and how they apply in new situations.
* **Analyze:** involves examining and breaking information into component parts, determining how the parts relate to one another, identifying motives or causes, making inferences, and finding evidence to support generalizations.
* **Evaluate:** involves presenting and defending opinions by making judgments about information, the validity of ideas, or quality of work based on a set of criteria
* **Create (original synthesize):**involves building a structure or pattern from diverse elements; it also refers to the act of putting parts together to form a whole.

It is important to highlight that, under this framework, **critical thinking is considered a combination of analyzing, evaluating and creating**, i.e. the top three layers of the taxonomy. Blooms taxonomy levels are very useful to align instruction with learners' level of complexity of thinking as well as experience.

What are the uses of this tool?
- For learners
    - Help to structure their knowledge
- For instructors
    - Knowing at which level of cognitive complexity you want to teach
    - Write Learning Outcomes

        ![6 levels of Bloom's taxonomy (and their verbs) from bottom (lower-order thinking skills) to top (higher-order 'critical' thinking skills): Remember (recall or reiterate information) with memorize, recognise, identify, describe, reproduce, list, define, label, name, state, outline, order, arrange; Understand (demonstrate understanding of facts) with distinguish, classify, select, review, discuss, indicate, explain, estimate, locate, summarise; Apply (apply knowledge to real situations) with operate, manipulate, experiment, choose, modify, prepare, produce; Analyze (resolve ideas into simple parts, identify patterns) with calculate, examine, model, test, break down, infer, predict, solve; Synthesise (pull ideas into a coherent whole, create new ideas) with combine, formulate, illustrate, imagine, design, invent, compose; Evaluate (make & defend judgements, assess theories & outcomes) with critique, compare ideas, solve, recommend, rate](./images/bloomsTaxonomy.png "ACTION verbs express levels of cognitive complexity")


    - Design instruction and learning experiences i.e. how you are to teach/train, which infrastructure and how to present the materials etc
    - Assess learning (taking place during curricula or course and after)
    - Align instruction with learners' level of complexity of thinking as well as experience.


## Critical Learning Theory: The Hidden Curriculum Theory

The concept of hidden curriculum refers to the learning that is incorporated by students though these aspects do not appear explicitly in the official curriculum, relying on the organisational aspects of classroom life that are generally not perceived by either students or teachers ({% cite Giroux2018 %}). 

According the Hidden Curriculum Theory, what students learn from the formally sanctioned content of the curriculum is much less important than what they learn from the ideological assumptions embodied in the communicative systems. Through the hidden curriculum, students would internalise values that emphasise respect for authority, punctuality, cleanliness, docility and conformity ({% cite Giroux2018 %}). It considers that elements such as exposure to a core of common subjects would lead to a dissolution of individuality in the mass, in the collective, to a standardisation and homogenisation of psychologies ({% cite Olivo2005 %}).

According the Hidden Curriculum Theory, if there is really an interest in **fostering critical thinking**, the **discursive mode that should dominate in the classroom should be the interrogative one**. It considers that questioning is a way of thinking, and that is why thinking is not only deriving conclusions, but often consists of the opposite: starting from the accepted conclusions of a community and turning them upside down ({% cite Sztajnszrajber2013 %}).

According this theory, technological tools, due to their design, do not act just as neutral means for transmitting information, but they also transmit values and habits of thought ({% cite Anders1980 %}). As {% cite Giroux2018 %} notes, no matter how progressive an approach to critical thinking may be, it will squander its own potential if it operates on the basis of a web of social relations in the classroom that are authoritarianly hierarchical and promote passivity, docility and silence.

# Mental models

**Mental model** is a collection of concepts and facts, along with the relationships between those concepts, that a person has about a topic or field. It is how people organize their knowledge

You can think of mental models as a graph with nodes (facts, concepts) and edges (connections), as seen in the figure of the mental model of animals below:

![A dendrogram showing the various descriptors of an animal, and the relationships between them. For example, the box "animal" is connected to the entities "eats", "breathes" and "can move", as well as sub-entities, such as "bird" and "fish". Each one of these, in turn, is connected to even more specific sub-entities (such as "penguin" for "bird" and "salmon" for "fish").](./images/animalMentalModel.png "mental model of animals")

The mental model of an expert in any given subject will be far larger and more complex than that of a novice, including both more concepts and more detailed and numerous relationships. Linking with Dreyfus model:

- A novice has a minimal mental model of surface features of the domain. Inaccuracies based on limited prior knowledge may interfere with adding new information. Predictions are likely to borrow heavily from mental models of other domains which seem superficially similar.
- A competent practitioner has a mental model that is useful for everyday purposes. Most new information they are likely to encounter will fit well with their existing model. Even though many potential elements of their mental model may still be missing or wrong, predictions about their area of work are usually accurate.
- An expert has a densely populated and connected mental model that is especially good for problem solving. They quickly connect concepts that others may not see as being related. They may have difficulty explaining how they are thinking in ways that do not rely on other features unique to their own mental model.

## Misconceptions

* **Simple factual errors**: These are the easiest to correct.
* **Broken models**: We can address these by having learners reason through examples to see contradictions.
* **Fundamental beliefs**: These beliefs are deeply connected to the learner’s social identity and are the hardest to change.

# Teaching or training?

<!--{% include _includes/quiz.html id="teaching_or_training.yaml" %}-->

> <hands-on-title>Teaching or training?  - {% icon time %} 15 min</hands-on-title>
>
> * Based on your experience, what are the differences between teaching and training?
> * Write your reflexions in the shared notes
{: .hands_on}

![A cartoon showing one of the differences between teaching and training. One the left side, the teacher is showing the information on a blackboards - in training, the instructor is showing, in practice, how this can be applied.](./images/teachingVStraining.png "A cartoon showing one of the differences between teaching and training. Source: [Open Science Training Handbook](https://www.fosteropenscience.eu/content/open-science-training-handbook)")

Teaching is more theoretical and abstract, while training (when done well) is more hands-on and practical.

Teaching seeks to impart knowledge and provide information, while training is intended to develop abilities.

- **Teach**: to provide knowledge, instruction or information
- **Train**: to develop abilities through practice with instruction or supervision

For example, it is possible to teach someone about buoyancy, fluid dynamics, water displacement and coastline safety, but that knowledge will not make them a good swimmer. Specific, practical and applied training is necessary to use abstract knowledge to learn or master a skill. In many cases, teaching and training are complementary. Vocational training programs often combine teaching and training quite nicely.

# Conclusion

After going through the theory of learning with the 6 strategies and the Ambrose 7 principles as well as the differences between teaching and training, the next step is to understand how these elements can be applied in designing an effective curriculum developmental trajectory.


topic: contributing
tutorial: fair-gtn
title: FAIR Galaxy Training Material
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "gtn", "training"]
content_hash: 3985de93d085c4db31d7e31696a9b773
last_modified: 2026-02-26T21:12:15.969105
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/fair-gtn/tutorial.html

content:
Encouraging computational reproducibility in research, we will present a variety of data stewardship recommendations that we have found useful in the process of training development. As part of that process, we are exploring the application of the FAIR (Findable, Accessible, Interoperable, Reusable) guidelines to the Galaxy Training Network (GTN) materials, in order to improve their secondary use and adaptation.

This tutorial outlines how to set and use existing resources to make Galaxy training development and content creation FAIR.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Here, we refer to a set of good practices as described in *"Ten simple rules for making training materials FAIR"* {% cite Garcia2020 %}.

![Ten simple rules for making training materials FAIR.]({% link topics/fair/images/fair_gtn.png %} "Ten simple rules for making training materials FAIR. The primary guideline is to share; the Findability rules are description, identity, and registration; the latter two, along with access rules, correspond to accessibility; and the first rule is to share; With the exception of the format rule, which stands alone for interoperability, the remaining four criteria all relate to various facets of reusability")

Image credit: Luc Wiegers and Celia van Gelder {% cite wiegers_luc_2019_3593258 %}

## Plan to share your training materials

The Galaxy Training Network (GTN) provides researchers with online training materials, connects them with local trainers, and helps promoting open data analysis practices worldwide. It provides a record of training content development and ensure materials curation via  cross-domain repositories (i.g., [GitHub](https://github.com/galaxyproject/training-material), [Zenodo](https://zenodo.org/)). Keeping your materials in the right places from the beginning will make it possible for you to more effectively and widely distribute your work without any duplication in contribution. Instruction where to start creating a new tutorial can be found at [GTN contributing tutorial]({% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %}).

## Improve findability of your training materials by properly describing them

Describing your training materials with structured metadata is fundamental to making them FAIR resource. Creating metadata boost findability and preserve information and can be updated after publication. GTN  tutorial require several mandatory metadata information such as learning objectives following Bloom’s taxonomy {% cite chevron2014metacognitive %}, prerequisites, time estimate, or questions addressed by the tutorial. Content developers needs to know where to find all of the available metadata to reference it later. [Schema.org](https://schema.org/) is a collaborative project with a mission to create the addition of structured metadata to web pages. It describes data using shared vocabulary. It is in this spirit that the [BioSchemas](https://bioschemas.org/) community initiative was created to extend the Schema.org standard to life-science resources with training content specification. GTN offers schemas command line [tutorial]({% link topics/contributing/tutorials/schemas/tutorial.md %}) where metadata can be added or updated. This reduces the complexity of keywords and set the scene for controlled vocabularies environment. This increases the effectiveness of metadata filtering, which decreases ambiguity and makes it easier to find and retrieve information.

## Give your training materials a unique identity

To support teachers and trainers, the GTN tutorials rely on specific tags and identifiers. Adding persistent identifiers (PIDs) to training materials makes them easy to cite and aids citation counting in research metric systems {% cite mcmurry2017identifiers %}. Providing PIDs ensure that tag/identifier will continue to work even if webpage location changes. Data used for tutorials are required to be stored at [Zenodo](https://zenodo.org/) and associated with [Digital Object Identifiers](https://www.doi.org/) (DOI).

## Register your training materials online
Sharing and publishing with the GTN training helps minimize the amount of time and effort required for instructors to prepare for and run their training courses and workshops, by providing templates and a complete training infrastructure. Online registry makes your content more discoverable and accessible to wider community. GTN tutorials are automatically registered on [TeSS](https://tess.elixir-europe.org/) (ELIXIR Training e-support system).

## Define access rules for your training materials
The GTN materials are designed to be flexible in their use, open access and community-driven. It is important to follow the hosting website disclaimers and keep materials metadata in place.

## Use an interoperable format for your training materials
GTN Tutorials aim to follow best practices in course design, so that they can be used in different environments. Content of the tutorials and slides are written in [Markdown]({% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md %}) and supported by templates. Metadata are stored in YAML and workflows in JSON. Self-contained structure allow others to tune or reuse existing content.

## Make your training materials (re)usable for trainers
To help others determine whether the training materials are relevant and adaptable to their particular situations, metadata published alongside training materials should include context and sufficient description including: contributor details, license, description, learning outcomes, audience, requirements, tags/keywords, duration and last revision date Applying the proper licence and tagging training materials with metadata can make it simpler for others to (re)use and adapt them.

GTN provide strong technical support and set of [contributing self learning material]({% link topics/contributing/index.md %}).

## Make your training materials usable for trainees
Prerequisites and learning outcomes are particularly helpful metadata. To be effective, learning objectives must be written using active verbs that describe the expected trainee behaviours as well as the knowledge, skills, and expertise they will have received. Rich metadata requirements and SMART—Specific (Measurable, Attainable, Relevant, and Time-bound) learning outcomes following Bloom's taxonomy {% cite chevron2014metacognitive %} helps to clarify which trainees will benefit most from the training. Self-learning structure of the GTN materials supported by slides and video walkthroughs/tours adds another layer of usability.

## Make your training materials contribution friendly
GTN have clear [guidelines]({% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %}) for contribution and involvement. Community supported CONTRIBUTING files and chat offer the chance to exchange information such as contact details, expectations for contributions, and more. All contributors should be listed and thanked in the acknowledgements; how to cite the tutorial and give credit to contributors can be found at the end of each tutorial.

## Keep your training materials up-to-date
It is crucial to keep your training materials up-to-date and so you are aware of any new features, trends, or improvements in the topic (such as updated tools or databases). Transparent peer-review and curation process in collaborative and open set up ensure materials quality.

# Conclusion
There are many ways to improve the training content. It is crucial that we work together to make training materials FAIR so that everyone can benefit from them. These simple suggestions are intended to encourage dialogue and cooperation among Galaxy training community and help bring the latest developments to users.

The Galaxy Training Network is an example of a robust, effective Community of Practice.
For more information please look at this great article {% cite hiltemann2023galaxy %}, the corresponding FAIR guidelines {% cite fair-training-materials %} and follow [short introduction to FAIR data stewardship](http://fellowship.elixiruknode.org/).


topic: contributing
tutorial: fair-by-design
title: FAIR-by-Design methodology
difficulty: introductory
time_estimation: 20M
tools_json: []
requirements_json: []
tags_json: ["'FAIR-by-Design Learning Materials'", "\"FAIR Learning Objects\"", "FAIR-by-Design Methodology"]
content_hash: a6f71fd3007c18c714a6e3d99c3f09da
last_modified: 2026-02-26T21:12:14.708402
zenodo_link: https://zenodo.org/records/11548062
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/contributing/tutorials/fair-by-design/tutorial.html

content:
Welcome to the FAIR-by-Design Methodology Microlearning GTN adapted tutorial.

> <agenda-title></agenda-title>
>
>Below are the main stages of the FAIR-by-Design Methodology {% cite filiposka_2023_8305540 %} that will help guide you on your journey of creating FAIR learning materials.
>
>![Visual representation of the FAIR-by-Design Methodology flow with its separate stages](../../images/methodology.png)
>
>The FAIR-by-Design Methodology is defined in [a number of stages](#main-stages) that help you incorporate the FAIR principles into the backward learning instructional design process so that the final output are FAIR learning materials from both learners and instructors perspective:
>
>1. [Prepare](#stage-1---prepare)
>	- Do you understand FAIR and its implications
>	- Define purpose, learning objectives, target audience
>2. [Discover](#stage-2---discover)
>	- Find existing resources
>	- Identify potential for reuse
>3. [Design](#stage-3---design)
>	- Define the syllabus and structure
>	- Identify granularity
>	- Define facilitation materials
>4. [Produce](#stage-4---produce)
>	- Develop content using common apps and file formats
>	- Define machine readable metadata
>	- Perform internal QA check
>5. [Publish](#stage-5---publish)
>	- Define license and other related info
>	- Release for instructors and learners
>	- Enable feedback gathering
>6. [Verify](#stage-6---verify)
>	- Final QA check
>	- Use gathered feedback for continuous improvement
>7. [Continuous Improvement](#stage-7---continuous-improvement)
>	- Create a list of potential improvements
>	- Choose a set of improvements to be implemented
>	- Start a new release cycle
>
>Review each stage and the essential steps that it includes.
>
>Essential information about the original microlearning unit that this adaptation is based on can be found in the [About](#about) section.
>
{: .agenda}

If you have any questions at any stage on your journey do not hesitate to [contact the FAIR-by-Design methodology team](mailto:sonja.filiposka@finki.ukim.mk).

May all your materials be FAIR!


# Main stages

## Stage 1 - Prepare

> Before anything else, preparation is the key to success!
{: .quote author="Alexander Graham Bell"}

### First things first: What is FAIR?

><question-title> What is FAIR? </question-title>
>
>> <solution-title>Findable</solution-title>
>> The editable learning material has a unique and persistent identifier (PID) and is described with sufficiently detailed metadata.
>{: .solution}
>
>
>> <solution-title>Accessible</solution-title>
>> The human and machine readable metadata and object are stored in a trusted repository with clear authentication and authorization procedures.
>{: .solution}
>
>
>> <solution-title>Interoperable</solution-title>
>> The metadata describing the learning material follows a the RDA minimum metadata schema combined with agreed-upon controlled vocabularies.
>> Formal, accessible, shared, and broadly applicable language(s) and format(s) are used to develop the material.
>{: .solution}
>
>
>> <solution-title>Reusable</solution-title>
>>The learning material has a clear usage license (CC-BY-4.0 recommended) and accurate information on provenance.
>{: .solution }
{: .question }

> <details-title>More details on the FAIR principles</details-title>
> [Follow the FAIR in a nutshell tutorial available on GTN]( {% link topics/fair/tutorials/fair-intro/tutorial.md %} )
>
> [Follow the FAIR Galaxy Training Material tutorial available on GTN]( {% link topics/fair/tutorials/fair-gtn/tutorial.md %} )
{: .details}

### Adopt a metadata schema

If you are not using a discipline specific metadata schema, then, to ensure that your learning materials are appropriately described using a common approach, you should adopt the:

  <a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects/#rda-minimal-metadata-for-learning-resources" class="btn btn-primary stretched-link">Go to the RDA Minimal Metadata Set for Learning Resources description details</a>

The following are the fields described in the RDA Minimal Metadata Set divided into three categories:

> <details-title>Descriptive info metadata fields</details-title>
> - **Title** =	The human readable name of the resource
> - **Abstract / Description** =	A brief synopsis about or description of the learning resource
> - **Author(s)** =	Name of entity(ies) authoring the resource
> - **Primary Language** =	Language in which the resource was originally published or made available
> - **Keyword(s)** =	Keywords or tags used to describe the resource
> - **Version Date** =	Version date for the most recently published or broadcast resource
{: .details}

> <details-title>Access info metadata fields</details-title>
> - **URL to Resource** =	URL that resolves to the learning resource or to a "landing page" for the resource that contains important contextual information including the direct resolvable link to the resource, if applicable.
> - **Resource URL Type**	 = Designation of the identifier scheme used for the resource URL, e.g., DOI, ARK, Handle
> - **License** =	A license document that applies to this content, typically indicated by URL
> - **Access Cost**	= Choice stating whether or not there is a fee for use of the resource (yes, no, maybe)
{: .details}

> <details-title>Educational info metadata fields</details-title>
> - **Target Group (Audience)** =	Principal users(s) for which the resource was designed
> - **Learning Resource Type** =	The predominant type or kind that characterizes the learning resource
> - **Learning Outcome** =	Descriptions of what knowledge, skills or abilities a learner should acquire on completion of the resource
> - **Expertise (Skill) Level** =	Target skill level in the topic being taught; example values include beginner, intermediate, advanced
{: .details}

More on the RDA minimal metadata schema here {% cite hoebelheinrich_2022 %}

> <tip-title>Using metadata in GTN tutorials</tip-title>
>
>- When developing materials in GTN, this information should be included in the Tutorial.md file header metadata.
>- Most of the fields are already defined in the GTN Tutorial metadata schema. The ones that are missing can be added using the "extra" field.
>
> > <details-title>More details on the GTN Metadata</details-title>
> > [Follow the GTN Metadata tutorial]( {% link topics/contributing/tutorials/schemas/tutorial.md %} )
> {: .details}
>
{: .tip}


><details-title>Learn more about FAIR learning objects</details-title>
>- [FAIR instructional design skills](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/01-FAIR%20skills%20%26%20principles/01-FAIR%20skills%20%26%20principles/#fair-instructional-design-skills)
 >- [FAIR guiding principles](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/01-FAIR%20skills%20%26%20principles/01-FAIR%20skills%20%26%20principles/#fair-guiding-principles)
 >- [FAIR learning objects](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/01-FAIR%20skills%20%26%20principles/01-FAIR%20skills%20%26%20principles/#fair-learning-objects)
{: .details}

### Start Ideating

<div class="row">
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-header"><i class="fa fa-fast-backward" aria-hidden="true"></i> Think backward</div>
      <div class="card-body">
        <h5 class="card-title">Step 1</h5>
        <p class="card-text">What are your desired effects, i.e. learning outcomes?</p>
      </div>
    </div>
  </div>

  <div class="col-sm-4">
    <div class="card bg-light mb-3" >
      <div class="card-header"><i class="fa fa-fast-backward" aria-hidden="true"></i> Think backward</div>
      <div class="card-body">
        <h5 class="card-title">Step 2</h5>
        <p class="card-text">How are you going to assess the learners achievement?</p>
      </div>
    </div>
  </div>

  <div class="col-sm-4">
    <div class="card bg-light mb-3" >
      <div class="card-header"><i class="fa fa-fast-backward" aria-hidden="true"></i> Think backward</div>
      <div class="card-body">
        <h5 class="card-title">Step 3</h5>
        <p class="card-text">How should you structure the material to provide effective learning experience?</p>
      </div>
    </div>
  </div>

</div>
><tip-title>Learn more about the backward learning process</tip-title>
> - [Steps of the backward learning process](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects_cont/#backward-instructional-design-process)
{: .tip }

### Define

> <question-title>Purpose</question-title>
> When and how the learning materials can be used and for what purposes?
{: .question }

> <question-title>Target Audience</question-title>
> Is there anything specific that needs to be taken into account, such as cultural context?
{: .question }

> <question-title>Prerequisites</question-title>
> What does the target audience need to know or understand before starting the learning process?
{: .question }

> <question-title>Scope</question-title>
> Is it going to be a single learning unit, or a group such as a course?
{: .question }

> <question-title>Learning Objectives</question-title>
> What competences will be gained after successful completing of the learning process?
>
> ><tip-title>Be SMART</tip-title>
> >Objectives should be **s**pecific, **m**easurable, **a**ttainable, **r**elevant and **t**ime-bound.
> {: .tip}
>
> ><tip-title>Use Blooms Taxonomy</tip-title>
> >Formulate the objectives as actionable verb + observable knowledge, skill, attitude, behavior or ability.
> {: .tip}
>
> [Read more](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects_cont/#defining-learning-objectives)
{: .question }

> <details-title>More details on ideating learning outcomes and related information</details-title>
> [Read about defining intended Learning Outcomes in the Design and plan session, course, materials tutorial available on GTN]( {% link topics/contributing/tutorials/design/tutorial.md#define-intended-learning-outcomes-los %} )
{: .details}

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/411-prepare/" class="btn btn-dark text-white stretched-link">Go to the full description of FAIR-by-Design Methodology: Prepare stage ...</a>

## Stage 2 - Discover

> Greater even than the greatest discovery is to keep open the way to future discovery.
{: .quote author="John Jacob Abel"}
### Get inspired

Reusable materials can be found anywhere. These are just some examples:

**GTN**

- [GTN learning resources](https://training.galaxyproject.org)
- [TeSS Catalogue by Elixir](https://tess.elixir-europe.org)

**OER**

- [DOAB](https://directory.doabooks.org/)
- [MERLOT](https://www.merlot.org/merlot/index.htm)
- [OASIS](https://oasis.geneseo.edu/index.php)
- [OER Commons](https://www.oercommons.org/)
- [OERTX CORA](https://www.projectcora.org/)
- [GALILEO](https://oer.galileo.usg.edu/)
- [FORRT](https://forrt.org/)

**EOSC**

- [EOSC Training catalogue on the EOSC Marketplace](https://search.marketplace.eosc-portal.eu/search/training?q=*)
- Most EOSC projects have their own training catalogues and/or platforms.

**General**

- [Creative Commons Search](https://search.creativecommons.org/) - content provided under a CC license
- [Zenodo](https://zenodo.org/) - a multi-disciplinary open repository
- [OSF](https://osf.io/) - a free, open research platform

### Potential for reuse

> <warning-title> Respect the licenses, to be respected!</warning-title>
> - Materials with non-permissible licenses can be used for inspiration only.
> - Materials with permissible licenses should be reused based on the license rules.
{: .warning}

### Don't forget the multimedia search

Different learners have different learning modalities (read/write, auditory, visual, kinesthetic). To elevate the learning experience you should use all types of multimedia in your learning materials.

[Go to the full description of FAIR-by-Design Methodology: Discover stage ...](https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/412-discover/){: .btn.btn-dark.text-white.stretched-link}

## Stage 3 - Design

> Design is intelligence made visible.
{: .quote author="Alina Wheeler"}

Now it's time to brainstorm.

### Concept map

Step 1: Build a concept map of your learning materials.

> <details-title>More details on concept maps</details-title>
> [Read about Concept Maps in the Design and plan session, course, materials tutorial available on GTN]( {% link topics/contributing/tutorials/design/tutorial.md#concept-maps %} )
{: .details}

Step 2: Make sure you align your concept map with the MVS profiles.

The aligned MVS profile can help you crystallize the learning objectives using the MVS taxonomy.

Each MVS profile defines a list of technical and soft skills required for the profile. Think on how to incorporate both aspects in your learning materials.

<a href="https://fair-by-design-methodology.github.io/MVS/latest/MVS%20Profiles/Civil%20Servant/civil_servant/" class="btn btn-primary stretched-link">Go to the MVS profiles catalogue</a>

### Structure is everything

><comment-title></comment-title>
>{%icon galaxy-gear %} Use an intuitive logical organisation of all learning materials.
{: .comment}

><comment-title></comment-title>
>{%icon tool-versions %} The goal is for other people to easily reuse a single item (plan, activity, unit, assessment, ...).
{: .comment}

><comment-title></comment-title>
>{%icon galaxy-dataset-map %} Take advantage of a hierarchical structure to combine learning units into larger compositions.
{: .comment}

><hands-on-title>How to organise the files</hands-on-title>
>
> GTN defines a specific hierarchical structure that needs to be followed when developing learning materials.
>
> > <details-title>Check out the following tutorials</details-title>
> > [How to create a skeleton for a new learning topic]( {% link topics/contributing/tutorials/create-new-topic/tutorial.md#creating-a-new-topic-with-its-own-materials %}  )
> >
> > [How to create the skeleton of a new tutorial]( {% link topics/contributing/tutorials/create-new-tutorial/tutorial.md %}  )
> {: .details}
>
{: .hands-on}

><comment-title>Syllabus is ready</comment-title>
> You should by now have the first draft of your <a href='https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%203%20–%20Design/04-Conceptualisation/04-Conceptualisation/#syllabus/'>syllabus</a>. It contains all the fields from the RDA min metadata set plus the high level topics covered by the learning material.
{: .comment}

><tip-title>Available feedback form</tip-title>
> GTN provides a readily available feedback form that is automatically added at the end of each tutorial. The feedback form is used to gather quantitative and qualitative feedback.
>
> > <details-title> Learn more about feedback in training</details-title>
> > [Follow the Assessment and feedback in training and teachings tutorial available on GTN]( {% link topics/teaching/tutorials/assessment/tutorial.md#dealing-with-feedback %}  )
> {: .details}
>
{: .tip}

><tip-title>Facilitation guide kit</tip-title>
> A <a href='https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%203%20–%20Design/07-Facilitation/07-Facilitation/'> facilitation guide </a> should help prepare for the actual training.
>
> The facilitation guide kit includes documentation of the process of organising and running a training. What different people need to have, know and do so that everything runs smoothly.
>
> > <details-title> Learn more about facilitation</details-title>
> > [Follow the Organizing a workshop tutorial available on GTN]( {% link topics/teaching/tutorials/organize-workshop/tutorial.md %}  )
> >
> > [Follow the Running a workshop as instructor tutorial available on GTN]( {% link topics/teaching/tutorials/running-workshop/tutorial.md %}  )
> {: .details}
>
> Another option is to use something like the [TRIPLE project TRAINING TOOLKIT](https://project.gotriple.eu/project-deliverables/triple-training-toolkit/).
>
{: .tip}

><question-title>What about instructor notes?</question-title>
>  They need to be detailed enough so that anyone can reuse the learning content, especially slides properly.
>
>  Any specific information relevant for instructors that would like to organise a training based on your GTN tutorial content should be added in Details box.
>
>  This practice significantly increases the reuse potential of the material.
{: .question}

### How to design the learning content

<a href="https://www.csun.edu/sites/default/files/Holle-Lesson-Planning.pdf" class="btn btn-primary stretched-link">Read more about the Hunter Model.</a>

<div class="row">
  <div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon curriculum %} 1. Set Learning Objectives</h5>
        <p class="card-text">
         What is the goal?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon pref-identities %} 2. Identify Needs</h5>
        <p class="card-text">
        How to get there?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-panelview %} 3. Plan</h5>
        <p class="card-text">
        Share the agenda.
        </p>
      </div>
    </div>
  </div>
</div>

<div class="row">
  <div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon version %} 4. Hook</h5>
        <p class="card-text">
        Why is the content important?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-wf-edit %} 5. Instruct</h5>
        <p class="card-text">
        Watch how I do it.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-rulebuilder-history %} 6. Practise</h5>
        <p class="card-text">
        You help me do it, I'll watch you do it.
        </p>
      </div>
    </div>
  </div>
</div>

<div class="row">
  <div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-barchart %} 7. Wrap-Up</h5>
        <p class="card-text">
         Foster retention and reinforcement.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon license %} 8. Evaluate</h5>
        <p class="card-text">
        Monitor progress.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <div class="card bg-light mb-3" >
      <div class="card-body">
        <h5 class="card-title">{% icon galaxy-history-answer %} 9. Reflect</h5>
        <p class="card-text">
        How did it go?
        </p>
      </div>
    </div>
  </div>
</div>

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/413-design/" class="btn btn-dark text-white stretched-link">Go to the full description of FAIR-by-Design Methodology: Design stage ...</a>

## Stage 4 - Produce

> To contrive is nothing! To construct is something! To produce is everything!
{: .quote author="Edward Rickenbacker"}

### Choose Tools & Formats

> <tip-title>Collaborative environment for team work</tip-title>
>  Choose an environment for producing the learning material that will enable multiple people to work on the same material at one.
>
>  - GitHub is one of the most popular options at the moment (find out more <a href='https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%204%20–%20Produce/08-Development%20Tools/08-Introduction%20to%20Markdown%20and%20Git/'>here</a>)
>  - GTN is also based on GitHub and defines [specific procedures on how one can contribute and collaborate](https://github.com/galaxyproject/training-material/blob/main/CONTRIBUTING.md) using GitHub
>
>  > <details-title> GTN: collaboration using GitHub</details-title>
> > [Follow the Contributing with GitHub via its interface tutorial available on GTN]( {% link topics/contributing/tutorials/github-interface-contribution/tutorial.md %}  )
> >
> > [Follow the Contributing with GitHub via command-line tutorial available on GTN]( {% link topics/contributing/tutorials/github-command-line-contribution/tutorial.md %}  )
> {: .details}
{: .tip}

> <tip-title>Granular versioning for easy rollback</tip-title>
>    - Versioning helps you maintain control over your changes.
>    - GitHub natively provides versioning and history retention that help easy roll back to an earlier stable state.
{: .tip}

> <tip-title>Open file formats to foster reuse</tip-title>
> - For other people to reuse the materials they should be made available using open file formats.
> - GTN promotes the use of the MD open file format for the main learning content empowered with open scientific notebooks and workflows.
> - If you use close file formats then you MUST clearly state the tools that have been used in more details.
{: .tip}

> <tip-title>Multimodal content to reach all audience</tip-title>
>  - Don't forget to include different types of multimedia to provide support for different learning modalities: read/write, auditory, visual, kinesthetic.
>  - In addition to hands-on exercises, GTN also provides support for audio and video modalities.
{: .tip}

> <tip-title>Two file sets: editable + final</tip-title>
> - The MD and supporting files are the main files used for development of the content. These files are what matters for you and other instructors.
> - Based on the editable files, GTN automatically generates the final (non-editable version) in HTML (and PDF). These  are shared with the learners.
> - GTN takes care of the revision numbering for you.
{: .tip}

> <tip-title>Don't forget to take advantage of co-creation</tip-title>
> Truly FAIR learning materials enable co-creation with external parties.
>  GitHub is a collaborative environment that supports co-creation in every step of the learning materials development and revision process.
>  The GTN fork-and-pull process of contributing to the learning material is a clear example of co-creation implementation.
{: .tip}


### Plan to reuse existing material?

#### Check the license

The existing materials you are reusing are available under a [CC license](https://creativecommons.org/share-your-work/cclicenses/), but it is different than the one you plan to use for your materials. Depending on how you want to reuse the material, you will need to consider the following aspects.

><details-title>I want to reuse it as a whole</details-title>
>- You can't use something that is licensed with ND (no derivatives).
>- In this case you must follow the rules on combining and adapting CC material.
>
>[Read more](https://creativecommons.org/faq/#combining-and-adapting-cc-material)
{: .details}

><details-title>I want to reuse a small part of it</details-title>
> No problem, you can reuse any existing CC licensed material in your learning materials as long as the reused portion is used as a showcase or to make a specific point and it is not the core of your work.
>
> Remember that if the work is licensed with ND, you can not modify it while reusing.
{: .details}

><tip-title>Learn more about IPR</tip-title>
>[How Intellectual Property Rights (IPR) protect the interests of the creators and owners by providing them with rights over their creation?](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects/#intellectual-property-rights-ipr)
{: .tip}

Still need help? [{% icon point-right %} Go to CC licensing FAQ](https://creativecommons.org/faq/#before-using-cc-licensed-material)


#### Attribute

All CC licenses require that you attribute the author, and this rule is recommended even if the license is public domain CC-0.

If the work you are reusing has a copyright notice ('© some text') you need to reproduce it while you credit the work.

You should also be able to remove attribution upon request.

##### How to attribute?

> <solution-title>Use the authors recommended attribution</solution-title>
> If the original author has provided a cite-as information, use it to attribute the work.
{: .solution}

> <solution-title>Use TASL</solution-title>
> Provide the Title, Author, Source and License of the work that you are reusing.
>
>- Source is the URL to the original work.
>- If there is a URL to the author personal pages, provide it together with the name.
>- Provide the name of the license and a URL to the license.
{: .solution}

> <details-title>Examples</details-title>
> Examples are taken from [Best Practices for Creative Commons attributions - how to attribute works you reuse under a Creative Commons license](https://www.newmediarights.org/guide/how_to/creative_commons/best_practices_creative_commons_attributions) submitted by [New Media Rights](https://www.newmediarights.org/) available under a [CC BY-NC 3.0 US DEED](https://creativecommons.org/licenses/by-nc/3.0/us/).
> - Webpage/Blog - Title (with link to original work), author (or username) (with link to author's website), and license (with link).
> 	- [Undercover Vampire Policeman](https://chriszabriskie.bandcamp.com/album/undercover-vampire-policeman) by [Chris Zabriskie](https://chriszabriskie.bandcamp.com/), available under a [Creative Commons Attribution 4.0 License](http://creativecommons.org/licenses/by/4.0/)
> - Book – Title, author, license written somewhere near the title and author if it’s a hard copy or if it’s an online book you should include a link to the licensed terms.
> 	- [From Dust to Digital: Ten Years of the Endangered Archives Programme](https://books.google.com/books?id=ImO3BgAAQBAJ&pg=PR4&dq=creative+commons+4.0&hl=en&sa=X&ved=0CEoQ6AEwCGoVChMIspCXhPPxxgIVSF0eCh27NA5X#v=onepage&q=creative%20commons%204.0&f=false) by Maja Kominko under a Creative Commons Attribution Non-commercial Non-Derivative 4.0 International license (CC BY-NC-ND 4.0)
> - Online Video - Title, author, license written into credits at end of video.  Ideally make the text clickable to the original work.  Put links to the original work and the license terms in the information section for the particular work (i.e. on the right in YouTube).
> 	- [http://www.youtube.com/watch?v=fDbbdeIXO0w#t=3m0s](http://www.youtube.com/watch?v=fDbbdeIXO0w#t=3m0s)
> - Podcast/Audio - Title, author, license read at the end of the entire work.
> 	- [“Je Suis Rick Springfield”](http://www.jonathancoulton.com/wiki/Je_Suis_Rick_Springfield) from the album [Artificial Heart](http://www.jonathancoulton.com/wiki/Artificial_Heart_(album)), by Jonathan Coulton, used under a [Creative Commons Attribution-Noncommercial 3.0 Unported License](http://creativecommons.org/licenses/by-nc/3.0/)
>- Photo/Drawing/Illustration – Title, author, license (with link online) or in close proximity to the tangible work (either in the border or directly on the work, if applicable).
> 	- [Comcast protest](http://www.flickr.com/photos/ari/8503459/in/set-214952/)” by Flikr user [Steve Rhodes](http://www.flickr.com/photos/ari/) used under [Creative Commons Attribution 2.0 license](http://creativecommons.org/licenses/by/2.0/deed.en)
{: .details}

> <details-title>Read more about how to attribute</details-title>
> [{% icon point-right %}  Attribution and Citing](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%201%20%E2%80%93%20Prepare/02-Preparing%20FAIR%20Learning%20Objects/02-Preparing%20FAIR%20Learning%20Objects/#attribution-and-citing)
>
> [{% icon point-right %} How to handle attribution?](https://courses.lumenlearning.com/suny-oerguide/chapter/how-to-handle-attribution/)
{: .details}


### Accessibility

The developed learning materials should cover the widest range of learner variability including the ones that use or do not use assistive technology.

![Universal access logo](../../images/universal-access-6602642_640.png)
<a href="https://pixabay.com/vectors/universal-access-human-icon-6602642/">"Universal Access Human"</a> by <a href="https://pixabay.com/users/inspire-studio-22128832/">J S</a> from <a href="https://pixabay.com/">Pixabay</a>


#### Standards

There are several standards that govern the rules on the level of accessibility.

Most commonly used is the <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">W3C Web Content Accessibility Guidelines (WCAG) standard version 2.1</a>. Three conformance levels exist, you should aim for AA which is the middle one.

PDF document accessibility is measured with a separate technical specification <a href="https://pdfua.foundation/en/why-pdf-ua/">PDF/UA (Universal Accessibility)</a>.


#### Accessibility guidelines

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%204%20%E2%80%93%20Produce/11-Accessibility/11-Checking_accessibility/#general-guidelines-for-development-of-accessible-materials" class="btn btn-primary stretched-link">Review some general guidelines for development of accessible material</a>

><tip-title>GTN Accessibility</tip-title>
> GTN has developed an <a href="https://training.galaxyproject.org/training-material/accessibility.html"> accessibility mission </a> aiming for WCAG 2.0 AA compliance by implementing a number of accessibility features.
>
{: .tip}


### Internal QA

<div class="row">
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon congratulations %} QA Self-assessment</h5>
        <p class="card-text">
         Check if everything is as it should be.
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon pref-list %} Quantitative</h5>
        <p class="card-text">
        Are all required elements produced?
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon solution %} Qualitative</h5>
        <p class="card-text">
        Do all learning units provide materials to reach the learning objectives with different modalities?
        </p>
      </div>
    </div>
  </div>
</div>

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/414-produce/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Produce stage ...</a>

## Stage 5 - Publish

> Publishing is the art of working on a creative idea and turning it into a masterpiece!
{: .quote author="Unknown"}

><warning-title>Publishing closed FAIR materials</warning-title>
> Having FAIR learning materials does not always mean that the materials are open to everyone and there are no costs or access rules attached. In this case the bundle that is going to be published in an open repository such as Zenodo should contain the following:
>  1. Syllabus, that contains all metadata that describe the materials. Metadata should always be open.
>  2. Accompanying information (optional) to augment the description of the materials and describe the details when it comes to accessing and using the materials from a trainer perspective.
>
> The complete learning materials package itself should be published in a closed repository where the corresponding access rules (and costs) can be implemented.
{: .warning}

### Final preparations

><tip-title>Time to provide the accompanying information</tip-title>
> Usually provided at (or near) the root of the learning materials, the purpose of the accompanying information is to further describe the content and define the rules of reuse.
{: .tip}

<div class="row">
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon text-document %} LICENSE</h5>
        <p class="card-text">
         The license defines the potential of reuse of your learning materials together with the rules regarding their attribution by others.<br>
         CC-BY-4.0 is the recommended license, which is default in GTN. <br>
         If you choose a different license, it must be supplied in the tutorial header as metadata.
         <br>
         <a href="https://creativecommons.org/licenses" > {% icon point-right %} Browse through the available CC license types
         </a>
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon sticky-note %} README</h5>
        <p class="card-text">
        A README is a text file that introduces and explains the contents of the learning topics and materials. It usually describes the context and written in a plain text format. <br> GTN provides a README file on the level of each topic, where context information about the topic and its tutorials can be provided.
        <br>
		<a href="https://www.makeareadme.com/">
        {% icon point-right %}  Make a README
        </a>
        </p>
      </div>
    </div>
  </div>
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon cofest %} CONTRIBUTORS</h5>
        <p class="card-text">
        GTN CONTRIBUTORS is a YAML file that lists the information about all contributors of learning materials on the platform.
        <br>You must have a GitHub user to be listed as a contributor. Additional information such as ORCID can also be provided.
        <br>
        <a href="{% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md#listing-contributors %}">
		{% icon point-right %} Add to CONTRIBUTORS.yaml
		</a>
        </p>
      </div>
    </div>
  </div>
</div>

<div class="row">
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon help %} FAQs</h5>
        <p class="card-text">
	        In addition to the learning content, the Frequently Asked Questions can further help both learners and instructors with specific information about the learning context or practicalities.
	        <br> Consider adding a tutorial-specific FAQ to the GTN tutorial, where you can answer questions about the challenges of working with the hands on activities, or provide other hints and guidelines to instructors and learners.
	        <br>
		<a href="{% link topics/contributing/tutorials/create-new-tutorial-content/tutorial.md#creating-new-faqssnippets %}">
        {% icon point-right %}  Create snippets
        </a>
        </p>
      </div>
    </div>
  </div>
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon event-cost %} FUNDERS</h5>
        <p class="card-text">
        If the creation of the learning materials is funded by an organisation or another body, they should be attributed accordingly. <br>
        In GTN this can be accomplished by adding the funding information into the special FUNDERS.yaml file.
         <br>
         <a href="https://github.com/galaxyproject/training-material/blob/main/FUNDERS.yaml">
         {% icon point-right %} Add to the Funders file
         </a>
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon new-history %} OTHER</h5>
        <p class="card-text">
        Additional miscellaneous information should be provided if possible.
        <br> For example, GTN also supports adding organisations information.
        <br>
        <a href="https://github.com/galaxyproject/training-material/blob/main/ORGANISATIONS.yaml">
        {% icon point-right %} Add org info
        </a>
		</p>
      </div>
    </div>
  </div>
</div>

<div class="row">
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon references %} CITATION</h5>
        <p class="card-text">
	        It is best practice to provide information on how you want others to cite your learning materials when they are referenced or reused. <br> GTN does this automatically by appending the "Citing this Tutorial" section at the end of each tutorial.
	        <br>
		<a href="https://training.galaxyproject.org/training-material/faqs/gtn/gtn_citing.html">
        {% icon point-right %}  See an example citation
        </a>
        </p>
      </div>
    </div>
  </div>
  <div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon license %} CODE OF CONDUCT</h5>
        <p class="card-text">
        A code of conduct defines the rules for how to engage in a co-creation community.
        <br>It is based on the premise of an inclusive environment that respects all contributions.
         <br>
         <a href="https://galaxyproject.org/community/coc/">
         {% icon point-right %} Read the Galaxy Project Code of Conduct
         </a>
        </p>
      </div>
    </div>
  </div>
<div class="col-sm-4">
   <!-- <div class="card text-white bg-secondary mb-3" > -->
   <div class="card bg-light mb-3" >
      <!-- <div class="card-header text-white"> -->
      <div class="card-body">
        <h5 class="card-title">{% icon comment %} REVISIONS</h5>
        <p class="card-text">
        Information about new or updated versions of your learning materials helps others keep track of the changes more easily. <br> GTN automatically updates the Revision number of each tutorial after each release.
        <br>
        <a href="https://github.com/galaxyproject/training-material/releases">
        {% icon point-right %} Compare changes between GTN releases
        </a>
		</p>
      </div>
    </div>
  </div>
</div>

### Storing and indexing

><tip-title>Automated publishing</tip-title>
>If you are working on GTN following the provided rules and procedures, GTN will automatically publish the new tutorial once the full review and release process is completed.
>
>Just follow the [GTN guide to contributing via GitHub]( {% link topics/contributing/#st-contribute %}).
{: .tip}

><tip-title>To training catalogue</tip-title>
>You are at the point when you should also consider making a record in a relevant training catalogue.
>
>GTN automatically creates a new record in the [Elixir TeSS training catalogue](https://tess.elixir-europe.org) that is the most relevant catalogue for its community.
{: .tip}

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/415-publish/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Publish stage ...</a>

## Stage 6 - Verify

> You will find it a very good practice always to verify your references, sir!
{: .quote author="Martin Routh"}

### External QA

> <tip-title> A fresh set of eyes</tip-title>
> - Have someone who has not participated in the development of the learning materials review the final work. This will guarantee a review free of cognitive bias.
> - GTN implements this automatically via the fork-and-pull request process. Before publication your new contribution is being reviewed by GTN peers.
{: .tip}

> <tip-title> Go through the QA checklists</tip-title>
> In Skills4EOSC T2.4 has developed a number of QA checklists that you and your external reviewer need to go through so that you can ensure high-quality learning materials {% cite sanchez_2023_8305482 %}.
{: .tip}

### FAIR or not FAIR, that is the question

#### Measure FAIRness

Use the [FAIR-by-Design methodology QA checklist](https://fair-by-design-methodology.github.io/FAIR-by-Design_ToT/latest/Stage%206%20–%20Verify/19-Final%20QA%20check/19-finalQA/#fair-by-design-methodology-qa-checklist) to check if you have followed the most important aspects of the methodology and managed to produce FAIR learning materials.

The questions marked as essential achieve bare minimum FAIRness.

><details-title>Essential requirements</details-title>
> - **Findable**
>   - Is the complete learning resource (including instructors info) registered or indexed in at least one searchable repository? Is it in a FAIR repository?
>   - Is metadata for the resource provided in both human- and machine-readable format (e.g JSON, XMLor YAML)?
> - **Accessible**
>   - Has an accessibility checker tool been utilised to improve the accessibility of all learning resource files (PDF, HTML, video, etc.)?
>   - Are access rules (authentication & authorisation) implemented for the learning resource?
> - **Interoperable**
>   - Is the RDA minimal (or domain specific) metadata schema used for the learning material description?
>   - Is the resource available in open file formats which are tool agnostic and compatible with a wide variety of existing software?
> - **Reusable**
>   - Is there clear attribution for all reused resources with compatible licenses?
>   - Has the learning resource been made available for use by defining a permissable license or policy information that allows derivations?
{: .details}

><details-title>Optional requirements</details-title>
> - Did you follow the stages of the backward instructional design process while developing the learning resource?
> - Are controlled vocabularies (CVs) used for describing the resource characteristics aligned with the chosen metadata schema?
> - Does the learning resource represent a complete learning object defined around minimum one learning objective?
> - Does the resource incorporate an instructor kit that aids in facilitating the process of others reusing learning material by offering helpful how-to guides?
>    - facilitator guide
>    - activities description
>    - assessment activities and strategy to assess
>    - general learning content or instructor notes
>    - lesson unit plan
>    - syllabus
> - Have you employed a versioning system to track and control changes in your materials?
> - Are the resource access rules (how to access, e.g. registration procedure) explicitly communicated to learners?
> - Is the learning resource searchable in at least one relevant catalogue?
>    - Is it FAIR (can be searched based on metadata)?
> - Does the course include the possibility to provide feedback or comments from users and-or trainers/designers?
>    - If so, do you regularly gather and analyse that feedback?
> - Does the resource adopt an open community approach regarding its quality and reachability?
> - Has the learning resource been checked by a third party regarding its learning experience quality?
{: .details}

### Feedback QA

> <comment-title>Regularly gather feedback from learners and instructors</comment-title>
> Ensure that you actively and regularly gather feedback from both perspectives: the learners and the instructors.
{: .comment}

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.1%20-%20Workflow%20stages%20description/416-verify/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Verify stage ...</a>

## Stage 7 - Continuous Improvement
> To improve is to change; to be perfect is to change often
{: .quote author="Winston Churchill"}

### {% icon galaxy-download %} Gather

- Gather feedback from all available internal & external sources.
- Potential sources:
	- Feedback form
	- QA recommendations
	- Self-reflection after training
	- Git Issues
	- Matrix Chat
	- Direct mail contact
	- Other means of communication

### {% icon galaxy-barchart %} Analyse

- Analyse the gathered information in a structured way.
- Create a list of potential improvements with impact level (high, moderate, low).

### {% icon galaxy-wf-edit %} Improve

- Select items from the list that will be part of a new version.
- Choose items that make sense to be in the same new release.

### {% icon galaxy-history-refresh %} Repeat

- Start a new cycle of the FAIR-by-Design methodology that will implement the selected items.
- After the Verify stage, you will reenter continuous improvement with the newly gathered information.

<a href="https://fair-by-design-methodology.github.io/FAIR-by-Design_Book/4%20-%20FAIR-by-design%20learning%20materials%20creation/4.2%20-%20Continuous%20Improvement/417-improvement/" class="btn btn-dark text-white stretched-link">Go to the full FAIR-by-Design Methodology: Continuous Improvement ...</a>

# About

## FAIR-by-Design Microlearning

### Self-paced how to FAIR-by-Design guide

### Original Location

- online: [https://fair-by-design-methodology.github.io/microlearning/latest/](https://fair-by-design-methodology.github.io/microlearning/latest/)
- this tutorial is an adapted version of the original microlearning guide

### Training Description

Short how-to that guides you through the stages of the FAIR-by-Design methodology without any specific choice on tools and formats.

### Target audience

All interested parties who need to develop learning materials for any type of project-related training.

### Expertise Level / Skill Level: Beginner

### Primary Language: English

### Access Cost: No

### Prerequisites

No prior knowledge is required.

### Learning Objectives

- Design FAIR learning materials
- Structure FAIR learning materials
- Create and publish FAIR-by-Design learning materials
- Assess FAIR-ness of learning objects

### Keywords

FAIR, learning objects, methodology, practical implementation

### Certification Information

This training has no certification or any recognition mechanism included.

### Original Author(s)

- Sonja Filiposka, Dominique Green, Anastas Mishev, Vojdan Kjorveziroski, Andrea Corleto, Eleonora Napolitano, Gabriella Paolini, Sara di Giorgio, Joanna Janik, Luca Schirru, Arnaud Gingold, Christine Chardosek, Irakleitos Souyioultzoglou, Carolin Leister, Emma Lazzeri


### Contact information

- For more information please contact the T2.3 FAIR-by-Design Methodology Task Leader Sonja Filiposka using [sonja.filiposka@finki.ukim.mk](mailto:sonja.filiposka@finki.ukim.mk).

### Original License

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.

### Original microlearning unit DOI

[https://doi.org/10.5281/zenodo.11548062](https://doi.org/10.5281/zenodo.11548062)

#### Acknowledgement

These learning materials have been developed by following the [FAIR-by-Design Methodology](https://zenodo.org/records/8305540).


topic: introduction
tutorial: vsi_qc
title: Very Short Introductions: QC
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: f0e0da4039c224fd2941b9babc96de80
last_modified: 2026-02-26T21:12:17.043566
zenodo_link: https://zenodo.org/records/10870107
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/vsi_qc/tutorial.html

content:
In this tutorial we will look at assessing quality of data from two short read technologies: [Illumina](http://www.nature.com/doifinder/10.1038/nature07517) and [Element Biosciences](http://dx.doi.org/10.1038/s41587-023-01750-7). 

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## FASTQ format

[FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) is not a very well defined format. In the beginning various manufacturers of sequencing instruments were free to interpret FASTQ as they saw fit, resulting in a multitude of FASTQ flavors. This variation stemmed primarily from different ways of encoding quality values as described [on the Wikipedia article for FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) (below you will find an explanation of quality scores and their meaning). Today, the [FASTQ Sanger](https://www.ncbi.nlm.nih.gov/pubmed/20015970) version of the format is considered to be the standard form of FASTQ. Galaxy is using FASTQ Sanger as the only legitimate input for downstream processing tools and provides [a number of utilities for converting FASTQ files](https://www.ncbi.nlm.nih.gov/pubmed/20562416) into this form (see **FASTQ Quality Control** section of Galaxy tools).

The FASTQ format looks like this:


```

@M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCTGAACCAGCCAAGTAGCGTGCAG
+
ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE<,,CD,CEFC,@E9<FCFCF?9
@M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACTGATCCAGCCATGCCGCGTGCAG
+
ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF<,CEF,CFGGD8FFCFCFEGCF
@M02286:19:000000000-AA549:1:1101:11116:1322 1:N:0:23
CCTACGGGAGGCAGCAGTAGGGAATCTTCGGCAATGGACGGAAGTCTGACCGAGCAACGCCGCGTGAGT
+
AAC<CCF+@@>CC,C9,F9C9@9-CFFFE@7@:+CC8-C@:7,@EFE,6CF:+8F7EFEEF@EGGGEEE

```

Each sequencing read is represented by four lines:

1. `@` followed by read ID and optional information about sequencing run
2. sequenced bases
3. `+` (optionally followed by the read ID and some additional info)
4. quality scores for each base of the sequence encoded as [ASCII symbols](https://en.wikipedia.org/wiki/ASCII)

## Paired end data

It is common to prepare pair-end and mate-pair sequencing libraries. This is highly beneficial for a number of applications discussed in subsequent topics. For now let's just briefly discuss what these are and how they manifest themselves in FASTQ form.

![Paired-end and mate-pair reads](../../images/pe_mp.png "<b>Paired-end and mate-pair reads</b>. In paired end sequencing (left) the actual ends of rather short DNA molecules (less than 1kb) are determined, while for mate pair sequencing (right) the ends of long molecules are joined and prepared in special sequencing libraries. In these mate pair protocols, the ends of long, size-selected molecules are connected with an internal adapter sequence (i.e. linker, yellow) in a circularization reaction. The circular molecule is then processed using restriction enzymes or fragmentation. Fragments are enriched for the linker and outer library adapters are added around the two combined molecule ends. The internal adapter can then be used as a second priming site for an additional sequencing reaction in the same orientation or sequencing can be performed from the second adapter, from the reverse strand. (From <i>Understanding and improving high-throughput sequencing data production and analysis</i>, Ph.D. dissertation by <a href='https://ul.qucosa.de/api/qucosa%3A11231/attachment/ATT-0/'>Martin Kircher</a>)")


Thus in both cases (paired-end and mate-pair) a single physical piece of DNA (or RNA in the case of RNA-seq) is sequenced from two ends and so generates two reads. These can be represented as separate files (two FASTQ files with first and second reads) or a single file were reads for each end are interleaved. Here are examples:

### Two single files

File 1

```
 @M02286:19:000000000-AA549:1:1101:12677:1273 1:N:0:23
 CCTACGGGTGGCAGCAGTGAGGAATATTGGTCAATGGACGGAAGTCT
 +
 ABC8C,:@F:CE8,B-,C,-6-9-C,CE9-CC--C-<-C++,,+;CE
 @M02286:19:000000000-AA549:1:1101:15048:1299 1:N:0:23
 CCTACGGGTGGCTGCAGTGAGGAATATTGGACAATGGTCGGAAGACT
 +
 ABC@CC77CFCEG;F9<F89<9--C,CE,--C-6C-,CE:++7:,CF
```

File 2

```
@M02286:19:000000000-AA549:1:1101:12677:1273 2:N:0:23
CACTACCCGTGTATCTAATCCTGTTTGATACCCGCACCTTCGAGCTTA
+
--8A,CCE+,,;,<CC,,<CE@,CFD,,C,CFF+@+@CCEF,,,B+C,
@M02286:19:000000000-AA549:1:1101:15048:1299 2:N:0:23
CACTACCGGGGTATCTAATCCTGTTCGCTCCCCACGCTTTCGTCCATC
+
-6AC,EE@::CF7CFF<<FFGGDFFF,@FGGGG?F7FEGGGDEFF>FF
```

> <comment-title>Read order is important</comment-title>
> Note that read IDs are **identical** in two files and they are listed in **the same** order. In some cases read IDs in the first and second file may be appended with `/1` and `/2` tags, respectively.
{: .comment}

### Interleaved file

```
@1/1
AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
+
EGGEGGGDFGEEEAEECGDEGGFEEGEFGBEEDDECFEFDD@CDD<ED
@1/2
CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
+
GHHHDFDFGFGEGFBGEGGEGEGGGHGFGHFHFHHHHHHHEF?EFEFF
@2/1
AGGGATGTGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTA
+
HHHHHHEGFHEEFEEHEEHHGGEGGGGEFGFGGGGHHHHFBEEEEEFG
@2/2
CCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAAC
+
HHHHHHHHHHHHHGHHHHHHGHHHHHHHHHHHFHHHFHHHHHHHHHHH
```

Here the first and the second reads are identified with `/1` and `/2` tags.

> <comment-title>FASTQ format is a loose standard</comment-title>
> FASTQ format is not strictly defined and its variations will always cause headache for you. See [this page](https://www.ncbi.nlm.nih.gov/sra/docs/submitformats/) for more information.
{: .comment}

## What are base qualities?

As we've seen above, FASTQ datasets contain two types of information:

- *sequence of the read*
- *base qualities* for each nucleotide in the read.

The base qualities allow us to judge how trustworthy each base in a sequencing read is. The following excerpt from an excellent [tutorial](http://chagall.med.cornell.edu/RNASEQcourse/Intro2RNAseq.pdf) by Friederike Dündar, Luce Skrabanek, Paul Zumbo explains what base qualities are:

> <comment-title>From "Introduction to differential gene expression analysis using RNA-seq"</comment-title>
> Illumina sequencing is based on identifying the individual nucleotides by the fluorescence signal emitted upon their incorporation into the growing sequencing read. Once the fluorescence intensities are extracted and translated into the four letter code. The deduction of nucleotide sequences from the images acquired during sequencing is commonly referred to as base calling.
><br><br>
> Due to the imperfect nature of the sequencing process and limitations of the optical instruments, base calling will always have inherent uncertainty. This is the reason why FASTQ files store the DNA sequence of each read together with a position-specific quality score that represents the error probability, i.e., how likely it is that an individual base call may be incorrect. The score is called [Phred score](http://www.phrap.com/phred/), $$Q$$, which is proportional to the probability $$p$$ that a base call is incorrect, where $$Q = −10lg(p)$$. For example, a Phred score of 10 corresponds to one error in every ten base calls ($$Q = −10lg(0.1)$$), or 90% accuracy; a Phred score of 20 corresponds to one error in every 100 base calls, or 99% accuracy. A higher Phred score thus reflects higher confidence in the reported base.
><br><br>
> To assign each base a unique score identifier (instead of numbers of varying character length), Phred scores are typically represented as ASCII characters. At http://ascii-code.com/ you can see which characters are assigned to what number.
><br><br>
> For raw reads, the range of scores will depend on the sequencing technology and the base caller used (Illumina, for example, used a tool called Bustard, or, more recently, RTA). Unfortunately, Illumina has been anything but consistent in how they calculated and ASCII-encoded the Phred score (see below)! In addition, Illumina now allows Phred scores for base calls with as high as 45, while 41 used to be the maximum score until the HiSeq X. This may cause issues with downstream sapplications that expect an upper limit of 41.
{: .comment}

![Illumina quality score](../../images/illumina_qs.png "Comparison between different encodings")

Base call quality scores are represented with the Phred range. Different Illumina (formerly Solexa) versions
used different scores and ASCII offsets. Starting with Illumina format 1.8, the score now represents the standard
Sanger/Phred format that is also used by other sequencing platforms and the sequencing archives.

![FASTQ quality score](../../images/fastq_qs.png "The ASCII interpretation and ranges of the different Phred score notations used by Illumina and the original Sanger interpretation. Although the Sanger format allows a theoretical score of 93, raw sequencing reads typically do not exceed a Phred score of 60. In fact, most Illumina-based sequencing will result in maximum scores of 41 to 45 (image from <a href='https://en.wikipedia.org/wiki/FASTQ_format'>Wikipedia</a>).")


# Assessing quality

This tutorial provides two sample datasets: one generated with Illumina and the other with ElemBio (AVITI) machine. Pick any. We will compare qualities of the two platforms at the and of this tutorial. 

## Get data

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the files from [Zenodo]({{ page.zenodo_link }}):
>
>    **Element** (AVITI). This datasets contains forward (r1) and reverse (r2) reads for two samples---`s1` and `s2`:
>
>    ```
>    https://zenodo.org/records/10870107/files/elem_s1_r1.fq.gz
>    https://zenodo.org/records/10870107/files/elem_s1_r2.fq.gz
>    https://zenodo.org/records/10870107/files/elem_s2_r1.fq.gz
>    https://zenodo.org/records/10870107/files/elem_s2_r2.fq.gz
>    ```
>
>    **Illumina**. This datasets contains forward (_1) and reverse (_2) reads for two samples---`bl` and `ch`:
>
>    ```
>    https://zenodo.org/record/5119008/files/M117-bl_1.fq.gz
>    https://zenodo.org/record/5119008/files/M117-bl_2.fq.gz
>    https://zenodo.org/record/5119008/files/M117-ch_1.fq.gz
>    https://zenodo.org/record/5119008/files/M117-ch_2.fq.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Check that the datatype is set to `fastqsanger.gz` (Galaxy usually automatically assigns the right datatype, but check anyway).
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="datatypes" %}
>
> 4. Create a paired collection from your data (relevant part beging at 0:57 and and the case of this tutorial we have 4 datasets): 
>
>    {% include _includes/youtube.html id="uZUt9XIHUQo" title="Create a paired collection tutorial" start=57 %}
>
{: .hands_on}

## Assess quality and clean the reads with `fastp`

[`fastp`](https://github.com/OpenGene/fastp) is fast utility for computing read qualities and performing initial clean up of the data such as [adapter removal](https://github.com/OpenGene/fastp?tab=readme-ov-file#adapters).

> <hands-on-title> Running `fastp` </hands-on-title>
>
> 1. {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/0.23.2+galaxy0) %} with the following parameters:
>    - *"Single-end or paired reads"*: `Paired Collection`
>        - {% icon param-collection %} *"Select paired collection(s)"*: `output` (Input dataset collection)
>    - In *"Output Options"*:
>        - *"Output JSON report"*: `Yes`
>
{: .hands_on}

## Sub-step with `MultiQC`[]

[`multiqc`](https://multiqc.info/) is a report generation tool. It can process outputs on multiple tools including `fastp`:

> <hands-on-title> Generating quality report </hands-on-title>
>
> 1. {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.11+galaxy1) %} with the following parameters:
>    - In *"Results"*:
>        - {% icon param-repeat %} *"Insert Results"*
>            - *"Which tool was used generate logs?"*: `fastp`
>                - {% icon param-file %} *"Output of fastp"*: `report_json` (output of **fastp** {% icon tool %})
>
{: .hands_on}

# Interpreting the results

The comparison between Element and Illumina QC plot is shown below:

![QC scores for Element and Illumina](../../images/qc_elem_ill.svg "Element (top) versus Illumina (bottom). Element base qualities are consistently over 40 for most bases within each read. Blue and black lines represent samples <tt>s1</tt> and <tt>s2</tt> in case of Element and <tt>bl</tt> and <tt>ch</tt> in case of Illumina.")

Both datasets are exceptionally good. However, the elemnt data has even higher quality.

# Conclusion

Now you can easily evaluate quality of your own data in Galaxy!


topic: introduction
tutorial: dummy
title: Upload data to Galaxy
difficulty: introductory
time_estimation: 40m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 97eff559278b0171dd4784621704c3aa
last_modified: 2026-02-26T21:12:17.029968
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/dummy/tutorial.html

content:
There are several ways to get data into Galaxy:
- Uploading a file by pasting in a URL
- Uploading a file from the computer
- Uploading using a FTP server
- Import an entire history
- Import data from a shared library
- ...

> <agenda-title></agenda-title>
> 1. TOC
> {:toc}
>
{: .agenda}

# Upload a file via an URL

> <hands-on-title>Data upload from URL</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Copy the following URL
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    ````
>
> 3. Import it in the history
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}

# Import from a shared library

> <hands-on-title>Data upload from URL</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Copy the following URL
>
>    ```
>    https://zenodo.org/record/582600/files/mutant_R1.fastq
>    ````
>
> 3. Import it in the history
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
{: .hands_on}


topic: introduction
tutorial: data-manipulation-olympics
title: Data Manipulation Olympics
difficulty: introductory
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: ["cyoa"]
content_hash: b357f52b15e5beb66ff6429e8e18faca
last_modified: 2026-02-26T21:12:17.029292
zenodo_link: https://zenodo.org/record/6803028
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/introduction/tutorials/data-manipulation-olympics/tutorial.html

content:
Scientific analyses often consist of a number of tools that run one after the other, in order to go from the raw data to scientific insight. Between these specialized tools, simple data manipulation steps are often needed as a kind of "glue" between tools. For example, the output of tool A may produce a file that contains all the information needed as input for tool B, but tool B expects the columns in a different order. Or in genomic data analysis, some tools expect chromosome X to be listed as `chrX`, while others simply expect `X`. In these situations, extra data manipulation steps are needed to prepare files for input to analysis tools.

Galaxy has a large collection of tools to perform such basic data manipulation tasks, and becoming familiar with these operations will allow to perform your analysis more easily in Galaxy (and outside).


<!--
Note to contributors: feel free to add sections here to include additional data manipulation options.
Make sure each section is independent of each other, i.e. each section should start with the olympics.tsv file.
Also make sure to include many exercises (with answers) for your section!
-->


<!-- set up some variables to easily update tool versions throughout tutorial, since most tools are used many times %} -->
{% assign version_cat="cat1" %}
{% assign version_compute="toolshed.g2.bx.psu.edu/repos/devteam/column_maker/Add_a_column1/2.0" %}
{% assign version_count="Count1" %}
{% assign version_cut_columns="Cut1" %}
{% assign version_cut_advanced="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cut_tool/1.1.0" %}
{% assign version_datamash="toolshed.g2.bx.psu.edu/repos/iuc/datamash_ops/datamash_ops/1.1.0+galaxy2" %}
{% assign version_filter="Filter1" %}
{% assign version_join="join1" %}

{% assign version_remove_beginning="Remove beginning1" %}
{% assign version_remove_columns_by_header="toolshed.g2.bx.psu.edu/repos/iuc/column_remove_by_header/column_remove_by_header/0.0.1" %}
{% assign version_replace_text_column="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1" %}
{% assign version_replace_text_line="toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regex1/1.0.1" %}
{% assign version_paste="Paste1" %}

{% assign version_select_first="Show beginning1" %}
{% assign version_sort="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sort_header_tool/1.1.1" %}
{% assign version_split="toolshed.g2.bx.psu.edu/repos/bgruening/split_file_on_column/tp_split_on_column/0.6" %}
{% assign version_unique="toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_sorted_uniq/1.1.0" %}
{% assign version_wc="wc_gnu" %}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Cheatsheet

Here is an overview table of the different data manipulations in this tutorial, with links to the tools in Galaxy.

If you've opened this tutorial via the {% icon level %} icon in Galaxy (top menu bar), you can click on the tool names in the last column to quickly open them in Galaxy and start using them on your own!

<!-- GTN:IGNORE:006 -->

| Operation              | Description                        | Galaxy Tool    |
|------------------------|------------------------------------|----------------|
| Convert format         | Change the file format             | {% icon galaxy-pencil%} Edit attributes |
| Word count             | Count the number of lines, words and characters in a file | {% tool [Line/Word/Character count]({{version_wc}}) %} |
| Sort on a column       | Change the order of the rows based on values in one or more columns | {% tool [Sort]({{version_sort}}) %} |
| Filter                 | Remove rows based on values in one or more columns | {% tool [Filter]({{version_filter}}) %}|
| Counting               | Count occurrences of values in a column   | {% tool [Count]({{version_count}}) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Group on a column      | And perform simple operations (count, mean, min, max etc) | {% tool [Group](Grouping1) %} , {% tool [Datamash]({{version_datamash}}) %} |
| Compute on rows        | to derive new column values from existing ones | {% tool [Compute]({{version_compute}}) %} |
| Find and Replace       | in a specific column               | {% tool [Column Regex Find and Replace]({{version_replace_text_column}}) %}|
| Find and Replace       | on every line                      | {% tool [Regex Find and Replace]({{version_replace_text_line}}) %}|
| Join two Datasets      | side by side on a specified field  | {% tool [Join two Datasets]({{version_join}}) %} |
| Concatenate datasets   | one after the other                | {% tool [Concatenate datasets]({{version_cat}}) %} |
| Remove Beginning       | Good for removing header lines     | {% tool [Remove beginning of a file]({{version_remove_beginning}}) %} |
| Select First lines     | Good for finding top 10s or saving header lines | {% tool [Select first lines]({{version_select_first}}) %} |
| Cut Columns            | By header name                     | {% tool [Remove columns by heading]({{version_remove_columns_by_header}}) %}|
| Cut Columns            | By column number                   | {% tool [Cut columns from a table]({{version_cut_columns}}) %}, {% tool [Advanced Cut columns from a table]({{version_cut_advanced}}) %}|
| Paste                  | Two files side by side             | {% tool [Paste]({{version_paste}}) %} |
| Split file             | Based on values of a column        | {% tool [Split]({{version_split}}) %} |
| Unique                 | Remove duplicate rows              | {% tool [Unique]({{version_unique}}) %} |

**TIP: Adding tools to your Favourites:** If you find yourself frequently using the same tool often but struggle to find it back in the long list of tools, you can **star** your favourite tools in Galaxy!

{% snippet faqs/galaxy/tools_favorite.md %}


In this tutorial, these tools are explained in more detail, and we provide some exercises for you to practice.


# Background

In this tutorial, we will use as our dataset a table with results from the Olympics, from the games in Athens in 1896 until Tokyo in 2020. The objective is to familiarize you with a large number of the most important data manipulation tools in Galaxy. Much like the Olympics, there are many different disciplines (types of operations), and for each operation there are often multiple techniques (tools) available to athletes (data analysts, you) that are great for achieving the goal.


![image of olympic rings, logo and two athletes around the words "Data Analysis Olympics"](./images/cover.jpg)


We will show you many of these commonly needed data manipulation operations, and some examples of how to perform them in Galaxy. We also provide many exercises so that you can train your skills and become a data manipulation Olympian!


# Upload Data

Before we can do any manipulation, we will need some data. Let's upload our table with Olympics results now.

> <hands-on-title>Get data</hands-on-title>
>
> 1. {% tool [Import](upload1) %} the file `olympics.tsv` via link
>
>    ```
>    {{page.zenodo_link}}/files/olympics.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. **Expand** on the item in your history to see some metadata and a short preview of the contents.
>
> 3. **View** {% icon galaxy-eye %} the dataset by clicking on the eye icon.
>
>    > <question-title></question-title>
>    >
>    > 1. What is the format of the file?
>    > 2. What does each row represent?
>    > 3. How many lines are in the file? (Hint: {% tool [Line/Word/Character count](wc_gnu) %})
>    > 4. How many columns?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. When you expand the `olympics.tsv` dataset in your history (see also screenshot below), you will see `format: tabular`, this is another term for a tab-separated (`tsv`) file.
>    > > 2. Each row represents an athlete's participation in an event. If an athlete competes in multiple events, there is a line for each event.
>    > > 3. 234,523. Look at the expanded view in your history, this tells us there are ~250,000 rows in the dataset. We can get the exact number using the  {% tool [Line/Word/Character count](wc_gnu) %} tool.
>    > > 4. There are 17 columns in this file. There are multiple ways to find this answer:
>    > >    - Count the columns manually (only doable for small files)
>    > >    - In the expanded view, scroll sideways on the dataset preview, at the top the columns are numbered
>    > >    - Click on the {% icon galaxy-info %} i icon on the dataset; here you will find more detailed information about the file and the job that created it.
>    > >      At the bottom is also a preview (peek) of the dataset and numbered columns
>    > >
>    > > ![a screenshot of the expanded view of the dataset in the history, it shows the datatype, number of lines in the file, and a preview
>    > > of the dataset with numbered columns](./images/columns-number.png)
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## About this dataset

The data was [obtained](https://github.com/UOSCS/Olympic_Athletes) from [Olympedia](https://www.olympedia.org/). The file `olympics.tsv` contains
234,522 rows and 17 columns. Each row corresponds to an individual athlete competing in an individual Olympic event. The columns are:

- **athlete_id** - Unique number for each athlete
- **name** - Athlete's name
- **sex** - M or F
- **birth_year** - 4-digit number
- **birth_day** - e.g. 24 July
- **birth_place** - town and/or country
- **height** - In centimeters (or `NA` if data not known)
- **weight** - In kilograms (or `NA` if data not known)
- **team** - Team name
- **noc** - National Olympic Committee 3-letter code
- **games** - Year and season
- **year** - Integer
- **season** - Summer or Winter
- **city** - Host city
- **sport** - Sport
- **event** - Event
- **medal** - Gold, Silver, Bronze (or `NA` if no medal was won)

We will use this dataset to practice our data manipulation skills in Galaxy.


# Choose your adventure!

This tutorial is structured a bit differently than most. **You do not have to do the steps in the order they are presented below.** Every section in this tutorial uses the dataset you just uploaded (the `olympics.tsv` file) as input, so you can jump to any section in this tutorial right now if you have a particular data manipulation operation in mind you want to learn more about.




# File Format Conversion

The file we uploaded is a `.tsv` file. This stands for *tab-separated values*. This means that this is a file containing rows and columns, where a TAB character is used to signify a column ends and a new one begins. Galaxy is great at understanding tab-separated files, and most of the data manipulation tools are designed to work with such files.

A similar format you may come across a lot in data science, is the `.csv` file, or *comma-separated values* file. This is the same as `.tsv`, but uses comma (`,`) characters to indicate new columns, instead of TAB (`\t`) characters.

Galaxy can convert these two formats into each other.



> <hands-on-title>Convert Tabular to CSV</hands-on-title>
>
> 1. Convert the `olympics.tsv` file into a CSV file
>
>    {% snippet faqs/galaxy/datasets_convert_datatype.md conversion="csv (using 'Convert tabular to CSV')" %}
>
> 2. {% icon galaxy-eye %} **View the converted dataset**
>
>    > <question-title></question-title>
>    >
>    > 1. What do you notice?
>    > 2. Why are some values in quotes?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The display in the center panel should be indistinguishable from before the conversion; however, if you look at the "quick view" of the data in the history step, you will see the commas separating columns.
>    > > 2. If the data in a column contains a comma (e.g. in this file we have events such as `swimming 5,000 meters`), we put the value in quotes to signify that that comma is part of the data, not a column delimiter.
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


## Converting vs Changing the datatype

The **file format conversion** step changed the dataset itself (by changing TABs to commas), and therefore created another dataset in your history. It is also possible to **change the datatype**; this does not change the underlying file/data, but just tells Galaxy what type of file it is. Galaxy uses this information in a variety of ways, such as:
  - How to display the data when you click the {% icon galaxy-eye %} **View** button
  - Which visualization options to offer
  - Filtering the list of possible inputs for tools (to only allow those of the correct datatype and protect you from mistakes)
  - Which file format conversions to make available
  - etc..


When you upload a file to Galaxy, by default it will attempt to auto-detect the file format. This is very often correct, but some times you may need to manually set the datatype to the correct value.


> <hands-on-title>Change datatype from Tabular to TXT</hands-on-title>
>
> 1. **Change** the datatype of the `olympics.tsv` file to `txt`.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="txt" %}
>
> 2. {% icon galaxy-eye %} **View the dataset** again
>
>    > <question-title></question-title>
>    >
>    > 1. What do you see?
>    > 2. Why didn't this step create a new item in your history?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Since Galaxy now no longer know that this is a tabular file with rows and columns,
>    > > it displays the data as-is, no longer arranging the view into neat columns.
>    > >
>    > > 2. The file itself did not change, only the metadata. We simply told Galaxy to treat this file as a plain text file, instead of a tabular file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Change** the file back from `txt` to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}



# Sorting

We have a lot of data in this file, but it is ordered by the athlete ID number, which is a somewhat arbitrary and meaningless number. But we can sort the rows in this file to something more convenient, for example alphabetically by name of the athlete, or chronologically by year of the Olympics.

> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the year?
>    > 2. Do we want ascending or descending order if we want the oldest games at the top?
>    > 3. What should we do with the very first row (the one with the header names?)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 12
>    > > 2. The file should be sorted in ascending (increasing) order
>    > > 3. The header line should always stay on top, so we want to ignore it when we sort the file.
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} tool
>    - Read the help text at the bottom of the tool form
>    - Which settings do you think we need to use?
>
> 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
>    - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
>    - {% icon param-text %} *"Number of header lines"*: `1``
>    - Column selections
>      - {% icon param-select %} *"on column"*: `Column 12`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
>
> 4. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top of the file now?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. J. Defert. Who competed in a Tennis event 1896 Summer Olympics in Athens.
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful (e.g. `Olympics in chronological order`)
>    - It can be easy to lose track of what steps you have performed and what each output contains, especially if you are doing many
>      data manipulation steps. Do future-you a favour and give datasets good names!
>
{: .hands_on}

This is great, but maybe it would make more sense to sort alphabetically by athlete name *within each year*.

## Sort on multiple columns at once

So we want to sort twice, first by year, and then within each year, we sort again alphabetically by name. The sort tool can do this!


> <hands-on-title>Sort table based on a column</hands-on-title>
>
> 1. We will sort the file in chronological order based on the year of the Olympic games
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains athlete names?
>    > 2. Do we want ascending or descending order if we want to sort alphabetically with A first?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 2
>    > > 2. The file should be sorted in ascending (increasing) order
>    > >
>    > {: .solution}
>    {: .question}
>
>
> 2. {% icon galaxy-refresh %} **Rerun** the sort tool with the following parameters:
>    - All parameter from the first step should already be set for you, and should remain the same
>    - {% icon param-repeat %} Insert Column selections
>      - {% icon param-select %} *"on column"*: `Column 2`
>      - {% icon param-select %} *"in"*: `Ascending order`
>      - {% icon param-select %} *"Flavor"*: `Alphabetical sort`
>
> 3. {% icon galaxy-eye %} **View** the sorted file.
>
>    > <question-title></question-title>
>    >
>    > Which athlete is listed at the top now? Which discipline (sport) did they compete in?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. A. Grigoriadis. He competed in the 500 meters freestyle swimming event.
>    > >
>    > {: .solution}
>    {: .question}
>
> 4. **Rename** {% icon galaxy-pencil %} the resulting file to somthing meaningful
>
{: .hands_on}

## Exercises

Ok, time to train! Let's see if you can use the sort tool to answer the following questions:

> <question-title noprefix>Exercise: Reverse the sort</question-title>
>
> Which athlete comes *last by alphabet*, in the *most recent* Olympics?
>
> > <solution-title></solution-title>
> >
> > `Žolt Peto` who competed in table tennis at the 2020 Summer Olympics in Tokyo.
> > <br>
> > We do this by repeating the previous sort (on year and then name), but changing the order to *descending* for both, to get the answer to the top of the file.
> >
> {: .solution}
{: .question}


> <question-title noprefix>Exercise: sort by height</question-title>
>
> 1. What is the height of the tallest competing athlete? Which athlete(s) are of this height?
> 2. What is the shortest?
> 3. Who was the tallest athlete from the most recent Olympics? How tall were they?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Height is listed in column 7. Since this is a simple number (no scientific notation), we can use *fast numerical sort*,
> >    and because we want the tallest on top, we will need to sort in *descending* (decreasing) order.
> > 2. Rerun the tool for step 1, but change the order to *ascending*
> > 3. First sort by year (descending), then by height (descending)
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. Adam Sandurski from Poland is the tallest athlete in the file, at 214 cm tall.
> >  2. Here we don't get a clear answer. That is because not all athletes have height data available, and blank fields (`NA`) are being sorted to the top. We can filter out rows with missing data to get our answer (see [Filter](#filtering) section to learn how to do this). For now we cannot answer this question with just the sort tool. Some times multiple tools are required to perform such tasks. The [exercise section at the end of this tutorial](#exercises-putting-it-all-together) has many exercises that require a combination of different tools.
> >  3. Gennaro Di Mauro, 210 cm. (2020 Summer Olympics in Tokyo)
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> > 1. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Descending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 2. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7`
> >     - {% icon param-select %} *"in"*: `Ascending order`
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> > 3. {% tool [Sort - data in ascending or descending order]({{version_sort}}) %} with the following parameters:
> >   - {% icon param-file %} *"Sort Query"*: `olympics.tsv`
> >   - {% icon param-text %} *"Number of header lines"*: `1``
> >   - Column selections
> >     - {% icon param-select %} *"on column"*: `Column 12` (year column)
> >     - {% icon param-select %} *"in"*: `Descending order` (most recent on top)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >   - {% icon param-repeat %} Insert Column selections
> >     - {% icon param-select %} *"on column"*: `Column 7` (height column)
> >     - {% icon param-select %} *"in"*: `Descending order` (tallest first)
> >     - {% icon param-select %} *"Flavor"*: `Fast numeric sort`
> >
> {: .solution}
>
{: .question}


# Filtering

This file contains a lot of data, but we may only be interested in a subset of this data. For example, we may only want to look at one particular Olympics, or one particular sport. In such cases we can filter the dataset. This will create a new dataset, removing any rows that are not of interest to us (i.e. that don't meet the criteria we provide).

## Filtering based on column contents

> <hands-on-title>Filter table based on a column</hands-on-title>
>
> We will filter the file to show only winter Olympics
>
> 1. Look at the `olympics.tsv` file and answer the following questions
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains this information?
>    > 2. Which values can this column have? (make sure to notice capitalisation, 'Winter' is not the same as 'winter' to these tools)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Column 13, the column with the *season* header
>    > > 2. The values can be `Summer` or `Winter`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. Open the tool {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %}
>    - read the help text at the bottom of the tool carefully
>    - get familiar with how to indicate columns and write expressions for filtering
>
>    > <question-title></question-title>
>    >
>    > 1. How would you write the expressions for the following conditions:
>    >    1. column 6 must be 'Yes'
>    >    2. column 13 must be smaller than 75
>    >    3. column 7 cannot be 'NA'
>    >    4. column 2 cannot be empty
>    >
>    > 2. It is also possible to combine multiple conditions, using `and`, `or`, `not` and parentheses
>    >    How would you write expressions for the following filtering conditions:
>    >    1. column 5 is larger than 2 or smaller than -2
>    >    2. column 5 is larger than 2 and smaller than 10
>    >    3. the sum of columns 4 and 6 is greater than or equal to 25,000
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. The answers are:
>    > >    1. `c6=='Yes'`
>    > >    2. `c13<75`
>    > >    3. `c7!='NA'`
>    > >    4. `c2!=''`
>    > >
>    > > 2. The answers are:
>    > >    1. `c5>2 or c5<-2`
>    > >    2. `c5>2 and c5<10`
>    > >    3. `c4+c6 >= 25000`
>    > >
>    > {: .solution}
>    {: .question}
>
>    Ok, great, now that you've got the hang of writing expressions for this tool, let's create a file with only Winter Olympics:
>
> 3. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c13=='Winter'`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 4. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > How many lines are in this file? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 44,681 (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 5. **Repeat** the step for the Summer Olympics
>
>    > <question-title></question-title>
>    >
>    > 1. How many lines do you expect in this file?
>    > 2. How many lines are in this file? Were you right?
>    >
>    > > <solution-title noprefix>Hints</solution-title>
>    > >
>    > > 1. Use the {% tool [Line/Word/Character count]({{version_wc}}) %} to find the number of lines in the `olympics.tsv` file and subtract the number of rows in the Winter Olympics file
>    > > 2. Be careful to consider whether these counts include the header line of the file or not
>    > >
>    > {: .solution}
>    >
>    > > <solution-title noprefix>Answers</solution-title>
>    > >
>    > > 1. The original file has 234,523 lines, and the Winter Olympics had 44,681 lines. So we would expect 234,523 - 44,681 = 189,842 rows of data. Since we have subtracted the header line in this equation as well, we expect the Summer Olympics file to have 1 more line that this, so 189,843 total lines.
>    > > 2. 189,843. If you were off by one or two lines, it may have been that you counted the header lines double.
>    > > <br>
>    > > It is always useful to take a moment to think about the expected outcome; this makes it easier to spot mistakes and will save you time in the long run.
>    > >
>    > {: .solution}
>    {: .question}
>
> 6. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive (e.g. `Filter: Winter Olympics only` etc)
>    - This is primarily to get you into the habit. If this is a file you might want to view or use again later, it will be hard to find back again unless you name it well.
>
{: .hands_on}

## Advanced Filtering with Python Syntax

In addition to basic filtering, the Galaxy filtering tool allows for more advanced filtering using Python syntax. This can be particularly useful when you need to apply more complex conditions or manipulate the data in specific ways.

> <comment-title> Understanding Python Syntax for Filtering </comment-title>
>
>    Before we dive into specific examples, let's briefly discuss some basic Python concepts that will help you understand how to write these filtering expressions:
>
> - **Methods**: In Python, methods are functions that are associated with an object. For example, in the expression `c2.startswith('Liu')`, `startswith` is a method that checks if the value in column 2 (`c2`) starts with the substring 'Liu'.
>
> - **Boolean Expressions**: These are expressions that evaluate to either `True` or `False`. For example, `c2.startswith('Liu')` will return `True` if the value in column 2 starts with 'Liu', and `False` otherwise.
>
> - **Comparison Operators**: These include `==` (equal to), `!=` (not equal to), `<` (less than), `>` (greater than), `<=` (less than or equal to), and `>=` (greater than or equal to). These operators are used to compare values and are commonly used in filtering.
>
> - **String Methods**: Strings in Python have several built-in methods that can be used for filtering. Some commonly used ones include:
>   - `startswith(substring)`: Checks if a string starts with a specified substring.
>   - `endswith(substring)`: Checks if a string ends with a specified substring.
>   - `find(substring)`: Returns the lowest index of the substring if it is found in the string; otherwise, it returns `-1`.
>   - `count(substring)`: Returns the number of non-overlapping occurrences of the substring in the string.
>
{: .comment}

These concepts will be used in the filtering expressions we write in Galaxy. Let's now explore some specific scenarios where you might use these expressions.


### Filtering Based on Substring Occurrence

We want to filter rows where a particular column contains a specific substring using the `cX.find()` method, where `X` is the column number.

For example, while you can easily filter Summer or Winter Olympics using `c13=="Summer"` or `c13=="Winter"` if the data is stored in a dedicated column, sometimes the data may not be that straightforward. The relevant information might be embedded in another column, like in `c11` with entries such as "1992 Summer Olympics".

In such cases, you can use substring filtering:

> <question-title></question-title>
>
> 1. How would you filter the rows where the 11th column contains "Summer"?
> 2. How would you filter the rows where the 11th column does **not** contain "Summer"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c11.find('Summer') != -1`
> > 2. `c11.find('Summer') == -1`
> >
> {: .solution}
{: .question}

> <hands-on-title>Filtering Based on Substring Occurrence</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file for rows where column 11 contains "Summer":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c11.find('Summer') != -1`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string "Summer" in column 11? (Hint: use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `189843`(this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on }


### Filtering Rows that Start or End with a Specific Value

Filter rows based on whether a column starts or ends with a particular value using `cX.startswith()` or `cX.endswith()`.

For example, if we want to:
 - Find all athletes whose names start with "Liu".
 - Find all teams whose names end with "China".


> <question-title></question-title>
>
> 1. How would you filter rows where the 2nd column starts with "Liu"?
> 2. How would you filter rows where the 9th column ends with "China"?
>
> > <solution-title>Answers</solution-title>
> >
> > 1. `c2.startswith('Liu')`
> > 2. `c9.endswith('China')`
> >
> {: .solution}
{: .question}


> <hands-on-title>Filtering Rows that Start or End with a Specific Value</hands-on-title>
>
> Ok, great, now that you've got the hang of writing expressions for this tool, let's filter the file to find all athletes whose names start with "Liu":
>
> 1. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>    - {% icon param-file %} *"Filter"*: `olympics.tsv`
>    - {% icon param-text %} *"With the following condition"*: `c2.startswith('Liu')`
>    - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 2. {% icon galaxy-eye %} **View** the filtered file.
>
>    > <question-title></question-title>
>    >
>    > 1. How many rows contained the string start with "Liu" in column 2? (Hint: expand the dataset in your history or use {% tool [Line/Word/Character count]({{version_wc}}) %} )
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `324` (this is including the header line)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} both outputs to something descriptive.
>
{: .hands-on}


### Filtering Based on the Count of Substring Occurrences

Sometimes, you may need to filter rows based on **how many times** a certain substring appears in a column. This can be done using the `cX.count()` method.

For example, if we want to find athletes with three-part names by counting spaces in the name column.

> <hands-on-title>Filtering Based on the Count of Substring Occurrences</hands-on-title>
>
> We want to filter the dataset to find athletes with three-part names.
>
> 1. Have a look at the `olympics.tsv` file
>
>    > <question-title></question-title>
>    >
>    > 1. Which column contains the names of athletes?
>    > 2. For athletes with 3-part names, how many spaces do we expect in that column?
>    > 3. What expression would you use to find these cases?
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. Column 2 contains the names
>    > > 2. For 3-part names we expect 2 space characters
>    > > 3. `c2.count(' ') == 2`
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [**Filter** data on any column using simple expressions]({{version_filter}}) %} with the following parameters:
>     - {% icon param-file %} *"Filter"*: `olympics.tsv`
>     - {% icon param-text %} *"With the following condition"*: `c2.count(' ') == 2`
>     - {% icon param-text %} *"Number of header lines to skip"*: `1`
>
> 3. {% icon galaxy-eye %} **View** the filtered file.
>
>     > <question-title></question-title>
>     >
>     > 1. How many rows contain three-part names (i.e., two spaces) in column 2?
>     >
>     > > <solution-title>Answers</solution-title>
>     > >
>     > > 1. `11259` (this is including the header line)
>     > >
>     > {: .solution}
>     {: .question}
>
> 4. **Exercises.** Now let's take it a step further and filter athletes with even longer names.
>
>    > <question-title></question-title>
>    >
>    > 1. How would you filter rows where the 2nd column contains **three spaces** (i.e., four-part names)?
>    > 2. How many medals were won by athletes with five-part names **or more**?
>    > 3. How many parts does the longest name have? Give an example of such a name.
>    > 4. Bonus: **How many** athletes have 6-part names won a medal?
>    >
>    > > <solution-title>Hints</solution-title>
>    > >
>    > > 1. None ;)
>    > > 2. You can also use greater than or less than symbols in your expression. You can filter on two lines using  logical operators like `and`.
>    > > 3. Keep increasing the count until no results are returned
>    > > 4. The number of lines returned indicates the number of medals won by long-named athletes, to
>    > >    find out how many different athletes have such a name, we can use the {% tool [Count - occurrences of each record](Count1) %}
>    > >    tool on Column 2.
>    > {: .solution}
>    >
>    > > <solution-title>Answers</solution-title>
>    > >
>    > > 1. `c2.count(' ') == 3`
>    > > 2. `c17 != 'NA' and c2.count(' ') >= 4` returns 27 lines, meaning 26 medals won accounting for the header line excluding the header line. Appart from the name you need to additionally filter out athletes that did not won a medal.
>    > > 3. `c2.count(' ') == 5` is the maximum value to still return results.
>    > >    Five spaces means a 6-part name. For example Patricia Galvin de la Tour d'Auvergne
>    > > 4. `c17 != 'NA' and c2.count(' ') == 5` returns 5 records. So 5 medals were won by these long-named athletes.
>    > >     Using the {% tool [Count - occurrences of each record](Count1) %} to count occurrences in column 2, we
>    > >     discover that there are **3 different athletes with 6-part names**.
>    > >
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Exercises

Ok, time to train! let's see if you can use the {% tool [Filter]({{version_filter}}) %} tool to answer the following questions:


> <question-title noprefix>Exercise: Medal winners</question-title>
>
> 1. How many gold medals were handed out?
> 2. How many total medals?
> 3. How many medals were handed out during the 2018 Olympics?
> 4. How many medals were won by individuals with a height between 170 and 180 cm?
> 5. How many gold medals were won by individuals shorter than 160cm or taller than 190?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > - Column 17 contains information about medals
> > - The possible values are `Gold`, `Silver`, `Bronze`, and `` (empty).
> > - Expand the output or use the tool {% tool [Line/Word/Character count]({{version_wc}}) %} to see the number of lines in the file
> > - Don't forget that the output (and line count) may include the header line
> > - Do not use quotes on number columns (e.g. year)
> > - You may need parentheses for complex conditions
> >
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. 8,110   (Expression: `c17=='Gold'`)
> >  2. 24,633  (Expression: `c17=='Gold' or c17=='Silver' or c17=='Bronze'`, or `c17!='NA'`)
> >  3. 131     (Expression: `c17=='Gold' and c12==2018` (note: do not use quotes around `2018`, as it is a numerical value))
> >  4. 8,086   (Expression: `c17!='NA' and c7>=170 and c7<=180`)
> >  5. 812     (Expression: `c17=='Gold' and (c7<160 or c7>190)` (note: parentheses are important here))
> >
> > Note: these numbers are found by determining the number of lines in the file after each filtering step, and subtracting 1 for the header line.
> >
> {: .solution}
{: .question}


# Counting

A common operation we might want to perform on tables of data, is simple counting. How many times does a certain value appear? For our dataset for instance, we might want to know how many countries participated in each Olympics, how many women, etc; any column that has categorical data that we can count. The tool {% tool [**Count** occurrences of each record]({{version_count}}) %} does exactly this.


> <hands-on-title>Count occurrences of values in columns</hands-on-title>
>
> Let's start by simply counting how many different Olympic Games we have in our dataset, and how many times it appears (so how many participations there were each year)
>
> 1. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11`
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How many different Olympic games are in our file?
>    > 2. Which Olympic games had the most participations? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 52; there are 53 lines in the resulting file, with one line containing the value of the column header (`games`).
>    > >
>    > >    The resulting file looks something like:
>    > >
>    > >     ```
>    > >     615	1896 Summer Olympics
>    > >     2503	1900 Summer Olympics
>    > >     2643	1904 Summer Olympics
>    > >     3213	1908 Summer Olympics
>    > >     4610	1912 Summer Olympics
>    > >     3448	1920 Summer Olympics
>    > >     5242	1924 Summer Olympics
>    > >     358	1924 Winter Olympics
>    > >     4493	1928 Summer Olympics
>    > >     ...
>    > >     ```
>    > >
>    > > 2. 1996 Summer Olympics. (10501 participations)
>    > >
>    > {: .solution}
>    {: .question}
>
>    You may have noticed that we could have selected multiple columns in this tool. This lets us count on combinations of columns.
>    Let's try counting the number of men and women in each olympic games.
>
> 3. {% tool [**Count** occurrences of each record]({{version_count}})%} with the following parameters
>    - {% icon param-file %} *"from dataset"*: `olympics.tsv`
>    - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 11, Column 3`
>
> 4. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > You see the resulting file has a line for every combination of the two columns (games and sex), providing the count for each.
>    >
>    > 1. How many women were in the first Olympic games?
>    >
>    > 2. Which Olympic games had the most women participants? (Tip: set the parameter *"How should the results be sorted?"* to `most common values first`)
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. 2 women participated in the 1896 Olympics. (note that we cannot be sure if this is two different women, or 1 woman participating twice).
>    > >    The file looks something like this:
>    > >    ```
>    > >    2	F	1896 Summer Olympics
>    > >    43	F	1900 Summer Olympics
>    > >    17	F	1904 Summer Olympics
>    > >    55	F	1908 Summer Olympics
>    > >    97	F	1912 Summer Olympics
>    > >    132	F	1920 Summer Olympics
>    > >    269	F	1924 Summer Olympics
>    > >    ```
>    > >
>    > > 2. 2020 Summer Olympics (4652)
>    > >
>    > {: .solution}
>    {: .question}
>
>
{: .hands_on}


Let's say we wanted to know how many different sports there were in each Olympics. If we used the counting tool above, we would get a results file for each combination of sport and olympics, with the number of lines (participations) of each. But we don't really care about the number of lines that have this combination, just the total number of unique sports in each games.

To answer these types of questions we can use a slightly more advanced tool, called {% tool [Datamash]({{version_datamash}}) %}. This tool can do a lot more than counting, but here we will show you how to use it to count in this way.


> <hands-on-title>Counting number of different sports per Olympic Games</hands-on-title>
>
> We will now determine how many different sport there were in each of the different Olympics
>
> 1. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
>    - {% icon param-toggle %} *"Input file has a header line"*: `Yes`
>    - {% icon param-toggle %} *"Sort Input"*: `Yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Count Unique values`
>      - {% icon param-select %} *"On Column"*: `Column: 15` (the `sport` column)
>
> 2. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. Why did we tell the tool to sort the file?
>    > 2. How many sport were in the first Olympics? How many in the latest?
>    > 3. Which Olympics had the most different sports?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. This is for technical reasons; the tool assumes that your file is sorted (on the column that you supplied to the "Group by fields" parameter).
>    > >    This assumption can greatly speed up the calculation, but will give incorrect results if the file was not actually sorted.
>    > >    If you know for sure that your data is sorted in the right way, you can skip this step. If you are unsure, enable this parameter to be safe, and the
>    > >    tool will perform the sorting step before its calculation.
>    > > 2. 10 and 38.
>    > > 3. The 2020 Summer Olympics had the most different sports (38)
>    > >
>    > {: .solution}
>    {: .question}
>
> 3. **Rename** {% icon galaxy-pencil %} the output to something descriptive
>
{: .hands_on}


Note: Datamash can do a lot more than counting,  and we will showcase some of these other operatins in the [Grouping](#grouping) section.

## Exercises

Ok, let's practice!

> <question-title noprefix>Exercise: Number of participations per country</question-title>
>
> 1. Which country has had the most participations in the Olympics?
> 2. How many countries participated in the first Olympics? How many in the last?
>
> > <solution-title noprefix>Hints</solution-title>
> >
> > 1. Since we are counting participations (rows), we can use the simple {% tool [Count]({{version_count}})%} tool
> > 2. Since we are counting a bit more complex question, we need the {% tool [Datamash]({{version_datamash}}) %} tool
> >
> {: .solution}
>
> > <solution-title noprefix>Answers</solution-title>
> >
> >  1. The United States with 17,286 participations
> >  2. 15 and 250.
> >
> {: .solution}
>
> > <solution-title noprefix>Full Solutions</solution-title>
> >
> >  1. {% tool [Count]({{version_count}}) %} with the following parameters:
> >     - {% icon param-file %} *"from dataset"*: `olympics.tsv`
> >     - {% icon param-select %} *"Count occurrences of values in column(s)"*: `Column 9` (the `team` column)
> >     - {% icon param-select %} *"How should the results be sorted?"*: `With the most common values first`
> >
> >     This gives an output like:
> >     ```
> >     17286	United States
> >     11700	France
> >     10230	Great Britain
> >     8898	Italy
> >     7988	Canada
> >     ```
> >
> >  2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
> >   - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
> >   - {% icon param-text %} *"Group by fields"*: `11` (the `games` column)
> >   - {% icon param-toggle %} *"Input file has a header line"*: `yes`
> >   - {% icon param-toggle %} *"Sort Input"*: `yes`
> >   - *"Operation to perform on each group"*:
> >       - {% icon param-select %} *"Type"*: `Count Unique values`
> >       - {% icon param-select %} *"On Column"*: `Column: 9` (the `team` column)
> >
> >   This gives an output like:
> >   ```
> >   1896 Summer Olympics	15
> >   1900 Summer Olympics	29
> >   1904 Summer Olympics	13
> >   ...
> >   2016 Summer Olympics	280
> >   2018 Winter Olympics	108
> >   2020 Summer Olympics	250
> >   ```
> >
> {: .solution}
{: .question}








# Grouping

Often we may want to group rows based on a value in a column, and perform some operation on the resulting rows. For example we would like to group the olympics data by one value (e.g. year, country, sport), and determine some value for each group (e.g. number of medals won, average age of athletes, etc).

In the [counting](#counting) section of this tutorial we show how to get answers that require a count (e.g. number of medals won), but sometimes we want to do something more complex, like calculating the average height of athletes in a group, say per country or per sport. This section will show some example of these types of questions.

We can use the {% tool [Datamash]({{version_datamash}}) %} tool for this purpose.

> <hands-on-title>Tallest athlete per sport</hands-on-title>
>
> We would like to answer the following question: *How tall was the tallest athlete of each sport?*
>
> 1. Open the {% tool [Datamash]({{version_datamash}}) %} tool and read the help section at the bottom
>
>    > <question-title></question-title>
>    >
>    > Which settings do you think we need to provide to answer our question?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > - *"Group by fields"*: We want to group by sport (Column 15).
>    > > - *"Sort"*: `Yes`. This may not be obvious, but because our file is currently not sorted by our chosen group (sport), we need to tell the tool to do this.
>    > > - *"Skip NA or NaN values"*: since we do have NA values for athletes for whom height data is unknown, we should set this to `Yes`
>    > > - Our file has a header line, so we should indicate this as well
>    > > - *"Operation"*: we want to determine the **Maximum** height (Column 7)
>    > >
>    > {: .solution}
>    {: .question}
>
> 2. {% tool [Datamash]({{version_datamash}}) %} with the following parameters:
>    - {% icon param-file %} *"Input tabular dataset"*: `olympics.tsv`
>    - {% icon param-text %} *"Group by fields"*: `15`
>    - {% icon param-toggle %} *"Sort input"*: `yes`
>    - {% icon param-toggle %} *"Input file has a header line"*: `yes`
>    - {% icon param-toggle %} *"Skip NA or NaN values"*: `yes`
>    - *"Operation to perform on each group"*:
>      - {% icon param-select %} *"Type"*: `Maximum`
>      - {% icon param-select %} *"On Column"*: `Column: 7`
>
> 3. {% icon galaxy-eye %} **View** the results.
>
>    > <question-title></question-title>
>    >
>    > 1. How tall was the tallest athlete in basketball? And what about karate?
>    > 2. Why do some sports have a value of `inf`?
>    >
>    > > <solution-title></solution-title>
>    > >
>    > > 1. Basketball's tallest athlete was 192cm. For Karate it is 163.
>    > > 2. Our dataset had quite a number of `NA` (unknown) values in the height column, especially for the earlier Olympics. For sports that had only NA values, there is no maximum so the tool outputs `inf` instead.
>    > {: .solution}
>    {: .question}
>
{: .hands_on}


## Grouping on multiple columns

You may have noticed that we could also provide multiple columns to group on. If we do this, we can compute values for combinations of groups, such as sex and sport, to find e.g. the tallest woman in basketball or the shortest man per Olympics. There are also many more options for the computation we perform, so perhaps we are more interested not in the tallest athlete, but the average height. Let's perform some of these slightly more a


