# tutorials

topic: galaxy-interface
tutorial: onedata-remote-import
title: Importing (uploading) data from Onedata
difficulty: introductory
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: ["storage"]
content_hash: 78af24f14b92e310b33d1636b9d294c2
last_modified: 2026-05-27T09:01:48.662748
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html

content:
> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prerequisites

1. This tutorial assumes that you have basic knowledge about Onedata and access 
   to a Onedata ecosystem. If needed, follow 
   [this tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html %})
   first!
2. To use Onedata as a remote file source for data import, you need the
   **domain** of the **Onezone service** and a suitable **access token**. Here
   is the relevant 
   [guide on how to get them]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#accessing-onedata-services %}). 
3. The Galaxy server must be properly configured by the admins for the Onedata
   remote and/or Onedata BYOD templates to be available. Here is the 
   [corresponding tutorial]({% link topics/admin/tutorials/onedata-configuration/tutorial.html %}).


# Introduction

You may import the data stored in Onedata into Galaxy (also referred to as data
upload). There are two ways how to do that; either use the Bring Your Own Data
(BYOD) feature to configure your own **Remote File Sources** (recommended), or
use the ones configured by Galaxy admins (if applicable). Below hands-on
tutorials cover both scenarios.


> <tip-title>Disambiguation</tip-title>
> While Onedata can be used for both, a **Remote File Source** is not the same
> as a **Storage Location**. In this tutorial, you will be setting up a
> Onedata-based **Remote File Source**, which allows importing (uploading) data
> to Galaxy and exporting it from Galaxy. If you are looking to use a Storage
> Location, refer to 
> [this tutorial]({% link topics/galaxy-interface/tutorials/onedata-byos/tutorial.html %}).
{: .tip}


# Configuration

## Configure your own Onedata Remote File Source

To configure a Onedata remote using the Bring Your Own Data (BYOD) approach,
follow these steps:

1. Log in to your Galaxy account.
2. Go to the **Manage Your Remote File Sources** section of the **Preferences** menu.
3. Click **Create** in the top right corner to create a new Remote File Source.
   ![Create a Remote File Source](../../images/onedata-remote-import/configure-create-rfs.png)
4. Choose the **Onedata Storage** template. If there is no such template, the 
   Galaxy server is not configured to support it. Consider contacting its admins.
5. Fill in the information. If you wish to perform exports to this remote, check
   the **Writable?** toggle and make sure that your Onedata access token is write-enabled.
   ![Configure Onedata Remote File Source](../../images/onedata-remote-import/configure-onedata-rfs.png)
6. Click **Create** and it's done! You can now [use the Onedata remote](#importing-uploading-data).


## Configure a generic Onedata remote

This option may be viable if it's not possible to configure your own 
[Onedata Remote File Source](#configure-your-own-onedata-remote-file-source),
but requires that a generic remote is preconfigured on your Galaxy server.

Follow these steps:
1. Log in to your Galaxy account.
2. Go to the **Manage information** section of the **Preferences** menu.
3. Find the section called **"Your Onedata account"** and fill in the information:
   ![Onedata remote source](../../images/onedata-remote-import/preferences-generic-rfs.png).
   If there is no such section, the Galaxy server is not configured to support it. 
   Consider contacting its admins.
4. Done! You can now [use the Onedata remote](#importing-uploading-data).


# Importing (uploading) data

Uploading data to Galaxy from Onedata:

1. Navigate to the **Upload** menu, use the **Choose remote files** action, and find the 
   **Onedata** remote. There may be more than one if you configured it so: 
   ![Choose remote source](../../images/onedata-remote-import/upload-choose-rfs.png)
2. At the top level of the file browser, you will see all the Onedata Spaces that are
   accessible with the access token that was put down in preferences. You can browse
   through them and select the files that should be imported to Galaxy.
   ![Browse Onedata remote](../../images/onedata-remote-import/upload-browse-rfs.png)
3. After selecting the files, confirm with **Select**.
4. In the main upload menu, if you are happy with the queued files, click **Start**.
   ![Start upload](../../images/onedata-remote-import/upload-start.png)
5. You may have to wait a bit until your uploads are processed.


## Using the preconfigured remotes

You may also use Onedata-based remotes that were predefined by Galaxy admins for
public use (if applicable on your Galaxy instance). Note the **GTN training data**
on the [screenshot](#importing-uploading-data).

Since the GTN training data comes with a public access token, you may also configure
it as your own or generic Remote File Source. Use the following credentials:

* **Onezone domain**: `datahub.egi.eu`

This token is read-only, so make sure to mark the remote as not writable.


# Troubleshooting

If you experience any problems, take a look at the 
[troubleshooting]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#troubleshooting %}) 
guide.


# Related topics

The Onedata Remote file source can also be used for data export — see the 
[tutorial]({% link topics/galaxy-interface/tutorials/onedata-remote-export/tutorial.html %}).


topic: galaxy-interface
tutorial: onedata-remote-export
title: Exporting to Onedata remote
difficulty: introductory
time_estimation: 5m
tools_json: []
requirements_json: []
tags_json: ["storage"]
content_hash: a69a81b8f10a78af222560832dadd363
last_modified: 2026-05-27T09:01:48.662554
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/onedata-remote-export/tutorial.html

content:
# Prerequisites

1. This tutorial assumes that you have basic knowledge about Onedata and access 
   to a Onedata ecosystem. If needed, follow 
   [this tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html %})
   first!
2. To use Onedata as a remote file source for data export, you need the
   **domain** of the **Onezone service** and a suitable **access token**. Here
   is the relevant 
   [guide on how to get them]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#accessing-onedata-services %}). 
3. The Galaxy server must be properly configured by the admins for the Onedata
   remote and/or Onedata BYOD templates to be available. Here is the 
   [corresponding tutorial]({% link topics/admin/tutorials/onedata-configuration/tutorial.html %}).


# Introduction

Onedata Remote File Source can be configured as writable. In such a case, you
can export your Galaxy datasets to a Onedata Space.


# Configuration

Follow the same steps as in the 
[Importing (uploading) data from Onedata]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html#configuration %})
tutorial to configure a Onedata Remote File Source.

Make sure that the Remote File Source is writable. In case of you own,
it's as simple as checking the toggle in the configuration. In case of
generic remotes, it's in the hands of admins to mark the Onedata remote
as writable.


# Exporting datasets

Follow these steps:

1. Find the **Export datasets to remote files source** tool in the toolbox.
2. Choose the dataset to be exported and choose your Onedata Remote File Source
   in the **Directory URI** section.
   ![Export dataset](../../images/onedata-remote-export/dataset-export.png)
3. Adjust the configuration if needed.
4. Run the tool.
5. Check your Onedata account to find the exported dataset at the specified path.


# Troubleshooting

In case of errors:

1. Check the section **Use distributed compute resources** in the
   **Manage Information** section of the **Preferences** menu. It's
   possible that the specific Pulsar endpoint is unable to run the
   job. Try using the default settings.
2. Avoid whitespace characters (spaces, tabs) in the **Directory URI**,
   as they are known to cause problems.
3. Take a look at the 
   [troubleshooting]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#troubleshooting %}) 
   guide.


# Related topics

The Onedata Remote file source can also be used for data import — see the 
[tutorial]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %}).


topic: galaxy-interface
tutorial: workflow-parameters
title: Using Workflow Parameters
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["workflows"]
content_hash: aeb5db4fe98316f3f1fb9aff42558588
last_modified: 2026-05-15T22:28:13.372238
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/workflow-parameters/tutorial.html

content:
Workflows are a powerful feature in Galaxy that allow you to chain multiple steps of an analysis together.
To make a workflow reusable with slightly different settings you can define and use workflow parameters.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add workflow parameters to a workflow

We will import a simple workflow and then demonstrate how workflow parameters
can be added.

> <hands-on-title>Import integer workflow</hands-on-title>
>
> 1. Click on Workflow in the top menu
> 2. Click on the {% icon galaxy-upload %} import workflow button on the right
> 3. Enter the following URL into the "Archived Workflow" URL box
>
>    ```
>    https://raw.githubusercontent.com/galaxyproject/training-material/master/topics/galaxy-interface/tutorials/workflow-parameters/workflows/cut_n_lines.ga
>    ```
>
> 4. Click on import workflow
{: .hands_on}


# Add an integer workflow parameter

This workflow takes an input dataset and outputs the first 10 lines of that input dataset.
The number of lines that should be output can be set using the `Number of lines` parameter
of the select first tool.

Instead of selecting a specific number of lines we can choose to insert a workflow parameter
that allows the user to set this parameter when running the workflow.

![Button for selecting connections](../../images/connection-module.png "The new button for creating a workflow parameter connection. When you click this, it will add a new 'input' to the tool that can be connected, just like other tools in workflows.")

> <hands-on-title>Updating the Workflow</hands-on-title>
>
> 1. Upload a dataset using "Paste/Fetch data" with the contents:
>
>    ```
>    E. coli (O19:H34. Extracellular pathogenic)
>    E. coli (O7:K1. Extracellular pathogenic)
>    E. coli 536
>    E. coli APEC O1 (O1:K12:H7. APEC
>    E. coli CFT083 (O6:K2:H1. UPEC)
>    E. coli E2348/69 (O127:H6. Enteropathogenic)
>    E. coli EC4115 (O157:H7 EHEC)
>    E. coli ED1a O81 (Commensal)
>    E. coli EDL933 (O157:H7 EHEC)
>    E. coli F11
>    E. coli S88 (O45:K1. Extracellular pathogenic)
>    E. coli SE15 (O150:H5. Commensal)
>    E. coli Sakai (O157:H7 EHEC)
>    E. coli TW14359 (O157:H7 EHEC)
>    E. coli UMN026 (O17:K52:H18. Extracellular pathogenic)
>    E. coli UTI89 O18:K1:H7. UPEC)
>    ```
>
> 2. Open the **workflow editor**
>
>    {% snippet faqs/galaxy/workflows_edit.md %}
>
> 3. Click the **Select first** {% icon tool %} tool in your workflow
> 4. Find the **Number of lines** parameter in the right hand panel.
> 5. Click on the workflow connection button {% icon galaxy-wf-connection %} to convert the Number of lines parameter into a workflow parameter connection
>
>    You will see that a new input with the text `Number of lines` has appeared on the **Select first** tool in the editor.
>
> 6. Add a **Simple inputs used for workflow logic**, found under the Inputs section of your toolbox
> 7. If you click on the Input parameter box in the editor you will see in the `Details panel` that the **Parameter type** is set to `Text`. If you try to connect this parameter to the "Number of lines" parameter you will see that the noodle turns orange and that you cannot create a connection.
>
> 8. Click on Input parameter, and change the **Parameter type** to **Integer**
>
> 9. Connect *"output"* from Input parameter {% icon tool %} to the *"Number of lines"* input of the Select first {% icon tool %}.
>
> 10. Save {% icon galaxy-save %} your workflow
>
> 11. Run your workflow
>
>     {% snippet faqs/galaxy/workflows_run.md %}
>
>    Notice the new input that can be changed before the workflow is run
{: .hands_on }


![Animation of simple integer workflow parameter](../../images/workflow_integer_param.gif "Integer workflow parameter")

> <tip-title>Parameter Reuse</tip-title>
>
> You can connect a parameter to multiple steps or to multiple parameters within a step, everywhere you need to use it.
{: .tip}

# Create a text parameter from multiple parameters

It is often necessary to create a text parameter that is flanked by additional text.
Take for instance a regular expression that finds `foo` in a string and replaces it
with `foobar` as in `s/(foo)/\1bar/`.
The Galaxy tool ``Regex Find And Replace`` can be used to tune such a regular expression. The find part of the regex (`s/(foo)/`) can be defined in the `Find regex` parameter and the replacement part can be entered in the `Replacement` parameter.
If we want to make the `foo` part of the regular expression configurable we can
compose this text parameter using the `Compose text parameter value` tool.

> <hands-on-title>Compose a text parameter</hands-on-title>
> 1. Create a new workflow
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
> 1. Add an Inputs → **Input Dataset** to the workflow
> 2. Add an Inputs → **Simple inputs used for workflow logic** to the workflow
>    - {% icon param-select %} *"Parameter type"*: `Text`
> 3. Add **Compose text parameter value** {% icon tool %} to the workflow
>     - Add two more "Components" using the "Insert components" button in the workflow editor tool form on the right of the screen. You will need three "Component" blocks in total.
>     - In the first component:
>       - *"Enter text that should be part of the computed value"*: `(`
>     - In the second component:
>       - *"Enter text that should be part of the computed value"*: Leave empty and click "Add connection to module" {% icon galaxy-wf-connection %}
>       - Connect the output of the **Input parameter** {% icon tool %} to this new input
>     - In the third component:
>       - *"Enter text that should be part of the computed value"*: `)`
>
> 4. Add the **Regex Find And Replace** {% icon tool %} to the workflow
>     - Click on `Insert Check`
>     - *"Find Regex"*: Click "Add connection to module" {% icon galaxy-wf-connection %}
>     - *"Replacement"*: `\1bar`
>     - Connect the output of the **Compose text parameter value** {% icon tool %} to the *"Find Regex"* parameter of **Regex Find And Replace** {% icon tool %}.
>     - Connect the output of the **Input dataset** {% icon tool %} to the *"Select lines from"* input of the **Regex Find And Replace** {% icon tool %}.
>
> 5. **Save** {% icon galaxy-save %} your workflow
{: .hands_on}

You've now built a workflow with a parameterised input! It's time to test it out.

> <hands-on-title>Running the workflow</hands-on-title>
>
> 1. Upload a dataset using "Paste/Fetch data" with the contents `wunder`
>
> 2. Run your workflow with the following parameters
>
>    - *"Input parameter"*: `wunder`
>
>      This is the value that will be looked for in your input dataset.
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> 3. Examine the outputs
{: .hands_on }

You should see two new datasets in your history. The first dataset has the data type `expression.json` and contains the composed parameter value `(wunder)`, the second dataset will contain the output of the **Regex Find And Replace** {% icon tool %} step. A click on the information {% icon details %} button will show the parameters for the tool. You will see that the *"Find Regex"* parameter will contain the values that you entered in the workflow run form. If you look at the dataset content you will see it is `wunderbar`.

# Read a parameter from a dataset

Often times it is necessary to calculate a parameter in one step of a workflow and then to use it in another step of the same workflow. This can be accomplished by reading the parameter from a dataset (As long as it is a text, integer, float, boolean or color parameter).
In this example we will construct a workflow where we calculate the sum of all values in a dataset and then divide the values in this dataset by the sum calculated in the previous step.

> <hands-on-title>Construct Workflow with Parameters read from a dataset</hands-on-title>
>
> 1. Create a new workflow
> 1. Add an **Input dataset** {% icon tool %}
> 2. Add **Datamash** (operations on tabular data) {% icon tool %} to the workflow
>     - {% icon wf-input %} *"Input tabular dataset"*: Connect the noodle from the output of the **Input dataset** {% icon tool %} to this input
>     - {% icon param-repeat %} **Operation to perform in each group**
>       - {% icon param-select %} *"Type"*: `sum`
>       - {% icon param-text %} *"On column"*: `1`
> 3. Add **Parse parameter value** {% icon tool %} to the workflow
>     - *"Select type of parameter to parse"*: integer
>     - {% icon wf-input %} *"Input file containing parameter to parse out of"*: Connect the **Datamash** {% icon tool %} output to this input
> 4.  Add **Compose text parameter value** {% icon tool %} to the workflow
>     - In the first repeat:
>       - {% icon param-select %} *"Choose the type of parameter for this field"*: Text Parameter
>       - {% icon param-text %} *"Enter text that should be part of the computed value"*: `c1/`
>     - Click Insert Repeat
>     - In the second repeat:
>       - {% icon param-select %} *"Choose the type of parameter for this field"*: Integer Parameter
>       - *"Enter integer that should be part of the computed value"*: Click on `Add connection to module` {% icon galaxy-wf-connection %}
>       - {% icon wf-input %} *"Input"*: Connect the output of the **Parse parameter value** {% icon tool %}
> 5. Add **Compute on rows** {% icon tool %} to the workflow
>     - *"Input file"*: Select the output of the **Input dataset** {% icon tool %}
>     - *"Add expression"*: click on `Add connection to module`, then connect the output of **Compose text parameter value** {% icon tool %}
{: .hands_on}

With this you're ready to run the workflow!

> <hands-on-title>Running the workflow</hands-on-title>
>
> 1. Upload a dataset using "Paste/Fetch data" with the contents, and set the filetype manually to "tabular" during upload
>
>    ```
>    16378
>    16014
>    2831
>    5702
>    149
>    24383
>    12220
>    4488
>    11500
>    24724
>    ```
>
> 2. Run your workflow with the following parameters
>
>    - *"Input dataset"*: the table you have just uploaded
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> 3. Examine the outputs
{: .hands_on }

This workflow will produce a new dataset, where the last column will be the
result of dividing the value in the first column by the sum of all values in
the first column.


> <tip-title>You can try many different parameter values at once</tip-title>
>
> * Often times you need to try a couple of different parameter values and pick the best one.
>   You can create or compute a dataset for each parameter you would like to try and run the
>   `Parse parameter value` tool on it, running all downstream tools once for each parameter.
{: .tip}

# Conclusion


Galaxy Workflows chain together different steps of an analysis. To make your workflows
more useful to your colleagues you can add workflow parameters. Sometimes a parameter is not
known in advance, but can be calculated as part of the workflow. Now you know how to read these
parameters from datasets!


topic: galaxy-interface
tutorial: download-delete-data
title: Downloading and Deleting Data in Galaxy
difficulty: introductory
time_estimation: 20M
tools_json: []
requirements_json: []
tags_json: []
content_hash: 997dc649c8e83e8e2b94a4e04d350c53
last_modified: 2026-05-27T09:01:48.661375
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/download-delete-data/tutorial.html

content:
This tutorial outlines the process to get your data out of Galaxy and to delete it from Galaxy afterwards.

After you have completed your analysis on a Galaxy server, you may need to download the results for use in reports, papers, or other requirements. Galaxy has a myriad of methods for downloading data from individual datasets to a collection of data to an entire history along with all of its metadata. The first section of this tutorial will walk you through all of these methods.

On some Galaxy servers, user disk quotas have been enabled meaning that users are only able to store a certain amount of data on that particular Galaxy server. Quotas can differ on different servers. User disk quotas for the `usegalaxy.*` servers are shown in the table below:

| Galaxy Server    | Unregistered User   | Registered User                      |
| ---------------  | ------------------- | -----------------                    |
| usegalaxy.org    | 5 GB                | 250 GB                               |
| usegalaxy.eu     | 5 GB                | 250 GB or 500 GB for Elixir Members  |
| usegalaxy.org.au | 5 GB                | 100 GB or 600GB for Australian Users |

Other Galaxy servers may have different quota systems.

As you use a Galaxy server and upload data, perform analyses etc. you will use up your quota. If you have filled your quota and you want to do some new work, you can download your old work and delete it off the Galaxy server to free up quota space. Galaxy uses a two step process to delete user data from disk. The first step is to delete the data - this marks the data as deleted but it can be undeleted for a certain time[^1]. The second step is to purge the deleted data - this removes the deleted data from the disk and it is no longer recoverable. Note that any data that has been shared with another user will not be deleted from disk unless the other users also delete it.

The second section of this tutorial will outline how to delete data from your user space and to purge it from disk if you so desire.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Downloading Data from Galaxy

There are multiple ways of downloading your data from Galaxy, the choice of which method to use depends on the amount of data you wish to download. You can download a single dataset, multiple datasets all the way to entire histories.

## Downloading a single file from Galaxy

The easiest way to get your data is to download a single dataset from a history.

The following example steps through the process using a generic history. The history pictured in the screenshots will most likely not appear in your Galaxy server but the process will be the same.

> <hands-on-title>Download a single file</hands-on-title>
>
> 1. In the History pane, click on the name of the file you wish to download. This will expand the file in the History to show more details about it.
>
>    ![expanded file history](../../images/download-delete/expanded_file_history.png)
>
> 2. Click the {% icon galaxy-save %} **Download** button. The dataset will now be downloaded to your computer.
>
>    ![download button](../../images/download-delete/file_download_button.png)
>
{: .hands_on}

Repeat this for any other data file you want in this or other histories.

## Download multiple datasets

Sometimes you may want to download multiple datasets at once and doing them one by one can be a pain. Luckily, Galaxy has a mechanism for creating Collections. For more details about collections and how to create them, see [this tutorial]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}) from the Galaxy Training Network.

The concept is very simple however, we are just going to lump together our files of interest and download them all at once.

> <hands-on-title>Download multiple datasets</hands-on-title>
>
> **Create a collection of the files you want**
>
> 1. Click on the **Collections** button at the top of the History pane (the little checkbox)
>
>    ![collections button](../../images/download-delete/collections_button.png)
>
>    This shows a different view of the History. Each file now has a checkbox next to it and there are some new buttons in the header section of the History.
>
> 2. Select the files you want by clicking on their checkboxes, you can also select all or none of them.
>
>    ![collection selection](../../images/download-delete/collection_selection.png)
>
> 3. Once you have selected everything you want, click the **for all selected** button and then **Build Dataset List**
>
> 4. A new interface opens, showing you the files you have selected. Give the list a name in the appropriate box and click **Create**
>
>    ![list interface](../../images/download-delete/list_interface.png)
>
> 5. A new collection item has now appeared in your History.
>
>    ![collection in history](../../images/download-delete/collection_in_history.png)
>
> **Download the collection**
>
> 1. Click on the name of the collection to show its contents
> 2. Click the {% icon galaxy-save %} **Download Collection** button to download the collection of files as a compressed archive to your computer.
>
>    ![collection_download](../../images/download-delete/collection_download.png)
>
{: .hands_on}

You can repeat this process for other sets of files you want to download in other histories.

## Download (export) an entire history including its metadata

The third way to download data from Galaxy is by Exporting a history in its entirety. This will allow you to download all of the data in a history as well as all of the metadata associated with it such as the tools used, their versions, the parameters chosen etc. It is downloaded as a single file archive. Once downloaded, you can also Import this history into another Galaxy server.

Depending on the size of the files and the complexity of the history, it can take quite some time to create the archive before download. For example it can take over 24 hours to create an archive file of a 50GB history. It may be simpler just to download the individual files of interest as outlined above.

> <hands-on-title>Export an entire history</hands-on-title>
>
> 1. In the History pane, click on the {% icon galaxy-gear %} history menu, then select **Export History to File**
>
>    ![history export](../../images/download-delete/history_export.png)
>
> 2. Galaxy will now start archiving the history and making it available for download. In the main Galaxy pane, a message will appear with a download URL. Once the history has been completely archived, the URL can be used to download the history archive file.
>
>    ![history export link](../../images/download-delete/history_export_link.png)
>
> 3. Click on the URL to start the download. If it isn’t ready yet, it will show you the following message. You will need to save the URL and try again later.
>
>    ![history export not ready](../../images/download-delete/history_export_not_ready.png)
>
{: .hands_on}

# Deleting Data from Galaxy

Once you have downloaded everything that you want to keep from your Galaxy account, you can delete the data and histories your no longer require.

Deletion in Galaxy is a two stage process. If you delete something, it actually only gets marked as deleted and then is hidden. To actually delete an object you must purge it from disk. The instructions for how to delete datasets and histories follow.

## Delete a single dataset

This section will show you how to delete and then purge a single dataset from a history. You can also delete multiple datasets and then purge them all at once.

> <hands-on-title>Delete a single dataset</hands-on-title>
>
> 1. Click on the {% icon galaxy-delete %} **Delete** button of the dataset you wish to delete
>
>    ![delete button](../../images/download-delete/delete_button.png)
>
> 2. The dataset has now disappeared from the history pane and the text `1 deleted` appears below the history name.
>
>    ![one deleted](../../images/download-delete/one_deleted.png)
>
> 3. You can see all of the deleted datasets by clicking on the `1 deleted` link at the top of the history pane. This will allow you to either: `Undelete it` OR `Permanently remove it from disk` by clicking on the appropriate link. **Note: Once you click on the permanently remove it from disk link you CANNOT undo it.**
>
>    ![show deleted](../../images/download-delete/show_deleted.png)
>
> 4. You can also delete and purge multiple datasets at a time by clicking on the **delete** button on all the datasets you want to remove and then purge them all at once by going to the history menu (the little cog wheel) and selecting **Purge Deleted Datasets**
>
>    ![purge deleted](../../images/download-delete/purge_deleted.png)
>
{: .hands_on}

## Delete an entire History

You can delete an entire history and purge it from disk.

> <hands-on-title>Delete an entire History</hands-on-title>
>
> 1. Open the **History List** display by clicking on the {% icon galaxy-columns %} history list button
>
>    ![saved histories](../../images/download-delete/saved_histories.png)
>
> 2. You will now see a list of all of your histories
>
>    ![all_histories](../../images/download-delete/all_histories.png)
>
> 3. You can now delete a history by clicking on the {% icon galaxy-dropdown %} dropdown arrow at the top of the history, and then selecting **Delete** OR **Purge**.
>
>    > <warning-title>You can lose data</warning-title>
>    > **NOTE: Selecting Delete Permanently will remove the entire history and all of its non shared datasets from disk. It CANNOT be undone.**
>    {: .warning}
>
> 4. If you only selected **Delete**, the history will be available to un-delete for a certain time[^1]
>
{: .hands_on}

# Conclusion

This tutorial should have explained to you how to manage your data on Galaxy. Different Galaxy servers will have different policies regarding storage quotas and storage times. You should check with the Galaxy server you are using.

## Footnotes:

[^1]: On some Galaxy servers, data and histories marked for deletion are periodically purged by the administrators. This period of time can vary between servers and you should check with the documentation or administrators of your server to know the exact amount of time.


topic: galaxy-interface
tutorial: name-tags
title: Name tags for following complex histories
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 771040c4d2ce448f357c5e1c222cce78
last_modified: 2026-05-15T22:28:13.339332
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/name-tags/tutorial.html

content:
Long ad hoc data analysis sessions in Galaxy can leave you with a multitude of
datasets named things like "1724: Summarize taxa on data 1628 and data 1629" or "1853: Pick
OTUs Combined: OTUs". While these names are useful in smaller analysis sessions
with fewer datasets, as the size of your history grows, so does the amount of
time it takes to figure out where your datasets came from.

You have two options to deal with this:

- Re-name datasets constantly with more useful titles
- Use name tags once, on the inputs to your analysis

Name tags "propagate", they will follow your datasets throughout their analysis.
To more clearly see this, look at the simple workflow below. If the input datasets are tagged `TB8_R1` and `TB8_R2`, these tags will propagate to the outputs of any tools run on these datasets. If a tool has multiple inputs with tags, the outputs will have all of the tags like in the bowtie2 tool.

![A workflow with some tags overlaid](../../images/nametag-wf.png)

It can turn your histories from a complicated mess into an easier to follow analysis, as each step is clearly labelled with the original input sample corresponding to each analysis:

Bad | Good
--- | ---
![History with untagged and poorly named datasets](../../images/nametag-why.png)| ![History with tagged datasets showing how much easier it is](../../images/nametag-why-fixed.png)
How would you identify which input sample corresponds with which featureCounts output? | With name tags applied before the start of analysis, each output is also tagged with the appropriate sample name

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Setting name tags manually

> <hands-on-title>Set name tags manually</hands-on-title>
>
> 1. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Using the **Upload** {% icon galaxy-upload %} tool select **Paste/Fetch data** and paste some data like "Hello, World"
>
> 3. When this is uploaded, click on the title of the dataset in your history panel to expand it
>
> 4. Add some name tags (hashtags) to your dataset:
>
>    - `#female`
>    - `#male`
>    - `#dmel6`
>    - `#ecoli`
>
>    {% snippet faqs/galaxy/datasets_add_tag.md type="name" %}
>
{: .hands_on}

These are example tags, but it is recommended that you use name tags to reflect the identity of the input sample, in some way that's important to you. If you're doing an analysis comparing wildtype and treated samples, you might tag things based on the condition. If you're processing NGS data you might want to tag based on the read set where each sample originated.

If you're using a Galaxy server newer than 19.01, you will notice that the tags each have different colours:

![Name tags have different colours based on their contents](../../images/nametag-colour.png)


> <details-title>In depth: Colours</details-title>
>
> The colours are generated from a checksum of the text of each tag. Every time the same tag is used, the same colour will be generated. Different words will usually result in different colours.
>
> This allows tags with very similar lengths and word shapes (e.g. `peaks` and `genes`) which can be mistake at small font size if you aren't careful.
> While current evidence ({% cite microsoft-word-recognition %}) doesn't support the "word shape" model of word recognition, it has still been seen to be beneficial to provide more contrast between words when mistakes can be costly ({% cite ntrs-19930010781 %}). In the context of Galaxy, mistakes are usually not incredibly costly, but can be annoying. This gives an easy additional signal on which you can mentally filter your datasets.
>
> You can read some of the discussion that went into implementing the tags [in the pull request to Galaxy](https://github.com/galaxyproject/galaxy/pull/7072).
{: .details}

# Setting name tags with the Rule Builder

In this section we will demonstrate how you can add name tags to datasets automatically using the rule-based uploader. This is especially useful when you have a large number of datasets you want to tag all at once.

For more information about the rule-based uploader, please see [this dedicated tutorial]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %})

> <hands-on-title>Set name tags during upload</hands-on-title>
>
> 1. Open the **Upload** {% icon galaxy-upload %} interface
> 2. Click on **Rule-based** at the top
>
>    ![Rule-based upload](../../images/group-tags/rule_upload.png)
>
>    As you can see in this dialog, data can be selected from a history dataset or pasted in directly
>
> 4. Set **Upload data as:** to `Datasets`
> 5. Paste the following links into the text box
>
>    ```
>    https://zenodo.org/record/1185122/files/GSM461176_untreat_single.counts
>    https://zenodo.org/record/1185122/files/GSM461177_untreat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461178_untreat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461179_treat_single.counts
>    https://zenodo.org/record/1185122/files/GSM461180_treat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461181_treat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461182_untreat_single.counts
>    ```
>
> 6. Click **Build**
>
> 7. We will add a regex that creates 3 new columns with accession, treatment and library type:
>    - Click on the **Columnn** button and then **Using a Regular Expression**
>    - Select **Create columns matching expression groups**
>
>        - Paste `.*(GSM.*)_(.*)_(.*).counts` in *"Regular Expression"*
>        - Set *"Number of Groups"* to 3
>        - Click on **Apply**
>
>      We should have now a table with 4 columns: link, sample name, treatment, sequencing type
>
>    - Click on **Rules** and then **Add / Modify Column Definitions**
>        - Click on **Add Definition** and select:
>          - *"URL"*: Column A
>          - *"Name"*: Column B
>          - *"Name Tag"*: Column B
>          - *"General Purpose Tags(s)"*: Columns C and D
>    - Click **Apply**
>    - Click **Upload**
{: .hands_on}

This will add 7 new datasets to your history, each one tagged and named appropriately.

![Automatically tagged uploads](../../images/nametag-rulebuilder.png)


The datasets now have two types of tags: name tags and general purpose tags. General purpose tags, unlike name tags, will not propagate. They're mainly useful for searching through your datasets (Try entering `tag=paired` in your History search box).

> <question-title></question-title>
>
> 1. How many tags does the dataset named `GSM461177` have? Which ones?
> 2. How many of those tags are propagating name tags?
>
> > <solution-title></solution-title>
> >
> > 1. 3 tags. You can find all tags of a dataset by first clicking on it to expand it, and then clicking on the tags icon {% icon galaxy-tags %}. The tags are `#GSM461177`, `paired`, and `untreat`
> > 2. 1 propagating name tag. Only hashtags (tags starting with a `#` symbol) will propagate to downstream results derived from it. So in this example only `#GSM461177` is a propagating name tag.
> >
> {: .solution}
{: .question}


# Name tags in Action

Now that we have our name tags in place, let's see them in action!

> <hands-on-title>Some analysis steps</hands-on-title>
> 1. **Select first** {% icon tool %} lines from a dataset (head)
>      - {% icon param-files %} *"File to select"*: Select everything starting with `GSM`
> 2. Click **Run Tool**
{: .hands_on}

We can already see the benefits of name tags. Even though our outputs are named things like "Select first on data 8", they're also visibly tagged with the name of the original input.

![Tagged datasets are clearly traceable](../../images/nametag-select.png)

Next we'll join two of these datasets together, to see the tagging behaviour on summaries of multiple datasets. In any box which allows selecting single files, you can also search on name tags.

![Filtering datasets is possible on tag names](../../images/nametag-filter.png)

> <hands-on-title>Multiple inputs</hands-on-title>
> 1. **Join two datasets** {% icon tool %} side by side on a specified field
>      - {% icon param-file %} *"Join"*: Click the select box and type "461181" to filter the list and use the "Select first on data ..." file
>      - {% icon param-file %} *"With"*: Click the select box and type "461182" to filter the list and use the "Select first on data ..." file
> 2. Click **Run Tool**
{: .hands_on}

The output will include all of the name tags from all of the inputs:

![The output dataset should be tagged with both of the inputs](../../images/nametag-join.png)

If you compare the tags of the last output joining the two tables, and the original inputs, you will notice that the "General Purpose" tags were lost along the way. Only name tags are propagated with each analysis step.


topic: galaxy-interface
tutorial: upload-rules
title: Rule Based Uploader
difficulty: intermediate
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["collections", "tags"]
content_hash: 1b98e800d24b33e4f8d4a70cd755c9ea
last_modified: 2026-05-15T22:28:13.369300
zenodo_link: https://zenodo.org/records/3263975
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/upload-rules/tutorial.html

content:
Through a series of examples, this tutorial aims to familiarize the reader with building Galaxy collections from tabular data containing URLs, sample sheets, list of accessions or identifiers, etc..

> ###  {% icon comment %} Audience
> This tutorial assumes a basic knowledge of using dataset collections in Galaxy but doesn't assume any particular knowledge of biology or bioinformatics. If you have not used collections with Galaxy previously, please check out the [using dataset collections]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}) tutorial.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Uploading Datasets with Rules


This approach could be used to manipulate lists of uploads coming from many different formats, but we will start with a tabular description of files for a study from the [European Nucleotide Archive](https://www.ebi.ac.uk/ena). We will be using the data from a study of [16s ribosomal RNA](https://www.ebi.ac.uk/ena/data/view/PRJDA60709).

> <hands-on-title>Downloading Data from ENA</hands-on-title>
> To start, navigate to the above study and click the option to "Show Column Selection". Here we will narrow the set of columns we consider to just a few relevant to uploading these files to Galaxy.
>
> Select only:
>
> - `study_accession`
> - `experiment_accession`
> - `fastq_ftp`
> - `sample_accession`
>
> The table should look like:
>
> Study accession | Sample accession | Experiment accession | FASTQ files (FTP)
> --------------- | ---------------- | -------------------- | -----------------
> PRJDA60709      | SAMD00016379     | DRX000475            | DRX000475.fastq.gz
> PRJDA60709      | SAMD00016383     | DRX000476            | DRX000476.fastq.gz
> PRJDA60709      | SAMD00016380     | DRX000477            | DRX000477.fastq.gz
> PRJDA60709      | SAMD00016378     | DRX000478            | DRX000478.fastq.gz
> PRJDA60709      | SAMD00016381     | DRX000479            | DRX000479.fastq.gz
> PRJDA60709      | SAMD00016382     | DRX000480            | DRX000480.fastq.gz
>
> Download the resulting tabular data describing the files by clicking the "Download report: TSV" link above the table. Alternatively, the [resulting sample sheet can be downloaded directly](https://www.ebi.ac.uk/ena/portal/api/filereport?accession=PRJDA60709&result=read_run&fields=study_accession,sample_accession,experiment_accession,fastq_ftp&format=tsv&download=true&limit=0). The number and size of the files for this example are relatively small for sequencing data but larger files and larger numbers of files should work as well - Galaxy will just need more time to download and process the files.
{: .hands_on}

Unfortunately the ENA ftp server is not operational at this moment,
so we have to use a backup from zenodo.
You can select all the data below and copy them to your clipboard.

```
study_accession	sample_accession	experiment_accession	fastq_ftp
PRJDA60709	SAMD00016379	DRX000475	https://zenodo.org/records/3263975/files/DRR000770.fastqsanger.gz
PRJDA60709	SAMD00016383	DRX000476	https://zenodo.org/records/3263975/files/DRR000771.fastqsanger.gz
PRJDA60709	SAMD00016380	DRX000477	https://zenodo.org/records/3263975/files/DRR000772.fastqsanger.gz
PRJDA60709	SAMD00016378	DRX000478	https://zenodo.org/records/3263975/files/DRR000773.fastqsanger.gz
PRJDA60709	SAMD00016381	DRX000479	https://zenodo.org/records/3263975/files/DRR000774.fastqsanger.gz
PRJDA60709	SAMD00016382	DRX000480	https://zenodo.org/records/3263975/files/DRR000775.fastqsanger.gz
```
{:#example-1-metadata}

> <hands-on-title>Accessing the Rule Based Uploader</hands-on-title>
>
> 1. **Click** the upload icon toward the top left corner.
>
>    By default the familiar simple upload dialog should appear. This dialog has more advanced options as different tabs across the top of this dialog though.
>
> 2. **Click** `Rule-based` as shown below.
>
>    ![screenshot](../../images/rules/rules_example_1_1_rules_landing.png)
>    As you can see in this dialog, data can be selected from a history dataset or pasted in directly. If Galaxy is configured to allow FTP uploads, the contents of your FTP directory may be loaded directly as well.
>
> 3. **Paste** your tabular data right into the textbox on this page as shown below
>     - **Click** the `Build` button.
>
>    ```
>    study_accession	sample_accession	experiment_accession	fastq_ftp
>    PRJDA60709	SAMD00016379	DRX000475	https://zenodo.org/records/3263975/files/DRR000770.fastqsanger.gz
>    PRJDA60709	SAMD00016383	DRX000476	https://zenodo.org/records/3263975/files/DRR000771.fastqsanger.gz
>    PRJDA60709	SAMD00016380	DRX000477	https://zenodo.org/records/3263975/files/DRR000772.fastqsanger.gz
>    PRJDA60709	SAMD00016378	DRX000478	https://zenodo.org/records/3263975/files/DRR000773.fastqsanger.gz
>    PRJDA60709	SAMD00016381	DRX000479	https://zenodo.org/records/3263975/files/DRR000774.fastqsanger.gz
>    PRJDA60709	SAMD00016382	DRX000480	https://zenodo.org/records/3263975/files/DRR000775.fastqsanger.gz
>    ```
>
>    ![screenshot](../../images/rules/rules_example_1_2_paste.png)
>
> 4. **Examine** the  the `rules editor` which now pops up.
>
>    ![screenshot](../../images/rules/rules_example_1_3_initial_rules.png)
>
{: .hands_on}

At first glance, this may be feel like Excel or another spreadsheet program and you may feel the urge to start editing cells but we strongly encourage defining rules for manipulating the data instead. There are a few reasons for this:

* **Manually modifying this metadata is not reproducible** - we will not belabor the point here but check out [Why not use excel for this?]({% link faqs/galaxy/why_not_excel.md %}) for more context. Building up rules for modifying this metadata will allow Galaxy to track and report your manipulations (providing traceability) and apply them to new sets of files (providing reproducibility).
* **Manually modifying this metadata is error prone** - we believe defining rules and treating the metadata in a systematic way minimizes the possibility to manual errors. These errors can be very hard to detect for large sets of data.
* **Manually modifying data is not scalable** - this rule-based technique potentially scales to importing tens of thousands of datasets.

So rather than modifying the data, we will define rules for manipulating it and setting up "column definitions" that tell Galaxy how to use the metadata during upload or collection creation.

In order to get these files into Galaxy, we will want to do a few things:
* Strip that header out of the data (it doesn't contain a URL Galaxy can download).
* Define column `C` as the dataset name.
* Define column `D` as the dataset URL (this is the location Galaxy can download the data from).
* Tell Galaxy to treat these files as `fastqsanger.gz` files.

> <hands-on-title>Using Tabular Inputs to the Rule Builder</hands-on-title>
>
> We will start by stripping that header out of the table. We call rules that strip rows out the table `Filter rules` in this dialog.
>
> 1. From **Filter** menu select `First or Last N Rows`
>    - *"Filter which rows?"*: `first`
>    - *"Filter how many rows?"*: `1`
>    - Click `Apply`
>    ![screenshot](../../images/rules/rules_example_1_4_filter_header.png)
>
>    Next we will define these columns for Galaxy.
>
> 2. From **Rules** menu select `Add / Modify Column Definitions`
>     - Click `Add Definition` button and select `Name`
>       - *"Name"*: `C`
>     - Repeat this again and select `URL` instead.
>       - *"URL"*: `D`
>
>    ![screenshot](../../images/rules/rules_example_1_5_mapping_edit.png)
>
> 3. **Click** `Apply`, and you should see your new column definitions listed as in the following screenshot.
>    ![screenshot](../../images/rules/rules_example_1_6_mapping_set.png)
>
>    You are now ready to start the upload
>
> 4. **Click** the `Upload` button
>
>    After some time, the result in this case will be six datasets in your history. The next example will show to use the rule builder to create a collection.
{: .hands_on}

# Creating a Simple Dataset List

This example will demonstrate using such history datasets as the source for collection uploads - this can be handy when you'd like to apply existing Galaxy tabular manipulation tools to the metadata before processing for instance.

> <hands-on-title>Creating a Simple Dataset List</hands-on-title>
>
> > <tip-title>Create a new history</tip-title>
> > Before we start uploading, it may be a good idea to create a new history for this example to keep things simple and match the following screenshots.
> {: .tip}
>
> For our second example, we will use the same initial metadata, but instead uploaded as a file to Galaxy. This is a relatively common use case wherein you, through some filtering and querying in Galaxy, build a list of accession numbers or other sample identifiers, and then want to fetch the associated datasets to begin working with them.
>
> > <comment-title>Loading Metadata from a History Element</comment-title>
> > In addition to directly pasting data into the Rule Based Uploader, you can also load the metadata from a dataset in your Galaxy History, or from a file in your FTP directory if the admin has enabled FTP upload
> {: .comment}
>
> 1. **Upload** the metadata from the [first example](#example-1-metadata) to your Galaxy, as a normal paste upload:
>
>    ```
>    study_accession	sample_accession	experiment_accession	fastq_ftp
>    PRJDA60709	SAMD00016379	DRX000475	https://zenodo.org/records/3263975/files/DRR000770.fastqsanger.gz
>    PRJDA60709	SAMD00016383	DRX000476	https://zenodo.org/records/3263975/files/DRR000771.fastqsanger.gz
>    PRJDA60709	SAMD00016380	DRX000477	https://zenodo.org/records/3263975/files/DRR000772.fastqsanger.gz
>    PRJDA60709	SAMD00016378	DRX000478	https://zenodo.org/records/3263975/files/DRR000773.fastqsanger.gz
>    PRJDA60709	SAMD00016381	DRX000479	https://zenodo.org/records/3263975/files/DRR000774.fastqsanger.gz
>    PRJDA60709	SAMD00016382	DRX000480	https://zenodo.org/records/3263975/files/DRR000775.fastqsanger.gz
>    ```
>
> 2. **Open** the `Rule-based` upload tab again, but this time:
>    - *"Upload data as"*: `Collection(s)`
>    - *"Load tabular data from"*: `a History Dataset`
>    - *"Select dataset to load"*: the dataset you just uploaded
>
>    ![screenshot](../../images/rules/rules_example_2_1_inputs.png)
>
>    > <tip-title>Don't see your dataset?</tip-title>
>    > If you don't see your dataset, refresh your page and try again.
>    {: .tip}
>
> 3. **Click** `Build` to bring up the rule builder.
>    ![screenshot](../../images/rules/rules_example_2_2_initial_rules.png)
> 4. **Repeat** the steps from last time, except define column `C` as a `List Identifier` instead of Name:
>    - From **Filter** menu select `First or Last N Rows`
>        - *"Filter which rows?"*: `first`
>        - *"Filter how many rows?"*: `1`
>    - From **Rules** menu, select `Add / Modify Column Definitions`
>        - `Add Definition`, `URL`, Select Column `D`
>        - `Add Definition`, `List Identifier(s)`, Column `C`
>    - *"Type"*: `fastqsanger.gz`
>
>    ![screenshot](../../images/rules/rules_example_2_3_rules.png)
>
>    Rather than assigning column `C` as `Name`, in this example we assigned it as a `List Identifier`. This is the description of the element in the resulting dataset collection. This identifier is preserved in mapped outputs as you map tools over collections and is useful for tracking sample names, replicate numbers, conditions, etc..
>
> 5. **Enter** a name for our collection, for example the ENA study identifier as shown below, `PRJDA60709`.
>    ![screenshot](../../images/rules/rules_example_2_4_name.png)
> 6. **Click** `Upload`
>
>    This time a single new entry will appear in your history panel corresponding to all the files gathered together in a simple list named **PRJDA60709**.
{: .hands_on}


# Creating a List of Dataset Pairs

For this next example we will again use ENA data, this time corresponding to the study [PRJDB3920](https://www.ebi.ac.uk/ena/data/view/PRJDB3920) instead.

> <hands-on-title>Creating a List of Dataset Pairs</hands-on-title>
>
> 1. **Open** the Rule Builder
>
> 2. **Open** the `Rule-based` upload tab again, but this time:
>    - *"Upload data as"*: Collection(s)
>    - *"Load tabular data from"*: a `Pasted Table`
>    - **Paste** the following table
>
>      ```
>      study_accession	sample_accession	experiment_accession	fastq_ftp
>      PRJDB3920	SAMD00034150	DRX036147	https://zenodo.org/records/3263975/files/DRX036147_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036147_2.fastq.gz
>      PRJDB3920	SAMD00034150	DRX036148	https://zenodo.org/records/3263975/files/DRX036148_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036148_2.fastq.gz
>      PRJDB3920	SAMD00034150	DRX036149	https://zenodo.org/records/3263975/files/DRX036149_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036149_2.fastq.gz
>      PRJDB3920	SAMD00034150	DRX036150	https://zenodo.org/records/3263975/files/DRX036150_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036150_2.fastq.gz
>      PRJDB3920	SAMD00034150	DRX036151	https://zenodo.org/records/3263975/files/DRX036151_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036151_2.fastq.gz
>      PRJDB3920	SAMD00034153	DRX036152	https://zenodo.org/records/3263975/files/DRX036152_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036152_2.fastq.gz
>      PRJDB3920	SAMD00034152	DRX036164	https://zenodo.org/records/3263975/files/DRX036164_1.fastq.gz;https://zenodo.org/records/3263975/files/DRX036164_2.fastq.gz
>      ```
>
>
> 3. **Click** `Build` and proceed to the rule builder.
>    ![screenshot](../../images/rules/rules_example_3_2_initial_rules.png)
>
> 4. **Repeat** the last example with:
>    - From **Filter** menu select `First or Last N Rows`
>        - *"Filter which rows?"*: `first`
>        - *"Filter how many rows?"*: `1`
>    - From **Rules** menu, select `Add / Modify Column Definitions`
>        - `Add Definition`, `List Identifier(s)`, Column `C`
>    - *"Type"*: `fastqsanger.gz`
>
>    In this dataset, column `D` contains two URLs, separated by a semicolon `;`. So we cannot define that column as the URL directly, we will need to split it into two columns first.
>    We will build a regular expression that captures two "groups" from this column with two URLs - one group for everything before the `;` and one group for everything after.
> 5. From **Column**, select `Using a Regular Expression`
>    - *"From Column"*: `D`
>    - Select `Create columns matching expression groups`
>    - *"Regular Expression"*: `(.*);(.*)`
>    - *"Number of Groups"*: `2`
>    ![screenshot](../../images/rules/rules_example_3_4_regex.png)
>
>    > <comment-title>Regular expression explained</comment-title>
>    > Here `.*` means match any number of any character - so basically match anything.
>    > The parentheses around `.*` means form a "group" from whatever is matched. The `;` will match with the actual `;` in the target cells.
>    > In his example we have two such matching groups.
>    {: .comment}
>
> 6. Click **Apply** to apply the regular expression to the metadata table
>
>    Column `D` is no longer needed, so we can remove it:
> 7. From **Rules** select `Remove Column(s)`
>     - *"From Column"*: `D`
>    ![screenshot](../../images/rules/rules_example_3_7_removed_column.png)
>
>    We now have two columns containing URLs - but we can only have one URL per row. So we will split each row into two (one for the forward reads and one for the reverse):
> 8. From **Rules** select `Split Column(s)`
>    - *"Odd Row Column(s)"*: `D`
>    - *"Even Row Column(s)"*: `E`
>    - Click `Apply`
>
>    ![screenshot](../../images/rules/rules_example_3_9_columns_are_split.png)
>    For every row before the transformation with columns `ABCDE`, this will produce two rows, `ABCD` and `ABCE`. Thus you will have twice as many rows as before.
>    <br><br>
>    Now we need to inform Galaxy which of these rows are "forward" reads and which are "reverse" reads. We will do this by adding a new column again using a regular expression. Here we will match on the `_1` or `_2` in the filename at the end of the URLs.
> 9. From **Column**, select `Using a Regular Expression`
>     - *"Column"*: `D`
>     - Select `Create columns matching expression groups`
>     - *"Regular Expression"*: `.*_(\d).fastq.gz`
>     - *"Number of groups"*: `1`
>    ![screenshot](../../images/rules/rules_example_3_10_regex_paired.png)
>
>    > <comment-title>Regular expression explained</comment-title>
>    > Here `\d` means match any digit. This regular expression will use a capturing group to select only the `1` or `2` part of the URL that appears before the extension.
>    > In his example we have one matching groups ( contained in the parentheses )
>    {: .comment}
>
>    At this point, you can swap the last two columns to bring this new paired indicator column toward the beginning of the table. This is not necessary but it makes the screenshots more informative.
> 1. From **Rule** select `Swap Columns`
>     - *"Swap Column"*: `D`
>     - *"With Column"*: `E`
>
> 1. **Open** the column definitions back up (**Rules** Menu, `Add / Modify Column Definitions`)
>    - *"paired-end indicator"*: column `D`
>    - *"URL"*: column `E`
>
>    > <comment-title>Paired Indicator Column Definition</comment-title>
>    > The column selector for paired indicator is a bit less obvious and a bit more picky than the other ones we have used. The row value of the paired indicator column must be one of the following to indicate "forward" or "reverse" reads.
>    >
>    > Column  | Acceptable Indicators
>    > ------  | ---------------------
>    > Forward | `1`, `R1`, `forward`, `f`, or `F`
>    > Reverse | `2`, `R2`, `reverse`, `r`, or `R`
>    {: .comment}
>
> 12. **Supply** a name for your collection and click **Upload**
> ![screenshot](../../images/rules/rules_example_3_14_paired_identifier_set.png)
>
{: .hands_on}


topic: galaxy-interface
tutorial: workflow-automation
title: Automating Galaxy workflows using the command line
difficulty: intermediate
time_estimation: 2h
requirements_json: []
tags_json: ["workflows", "variant-analysis", "covid19"]
content_hash: ce9ab86f7e5129f86aad41290baf9d8b
last_modified: 2026-05-27T09:01:48.663979
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/workflow-automation/tutorial.html

content:
Galaxy is well-known as a web-based data analysis platform which provides a graphical interface for executing common bioinformatics tools in a reproducible manner. However, Galaxy is not just a user-friendly interface for executing one tool at a time. It provides two very useful features which allow scaling data analyses up to a high-throughput level: dataset collections and workflows.

* Dataset collections represent groups of similar datasets. This doesn't sound especially exciting, but it gets interesting when a tool is executed using a collection as input, where a single dataset would normally be created. In this case, Galaxy creates a new job for every dataset contained within the collection, and stores the tool outputs in a new collection (or collections, if there are multiple outputs) with the same size and structure as the input. This process is referred to as 'mapping over' the input collection.

* Workflows are pipelines made up of multiple Galaxy tools executed in sequence. When a workflow is executed, Galaxy schedules all the jobs which need to be run to produce the workflow outputs. They will remain scheduled (represented by the familiar grey color in the Galaxy history) until the required inputs become available. After they complete, they will make their outputs available, which allows the next set of jobs to begin.

Between them, collections and workflows make it possible to scale-up from running a single tool on a single dataset to running multiple tools on multiple datasets.

## Why use the command line?

All the functionality described so far is available through the graphical interface. So why use the command line? Let's consider a couple of scenarios:

1. You want to run some molecular dynamics simulations to perform free energy calculations for protein-ligand binding. You have a compound library of 1000 ligands and would like to run an ensemble of 10 simulations for each. You would like to have the analysis for each ligand in a separate history, with the ensembles represented as dataset collections, which means you need to invoke your workflow 1000 times - theoretically possible to do in the graphical interface, but you probably want to avoid it if possible.

2. You are conducting research on a virus which is responsible for a deadly pandemic. New genomic data from the virus is being produced constantly, and you have a variant calling workflow which will tell you if a particular sample contains new mutations. You would like to run this workflow as soon as the data appears - every day perhaps, or even every hour. This will be quite tough if you have to click a button each time yourself.

If you are encountering similar problems as these in your research with Galaxy, this tutorial is for you! We will explain how to trigger workflow execution via the command line using Planemo, and provide some tips on how to write scripts to automate the process. The result will be a bot which takes care of the the whole analysis process for you.


> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

## A short guide to Planemo

The main tool we will use in this tutorial is [Planemo](https://planemo.readthedocs.io/en/latest/readme.html), a command-line tool with a very wide range of functionality. If you ever developed a Galaxy tool, you probably encountered the `planemo test`, `planemo serve` and `planemo lint` subcommands. In this tutorial we will be using a different subcommand: `planemo run`.

> <comment-title>Note</comment-title>
> Planemo provides a more detailed tutorial on the [`planemo run` functionality here](https://planemo.readthedocs.io/en/latest/running.html). The pages on '[Best Practices for Maintaining Galaxy Workflows](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html)' and '[Test Format](https://planemo.readthedocs.io/en/latest/test_format.html)' also contain a lot of useful information.
{:.comment}

For the purposes of this tutorial, we assume you have a recent version of Planemo (0.74.4 or later) installed in a virtual environment. If you don't, please follow the [installation instructions](https://planemo.readthedocs.io/en/latest/installation.html#pip).

{% include _includes/cyoa-choices.html option1="Short Version with an existing workflow" option2="Long Version from scratch" option3="Automated runs of a workflow for SARS-CoV-2 lineage assignment" default="Short Version with an existing workflow" %}

<div class="Short-Version-with-an-existing-workflow" markdown="1">

# A quick tutorial with an existing workflow

## Get workflows and data

Workflows and data for this tutorial are hosted on [GitHub](https://github.com/usegalaxy-eu/workflow-automation-tutorial).

> <hands-on-title>Download workflow and data</hands-on-title>
> Download the workflows and data for this tutorial using `git clone`.
>
> > <code-in-title>git clone</code-in-title>
> > ```shell
> > git clone https://github.com/usegalaxy-eu/workflow-automation-tutorial.git
> > ```
> {: .code-in}
>
> Next, step into the cloned folder and take a look around.
>
> > <code-in-title></code-in-title>
> > ```shell
> > cd workflow-automation-tutorial
> > ls
> > ```
> {: .code-in}
>
> > <code-out-title>Folder contents</code-out-title>
> > ```
> > example  LICENSE  pangolin  README.md
> > ```
> {: .code-out}
>
> Of the two subfolders, `example/` contains just a toy workflow used in the following to guide you through the basics of running workflows from the command line.
>
> The `pangolin/` folder holds the workflow and other material you will use in the second part of the tutorial, in which you will be setting up an automated system for assigning batches of SARS-CoV-2 variant data to viral lineages.
>
{: .hands_on}

## Get the workflow and prepare the job file

For this section, we will use a very simple workflow consisting of two text manipulation tools chained together.

> <hands-on-title>Step into and explore the example folder</hands-on-title>
>
> > > <code-in-title></code-in-title>
> > > ```shell
> > > cd example
> > > ls
> > > ```
> > {: .code-in}
> >
> > > <code-out-title>Folder contents</code-out-title>
> > > ```
> > > tutorial.ga
> > > ```
> > {: .code-out}
> {: .code-2col}
>
{: .hands_on}

The `tutorial.ga` file defines the workflow in JSON format; if we are confident we have a functional workflow, we don't need to worry about its contents or modify it. However, we need a second file, the so-called 'job file', which specifies the particular dataset and parameter inputs which should be used to execute the workflow. We can create a template for this file using the `planemo workflow_job_init` subcommand.

> <hands-on-title>Creating the job file</hands-on-title>
>
> 1. Run the `planemo workflow_job_init` subcommand.
>
>    > <code-in-title>workflow_job_init</code-in-title>
>    > ```shell
>    > planemo workflow_job_init tutorial.ga -o tutorial-init-job.yml
>    > # Now let's view the contents
>    > cat tutorial-init-job.yml
>    > ```
>    > The `planemo workflow_job_init` command identifies the inputs of the workflow provided and creates a template job file with placeholder values for each.
>    {:.code-in}
>
>    > <code-out-title>File contents</code-out-title>
>    > ```yaml
>    > Dataset 1:
>    >   class: File
>    >   path: todo_test_data_path.ext
>    > Dataset 2:
>    >   class: File
>    >   path: todo_test_data_path.ext
>    > Number of lines: todo_param_value
>    > ```
>    >
>    > The job file contains three inputs: two dataset inputs and one integer (parameter input).
>    {:.code-out}
>
> 2. Create two files which can be used as inputs:
>
>    > <code-in-title>Creating the input files</code-in-title>
>    > ```shell
>    > printf "hello\nworld" > dataset1.txt
>    > printf "hello\nuniverse!" > dataset2.txt
>    > ls
>    > ```
>    >
>    {:.code-in}
>
>    > <code-out-title></code-out-title>
>    > ```shell
>    > dataset1.txt  dataset2.txt  tutorial.ga  tutorial-init-job.yml
>    > ```
>    {:.code-out}
>
> 3. Replace the placeholder values in the job file, so that it looks like the following:
>
>    ```yaml
>    Dataset 1:
>      class: File
>      path: dataset1.txt
>    Dataset 2:
>      class: File
>      path: dataset2.txt
>    Number of lines: 3
>    ```
>
> Now we are ready to execute the workflow with our chosen parameters!
{: .hands_on}

## Running the workflow

Now we have a simple workflow, we can run it using `planemo run`. At this point you need to choose a Galaxy server on which you want the workflow to run. One of the big public servers would be a possible choice. You could also use a local Galaxy instance. Either way, once you've chosen a server, the next step is to get your API key.

{% snippet faqs/galaxy/preferences_admin_api_key.md %}

> <hands-on-title>Running our workflow</hands-on-title>
>
> 1. Run the `planemo run` subcommand.
>
>    > <code-in-title>planemo run</code-in-title>
>    > ```shell
>    > planemo run tutorial.ga tutorial-init-job.yml --galaxy_url <SERVER_URL> --galaxy_user_key <YOUR_API_KEY> --history_name "Test Planemo WF" --tags "planemo-tutorial"
>    > ```
>    {:.code-in}
>
>
> 2. Navigate to the web browser - you should be able to see a new history has been created with the chosen name and tag.<br>One potential disadvantage of the previous command is that it waits until the invoked workflow has fully completed. For our very small example, this doesn't matter, but for a workflow which takes hours or days to finish, it might be undesirable. Fortunately, `planemo run` provides a `--no_wait` flag which exits as soon as the workflow has been successfully scheduled.
>
> 3. Run the `planemo run` subcommand with the `--no_wait` flag.
>    > <code-in-title>planemo run</code-in-title>
>    > ```shell
>    > planemo run tutorial.ga tutorial-init-job.yml --galaxy_url <SERVER_URL> --galaxy_user_key <YOUR_API_KEY> --history_name "Test Planemo WF with no_wait" --tags "planemo-tutorial" --no_wait
>    > ```
>    > This time you should see that the `planemo run` command exits as soon as the two datasets have been uploaded and the workflow has been scheduled.
>    {:.code-in}
{: .hands_on}

</div>

<div class="Long-Version-from-scratch" markdown="1">

# A longer tutorial "From scratch" to CLI

## Get the toy data

> <hands-on-title>Download some data</hands-on-title>
> For this tutorial, we will use a training dataset: [M. tuberculosis bioinformatics training data](https://zenodo.org/records/3960260)
> Download data for this tutorial using `curl` or `wget`.
>
> > <code-in-title>Bash</code-in-title>
> > ```shell
> > wget https://zenodo.org/record/3960260/files/018-1_1.fastq.gz
> > wget https://zenodo.org/record/3960260/files/018-1_2.fastq.gz
> > ```
> {: .code-in}
{: .hands_on}

## Creating a new workflow

> <hands-on-title>Find a Galaxy instance</hands-on-title>
> Logging into your favorite Galaxy instance (Ex: `https://usegalaxy.eu/` or `https://usegalaxy.fr/`)
{: .hands_on}

> <hands-on-title>Create the workflow into Galaxy.</hands-on-title>
> 1. Go to the workflow panel {% icon galaxy-workflows-activity %}
> 2. Create a new workflow: `Fastq cleaning and check`
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
> 3. Add from Inputs an Input Dataset Collection
>    - **Label**: `Fastq Inputs`
>    - **Format(s)**: `fastqsanger.gz`
>
> 4. Add {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %}
> 5. Add {% tool [fastp](toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3) %}
> 6. Connect the **Fastq Inputs** to **Falco** and **fastp**
> 7. Add {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy4) %}
>    - **1: Results** -> **Which tool was used to generate logs?**: `FastQC`
>    - **+ Insert Results**
>    - **2: Results** -> **Which tool was used to generate logs?**: `fastp`
> 8. Connect:
>    - **Falco** / **text_file (txt)** to **MultiQC Result 1**
>    - **fastp** / **report_json (json)** to **MultiQC Result 2**
> 9. Select the desired outputs:
>    - In **fastp**, tick **out1 (input)**
>    - In **MultiQC**, tick **html_report (html)**
> 10. Rename the desired outputs:
>    - In **fastp**, right panel, scroll down -> Configure Output: 'out1' -> **Rename dataset**: `#{in1 | basename}_cleaned`
>    - In **MultiQC**, right panel, scroll down -> Configure Output: 'html_report' -> **Rename dataset**: `multiqc_report`
>
> ![This image show a preview of our workflow with tools as box and noodle as connectors).](./images/Workflow-Fastq_cleaning_and_check.png "Preview of our workflow")
{: .hands_on}

## Create a job parameter template

> <comment-title></comment-title>
> Those following steps could be processed once when you discover your new workflow. It wouldn't be necessary at every update for example.
{: .comment}

> <hands-on-title>Download your workflow</hands-on-title>
> {% snippet faqs/galaxy/workflows_download.md %}
{: .hands_on}

> <hands-on-title>Generate a job parameter.</hands-on-title>
> We will use **Planemo** to generate a template for our job parameters before adaptating it.
>
> There are many way to install planemo: <https://planemo.readthedocs.io/en/latest/installation.html>
>
> > <tip-title>What about using Docker?</tip-title>
> >
> > An easy way to use planemo without "installation" for this turorial could be the Docker image provided by BioContainer
> > > <code-in-title>Bash</code-in-title>
> > > ```bash
> > > docker run --rm -v $PWD:/data/ -w /data quay.io/biocontainers/planemo:0.75.31--pyhdfd78af_0 planemo --version
> > > ```
> > {: .code-in}
> {: .tip}
>
> > <code-in-title>Bash</code-in-title>
> > ```bash
> > planemo workflow_job_init Galaxy-Workflow-Fastq_cleaning_and_check.ga -o Galaxy-Workflow-Fastq_cleaning_and_check-job.xml
> >
> > cat Galaxy-Workflow-Fastq_cleaning_and_check-job.xml
> > ```
> {: .code-in}
>
> > <code-out-title>Galaxy-Workflow-Fastq_cleaning_and_check-job.xml</code-out-title>
> > ```yaml
> > Fastq Inputs:
> >   class: Collection
> >   collection_type: list
> >   elements:
> >   - class: File
> >     identifier: todo_element_name
> >     path: todo_test_data_path.ext
> > ```
> {: .code-out}
{: .hands_on}

> <hands-on-title>Adapt the job parameter</hands-on-title>
> Modify `Galaxy-Workflow-Fastq_cleaning_and_check-job.xml` to look like the following:
>
> ```yaml
> Fastq Inputs:
>   class: Collection
>   collection_type: list
>   elements:
>   - class: File
>     identifier: 018-1_1
>     path: 018-1_1.fastq.gz
>   - class: File
>     identifier: 018-1_2
>     path: 018-1_2.fastq.gz
> ```
{: .hands_on}

## Running the workflow on a remote external Galaxy instance

Now we have a simple workflow, we can run it using `planemo run`. At this point you need to choose a Galaxy server on which you want the workflow to run. One of the big public servers would be a possible choice. You could also use a local Galaxy instance. Either way, once you've chosen a server, the next step is to get your API key.

{% snippet faqs/galaxy/preferences_admin_api_key.md %}

> <hands-on-title>Running our workflow</hands-on-title>
> You can either provide the workflow file `.ga` or a workflow ID within a Galaxy instance.
> Let's do it simple and "ecologic" with a workflow ID
>
> 1. Get the ID of your workflow just created:
>
>    {% snippet faqs/galaxy/workflows_get_id.md %}
>
> 2. Run the `planemo run` subcommand.
>
>    > <code-in-title>planemo run</code-in-title>
>    > ```shell
>    > planemo run <WORKFLOW_ID> Galaxy-Workflow-Fastq_cleaning_and_check-job.xml --galaxy_url <SERVER_URL> --galaxy_user_key <YOUR_API_KEY> --outdir . --download_outputs
>    > ```
>    {:.code-in}
>
>    > <comment-title>Checking within Galaxy</comment-title>
>    > You may want to check the ongoing process or the results within the Galaxy instance interface.
>    {: .comment}
>
>    One potential disadvantage of the previous command is that it waits until the invoked workflow has fully completed. For our very small example, this doesn't matter, but for a workflow which takes hours or days to finish, it might be undesirable. Fortunately, `planemo run` provides a `--no_wait` flag which exits as soon as the workflow has been successfully scheduled.
> 3. Enjoy your result files
>
>    Contents of your directory:
>    ```
>    [...]
>    018-1_1.fastq.gz
>    018-1_2.fastq.gz
>    Galaxy-Workflow-Fastq_cleaning_and_check-job.xml
>    018-1_1.fastq_cleaned__b09ee414-9968-4e46-9415-535a5ddf2e0c.fastqsanger.gz
>    018-1_2.fastq_cleaned__b61fd0af-6353-4222-a031-04af0b1ee993.fastqsanger.gz
>    multiqc_report__82059f85-70ca-4e66-804a-beae024e108c.html
>    ```
>
>    ![This image show the MultiQC webpage).](./images/Workflow-Fastq_cleaning_and_check-multiqc.png "MultiQC result webpage")
>
{: .hands_on}

</div>

## Going further

### Using Planemo profiles

Planemo provides a useful profile feature which can help simplify long commands. The idea is that flags which need to be used multiple times in different invocations can be combined together and run as a single profile. Let's see how this works below.

> <hands-on-title>Creating and using Planemo profiles</hands-on-title>
>
> 1. Create a Planemo profile with the following command:
>    > <code-in-title>planemo run</code-in-title>
>    > ```shell
>    > planemo profile_create planemo-tutorial --galaxy_url <SERVER_URL> --galaxy_user_key <YOUR_API_KEY>
>    > ```
>    {:.code-in}
>
>    > <code-out-title>Terminal</code-out-title>
>    > ```shell
>    > Profile [planemo-tutorial] created.
>    > ```
>    > You can view and delete existing profiles using the `profile_list` and `profile_delete` subcommands.
>    {:.code-out}
> 2. Now we can run our workflow yet again using the profile we have created:
>    > <code-in-title>planemo run</code-in-title>
>    > ```shell
>    > planemo run <WORKFLOW ID> tutorial-init-job.yml --profile planemo-tutorial --history_name "Test Planemo WF with profile" --tags "planemo-tutorial"
>    > ```
>    > This invokes the workflow with all the parameters specified in the profile `planemo-tutorial`.
>    {:.code-in}
{: .hands_on}

### Using Galaxy workflow and dataset IDs

If you execute the same workflow twice. If you inspect your histories and workflows through the Galaxy web interface, you will see that a new workflow was created on the server for each invocation, and both `Dataset 1` and `Dataset 2` were uploaded twice. This is undesirable - we are creating a lot of clutter and the uploads are creating additional unnecessary work for the Galaxy server.

Every object associated with Galaxy, including workflows, datasets and dataset collections, have hexadecimal IDs associated with them, which look something like `6b15dfc0393f172c`. Once the datasets and workflows we need have been uploaded to Galaxy once, we can use these IDs in our subsequent workflow invocations.

> <hands-on-title>Running our workflow using dataset and workflow IDs</hands-on-title>
>
> 1. Navigate to one of the histories to get dataset IDs for the input datasets. For each one:
>    1. Click on the {% icon galaxy-info %} *View details* icon on the dataset in the history.
>    2. Under the heading `Dataset Information`, find the row `History Content API ID` and copy the hexadecimal ID next to it.
> 2. Modify `tutorial-init-job.yml` to look like the following:
>
>    ```yaml
>    Dataset 1:
>      class: File
>      # path: dataset1.txt
>      galaxy_id: <ID OF DATASET 1>
>    Dataset 2:
>      class: File
>      # path: dataset2.txt
>      galaxy_id: <ID OF DATASET 2>
>    Number of lines: 3
>    ```
>
> 3. Now we need to get the workflow ID:
>
>    {% snippet faqs/galaxy/workflows_get_id.md %}
>
> 4. Run the `planemo run` subcommand using the new workflow ID.
>
>    > <code-in-title>planemo run</code-in-title>
>    > ```shell
>    > planemo run <WORKFLOW ID> tutorial-init-job.yml --galaxy_url <SERVER_URL> --galaxy_user_key <YOUR_API_KEY> --history_name "Test Planemo WF with Planemo" --tags "planemo-tutorial" --no_wait
>    > ```
>    {:.code-in}
{: .hands_on}

### Ensuring Workflows meet Best Practices

{% snippet faqs/galaxy/workflows_best_practices.md %}

<div class="Automated-runs-of-a-workflow-for-SARS-CoV-2-lineage-assignment" markdown="1">

# Automated runs of a workflow for SARS-CoV-2 lineage assignment

It's now time to apply your newly acquired knowledge of workflow execution with Planemo to a relevant scientific problem.

## Scientific background

The SARS-CoV-2 pandemic has been accompanied by unprecedented world-wide sequencing efforts. One of the accepted goals behind sequencing hundreds of thousands of individual viral isolates is to monitor the evolution and spreading of viral lineages in as close as real time as possible. Viral lineages are characterized by defining patterns of mutations that make them different from each other and from the original virus that started the pandemic at the beginning of 2020. Examples of viral lineages are B.1.1.7, first observed in the UK in the fall of 2020 and now termed *variant of concern (VOC) alpha* according to the WHO's classification system, and B.1.617.2, first seen in India at the beginning of 2021 and now recognized as *VOC delta*.

[Pangolin](https://cov-lineages.org/pangolin.html) is a widely used tool for assigning newly sequenced viral isolates to established viral lineages, and in this final section of this tutorial you are going to run a workflow that:

1. takes a collection of variant datasets in the *variant call format* VCF,

   where you can think of a collection as representing a batch of freshly sequenced viral isolates with each of its VCF datasets listing the nucleotide differences between one sample and the sequence of an original SARS-CoV-2 reference isolate

2. reconstructs the viral genome sequence of each sample by incorporating its variants into the reference isolate's sequence

3. uses Pangolin to classify the resulting collection of genome sequences in FASTA format and to create a report of lineage assignments for all samples.

Just like in a real world situation, you will receive VCF files for several batches of samples and you will face the challenge of uploading the files from each batch as a collection into Galaxy and of triggering a run of the workflow for each of them.

## Setting up the bot

Unlike for the previous toy example you will not get complete step-by-step instructions, but you are supposed to try yourself to transfer the knowledge from part 1 to this new, more complex task.

Every step along the way comes with solutions, which you can expand at any time, but you're encouraged to give each problem some thought first.

As a very first step, however, let's look at how the material for this part is arranged.

> <hands-on-title>Step into and explore the pangolin folder</hands-on-title>
>
> > <code-in-title></code-in-title>
> > ```shell
> > cd ../pangolin
> > ls
> > ```
> {: .code-in}
>
> > <code-out-title>Folder contents</code-out-title>
> > ```
> > data  solutions  vcf2lineage.ga
> > ```
> {: .code-out}
>
{: .hands_on}

The file `vcf2lineage.ga` defines the workflow just described, while the `data/` folder holds the batches of VCF files we would, ultimately, like to run the workflow on.

Now, as a start, let's get the workflow running on the first batch of files in the `data/batch1/` subfolder.

> <hands-on-title>An initial workflow run</hands-on-title>
>
> 1. Create a template job file for the `vcf2lineage.ga` workflow.
>    > <details-title>Solution</details-title>
>    > We need to run `workflow_job_init` on the `vcf2lineage.ga` workflow.
>    > > <code-in-title>workflow_job_init</code-in-title>
>    > > ```shell
>    > > planemo workflow_job_init vcf2lineage.ga -o vcf2lineage-job.yml
>    > > # Now let's view the contents
>    > > cat vcf2lineage-job.yml
>    > > ```
>    > {: .code-in}
>    >
>    > > <code-out-title>File contents</code-out-title>
>    > > ```yaml
>    > > Reference genome:
>    > >   class: File
>    > >   path: todo_test_data_path.ext
>    > > Variant calls:
>    > >   class: Collection
>    > >   collection_type: list
>    > >   elements:
>    > >   - class: File
>    > >     identifier: todo_element_name
>    > >     path: todo_test_data_path.ext
>    > > min-AF for consensus variant: todo_param_value
>    > > ```
>    > {: .code-out}
>    >
>    > The job file contains three inputs: the `Reference genome`, a collection of VCF files (`Variant calls`) and a float parameter for the minimum allele frequency (`min-AF for consensus variant`). Later, we will need to specify each element of the collection under `elements` - currently there is just a single placeholder element.
>    {: .details}
>
> 2. Replace the placeholder values: `Reference genome` should point to `data/NC_045512.2_reference_sequence.fasta`, `Variant calls` should contain all the VCF files in `data/batch1`, and `min-AF for consensus variant` should be set to `0.7`.
>
>    > <tip-title>Hint: Creating the `Variant calls` collection</tip-title>
>    > This is the trickiest part and you should consider writing a script to solve it. The entry in the job file should look something like this at the end:
>    >
>    > ```yaml
>    > Variant calls:
>    >   class: Collection
>    >   collection_type: list
>    >   elements:
>    >   - class: File
>    >     identifier: ERR5879218
>    >     path: data/batch1/ERR5879218.vcf
>    >   - class: File
>    >     identifier: ERR5879219
>    >     path: data/batch1/ERR5879219.vcf
>    >   - class: File
>    >     ...
>    > ```
>    > There should be one entry under `elements` for each of the 312 files under `data/batch1`.
>    {: .tip}
>
>    > <details-title>Solution</details-title>
>    > Adding the `Reference genome` dataset and `min-AF for consensus variant` parameter is straightforward. Modify the `vcf2lineage-job.yml` file and save it:
>    >
>    > ```yaml
>    > Reference genome:
>    >   class: File
>    >   path: data/NC_045512.2_reference_sequence.fasta
>    > Variant calls:
>    >   class: Collection
>    >   collection_type: list
>    >   elements:
>    >   - class: File
>    >     identifier: todo_element_name
>    >     path: todo_test_data_path.ext
>    > min-AF for consensus variant: 0.7
>    > ```
>    >
>    > To add entries for every element of the `Variant calling ` collection we should write a script. There are many possible solutions; here is an example:
>    > ```python
>    > import sys
>    > from glob import glob
>    > from pathlib import Path
>    >
>    > import yaml
>    >
>    > job_file_path = sys.argv[1]
>    > batch_directory = sys.argv[2]
>    >
>    > with open(job_file_path) as f:
>    >     job = yaml.load(f, Loader=yaml.CLoader)
>    >
>    > vcf_paths = glob(f'{batch_directory}/*.vcf')
>    > elements = [{'class': 'File', 'identifier': Path(vcf_path).stem, 'path': vcf_path} for vcf_path in vcf_paths]
>    > job['Variant calls']['elements'] = elements
>    >
>    > with open(job_file_path, 'w') as f:
>    >     yaml.dump(job, f)
>    > ```
>    >
>    > Save this as `create_job_file.py` and run it with `python create_job_file.py vcf2lineage-job.yml data/batch1`. This has the effect of updating the `vcf2lineage-job.yml` with all the VCF files in the `data/batch1` directory.
>    {: .details}
>
> 3. Now that we have a complete job file, let's run the workflow.
>    > <details-title>Solution</details-title>
>    > > <code-in-title>workflow_job_init</code-in-title>
>    > > ```shell
>    > > planemo run vcf2lineage.ga vcf2lineage-job.yml --profile planemo-tutorial --history_name "vcf2lineage test"
>    > > ```
>    > {: .code-in}
>    > You should see the new invocation in the Galaxy interface.
>    {: .details}
{: .hands_on}

We have now performed a test invocation of the vcf2lineage workflow. It was already more challenging than the first example; for the first time, we needed to resort to writing a script to achieve a task, in this case the construction of the job file.

The next step is to automate this process so we can run the workflow on each of the 10 `batch*/` directories in the `data/` folder. We can imagine that these are newly produced data released at regular intervals, which need to be analysed.

> <hands-on-title>Automating vcf2lineage execution</hands-on-title>
>
> 1. If we want to execute the workflow multiple times, we will once again encounter the issue that the datasets and workflow will be reuploaded each time. To avoid this, let's obtain the dataset ID for the `Reference genome` (which stays the same for each invocation) and the workflow ID for the `vcf2lineage.ga` workflow.
>
> 2. Now let's create a template job file `vcf2lineage-job-template.yml` which we can modify at each invocation as necessary. We can start with the output of `workflow_job_init` and add the `Reference genome` dataset ID and set `min-AF for consensus variant` to `0.7` again.
>
>    > <details-title>Solution</details-title>
>    > We need to run `workflow_job_init` on the `vcf2lineage.ga` workflow.
>    > > <code-in-title>Original output of workflow_job_init</code-in-title>
>    > > ```yaml
>    > > Reference genome:
>    > >   class: File
>    > >   path: todo_test_data_path.ext
>    > > Variant calls:
>    > >   class: Collection
>    > >   collection_type: list
>    > >   elements:
>    > >   - class: File
>    > >     identifier: todo_element_name
>    > >     path: todo_test_data_path.ext
>    > > min-AF for consensus variant: todo_param_value
>    > > ```
>    > {:.code-in}
>    >
>    > > <code-out-title>Prepared `vcf2lineage-job-template.yml` file contents</code-out-title>
>    > > ```yaml
>    > > Reference genome:
>    > >   class: File
>    > >   galaxy_id: '1234567890abcdef' # replace with the ID you got from your own server
>    > > Variant calls:
>    > >   class: Collection
>    > >   collection_type: list
>    > >   elements:
>    > >   - class: File
>    > >     identifier: todo_element_name
>    > >     path: todo_test_data_path.ext
>    > > min-AF for consensus variant: 0.7
>    > > ```
>    > {:.code-out}
>    > We can copy and modify this new `vcf2lineage-job-template.yml` file iteratively to invoke the workflow on each of the data batches.
>    {: .details}
>
> 3. Write a shell script to iterate over all the batches, create a job file and invoke with `planemo run`. After execution, move the processed batch to `data/complete`.
>
>    > <details-title>Solution</details-title>
>    > Once again, there is no single exact solution. Here is one possibility:
>    >
>    > ```shell
>    > for batch in `ls -d data/batch*`; do
>    >     batch_name=`basename $batch`
>    >     cp vcf2lineage-job-template.yml vcf2lineage-${batch_name}-job.yml
>    >     python create_job_file.py vcf2lineage-${batch_name}-job.yml $batch
>    >     # replace with your own workflow ID below
>    >     planemo run f4b02af7e642e75b vcf2lineage-${batch_name}-job.yml --profile planemo-tutorial
>    >     sleep 300
>    >     mv $batch data/complete/
>    > done
>    > ```
>    >
>    > Save this as `run_vcf2lineage.sh`.
>    {: .details}
>
> 4. Run your script. Do you notice any issues? What could be improved?
>
>    > <tip-title>Upload problems</tip-title>
>    > Uploads can cause several issues:
>    > * The default planemo behavior is to upload datasets one at a time, which ensures the server cannot be overloaded, but is slow. To upload all datasets simultaneously, you could use the `--simultaneous_uploads` flag.
>    > * If one of the upload jobs fails, the workflow will by default not be invoked - Planemo will just upload the datasets and finish. To override this behavior and start a workflow even if one or more uploads fail, you can use the `--no_check_uploads_ok` flag.
>    > * For a more complex solution to the issue of failed uploads, see [this script](https://github.com/usegalaxy-eu/ena-cog-uk-wfs/blob/main/bioblend-scripts/ftp_links_to_yaml.py) from the `ena-cog-uk-wfs` repo.
>    {: .tip}
{: .hands_on}


## More advanced solutions

This was a very basic example of a workflow. Perhaps for your case, you need a more customized solution.

For example, it might be the case that you want to run multiple different workflows, one after another. In this case you would need to implement some sort of check to verify if one invocation had finished, before beginning with the next one. Planemo will probably not be enough for a task like this; you will need to resort to using the lower-level [BioBlend](https://bioblend.readthedocs.io/) library to interact directly with the Galaxy API.

The [Galaxy SARS-CoV-2 genome surveillance bot](https://github.com/usegalaxy-eu/ena-cog-uk-wfs/) provides an example of a more advanced customized workflow execution solution, combining Planemo commands, custom BioBlend scripts and bash scripts, which then get run automatically via continuous integration (CI) on a Jenkins server.

</div>

# Conclusion


You should now have a better idea about how to run Galaxy workflows from the command line and how to apply the ideas you have learnt to your own project.

tools_json:
["Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "fastp", "toolshed.g2.bx.psu.edu/repos/iuc/fastp/fastp/1.0.1+galaxy3", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy4"]


topic: galaxy-interface
tutorial: rstudio
title: RStudio in Galaxy
difficulty: intermediate
time_estimation: 3H
tools_json: ["RStudio", "interactive_tool_rstudio"]
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 19350ed40f0cdafd6e7c270ec5766c14
last_modified: 2026-05-27T09:01:48.663070
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/rstudio/tutorial.html

content:
This tutorial will introduce you to how to run RStudio in Galaxy

{% include topics/data-science/tutorials/r-basics/tutorial_origin.md %}

[RStudio](https://www.rstudio.com/products/RStudio/) is an [Integrated Development Environment (IDE)](https://en.wikipedia.org/wiki/Integrated_development_environment). Like most IDEs, it provides a graphical interface to R, making it more user-friendly, and providing dozens of useful features. We will introduce additional benefits of using RStudio as you cover the lessons. In this case, we are specifically using [RStudio Server](https://www.rstudio.com/products/RStudio/#Server), a version of RStudio that can be accessed in your web browser. RStudio Server has the same features as the Desktop version of RStudio you could download as standalone software.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# RStudio

Opening up {% tool [RStudio](interactive_tool_rstudio) %} in Galaxy is easy:

{% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}

You should now be looking at a page with the RStudio interface:

![rstudio default session](../../images/rstudio/rstudio_session_default.png)

## Creating your first R script

Now that we are ready to start exploring R, we will want to keep a record of the commands we are using. To do this we can create an R script.

> <hands-on-title>Create a R script</hands-on-title>
>
> 1. Click the **File** menu
> 2. Select **New File**
> 3. Click on **R Script**
{: .hands_on}

A new panel appears on the top left. Before we go any further, you should save your script.

> <hands-on-title>Save a R script</hands-on-title>
>
> 1. Click the {% icon galaxy-save %} icon (**Save current document**) in the bar above the first line in the script editor
>
>    Alternatively, you can also:
>    - Click the **File** menu and select **Save**
>    - Type <kbd>CTRL</kbd>+<kbd>S</kbd> (<kbd>CMD</kbd>+<kbd>S</kbd> on OSX)
>
> 2. In the **Save File** window that opens, name your file `genomics_r_basics`
{: .hands_on}

The new script `genomics_r_basics.R` should appear under **Files** in the bottom right panel. By convention, R scripts end with the file extension `.R`.

## Overview and customization of the RStudio layout

Here are the major windows (or panels) of the RStudio environment:

![rstudio default session](../../images/rstudio/rstudio_session_4pane_layout.png)

- **Source**: This panel is where you will write/view R scripts

    Some outputs (such as if you view a dataset using `View()`) will appear as a tab here.

- **Console/Terminal**: This is actually where you see the execution of commands

    This is the same display you would see if you were using R at the command line without RStudio. You can work interactively (i.e. enter R commands here), but for the most part we will run a script (or lines in a script) in the source pane and watch their execution and output here.

- **Environment/History**: RStudio will show here you what datasets and objects (variables) you have created and which are defined in memory.

    You can also see some properties of objects/datasets such as their type and dimensions. The **History** tab contains a history of the R commands you've executed in R.

- **Files/Plots/Packages/Help**: This multipurpose panel will show you the contents of directories on your computer

    - **Files**: You can also use this tab to navigate and set the working directory
    - **Plots**: This tab will show the output of any plots generated
    - **Package**: In this tab, you will see what packages are actively loaded, or you can attach installed packages
    - **Help**: It will display help files for R functions and packages.


All of the panels in RStudio have configuration options. For example, you can minimize/maximize a panel, or by moving your mouse in the space between panels you can resize as needed. The most important customization options for panel layout are in the **View** menu. Other options such as font sizes, colors/themes, and more are in the **Tools** menu under **Global Options**.

> <comment-title>Working with R at the terminal</comment-title>
> Although we won't be working with R at the terminal, there are lots of reasons to.
>
> For example, once you have written an RScript, you can run it at any Linux or Windows terminal without the need to start up RStudio. We don't want you to get confused - RStudio runs R, but R is not RStudio.
>
> For more on running an R Script at the terminal see the dedicated [Software Carpentry lesson](https://swcarpentry.github.io/r-novice-inflammation/05-cmdline.html).
{: .comment}

## How to call functions in R, without needing to master them?

A function in R (or any computing language) is a short program that takes some input and returns some output.

> <hands-on-title>Calling a function in R</hands-on-title>
>
> 1. Type `date()` in the **Console** panel
> 2. Type <kbd>Enter</kbd>
> 2. Check what is displayed in the **Console** panel
>
{: .hands_on}

You should obtain something like:

```R
[1] "Tue Mar 26 15:12:24 2019"
```

> <comment-title>Display of function call in the tutorial</comment-title>
> Now in the tutorial, we will display the function call like this:
>
> ```R
> > date()
> [1] "Tue Mar 26 15:12:24 2019"
> ```
{: .comment}

The other way to execute these functions is to use the script we just created and then keep track of the functions.

> <hands-on-title>Running a function via a script</hands-on-title>
>
> 1. Type `date()` in the **Script** panel
> 2. Click on the **Run the current line or selection** or type <kbd>CTRL</kbd>+<kbd>Enter</kbd> (or <kbd>CMD</kbd>+<kbd>Enter</kbd>)
{: .hands_on}

You should see in the **Console** panel the same as when we run the function directly via the console.

We would like now to keep information about this function

> <hands-on-title>Comment in a script</hands-on-title>
>
> 1. Write on the line before `date()` a comment:
>
>    ```R
>    # Gives the current date
>    ```
>
>    Any lines starting with a `#` will be treated as comments, and not executed
>
> 2. Select both lines
> 3. Execute them
> 4. Check the output
{: .hands_on}

You should obtain something like:

```R
> # Gives the current date
> date()
[1] "Tue Mar 26 15:12:24 2019"
```

The comment line is displayed in the console but not executed.

> <question-title>What do these functions do?</question-title>
>
> Try the following functions by writing them in your script. See if you can
> guess what they do, and make sure to add comments to your script about your
> assumed purpose.
> 1. `dir()`
> 2. `sessionInfo()`
> 3. `Sys.time()`
>
> > <solution-title></solution-title>
> >
> > 1. `dir()` lists files in the working directory
> > 2. `sessionInfo()` gives the version of R and additional info including on attached packages
> > 3. `Sys.time()` gives the current time
> >
> {: .solution}
{: .question}

> <warning-title>Commands are case sensitive!</warning-title>
> In R, the commands are case sensitive. So be careful when you type them.
{: .warning}

You have hopefully noticed a pattern - an R function has three key properties:
1. A name (e.g. `dir`, `getwd`) first
2. A pair of `()` after the name
3. 0 or more arguments inside the parentheses

   An argument may be a specific input for your function and/or may modify the function's behavior. For example, the function `round()` will round a number with a decimal:

   > <code-in-title></code-in-title>
   > ```R
   > # This will round a number to the nearest integer
   > > round(3.14)
   > [1] 3
   > ```
   {: .code_in}

## Getting help

What if you wanted to round to one significant digit, `round()` can do this, but you may first need to read the help to find out how.

To see the help you need enter a `?` in front of the function name. The **Help** tab (in the bottom-right panel) will show you information.

> <hands-on-title>Get help</hands-on-title>
>
> 1. Add a `?` in front of the function name to see the help
>
>    ```R
>    > ?round()
>    ```
>
> 2. Check the **Help** tab
>
{: .hands_on}

In R, this help is sometimes also called a "vignette". Often there is too much information. You will slowly learn how to read and make sense of them:

1. Checking the **Usage** or **Examples** headings is often a good place to look first
2. Under **Arguments**, we can also see what arguments we can pass to this function to modify its behavior

We can also see the arguments of a function without opening its help.

> <hands-on-title>Get the function arguments</hands-on-title>
>
> 1. Type `args()` to see the arguments of `round` function
>
>    ```R
>    > args(round)
>    function (x, digits = 0)
>    NULL
>    ```
>
{: .hands_on}

`round()` takes two arguments:

1. `x`: the number to be rounded
2. `digits`: integer indicating the number of decimal places to be used

    The `=` sign indicates that a default (in this case 0) is already set.

Since `x` is not set, `round()` requires we provide it, in contrast to `digits` where R will use the default value 0 unless you explicitly provide a different value.

We can explicitly set the digits parameter when we call the function.

> <hands-on-title>Call a function with several parameters</hands-on-title>
>
> 1. Call `round` with 2 arguments
>    - *x*: `3.14159`
>    - *digits*: `2`
>
>    ```R
>    > round(3.14159, digits = 2)
>    [1] 3.14
>    ```
>
> 2. Call `round` with 2 arguments
>    - 3.14159
>    - 2
>
>    ```R
>    > round(3.14159, 2)
>    [1] 3.14
>    ```
{: .hands_on}

R accepts what we call "positional arguments". If you pass a function arguments separated by commas, R assumes that they are in the order you saw when we used `args()`. In the case below that means that `x` is 3.14159 and `digits` is 2.

Finally, what if you are using `?` to get help for a function in a package not installed on your system, such as when you are running a script which has dependencies?

> <hands-on-title>Get help for a missing function</hands-on-title>
>
> 1. Ask help for `geom_point()`
> 2. Check the generated error
>
>
>    ```R
>    > ?geom_point()
>    Error in .helpForCall(topicExpr, parent.frame()) :
>      no methods for ‘geom_point’ and no documentation for it as a function
>    ```
>
> 3. Type `??geom_point()`
> 4. Check the **Help** tab
{: .hands_on}

Using the two question marks (here `??geom_point()`), R returns results from a search of the documentation for packages you have installed on your computer in the **Help** tab.

Finally, if you think there should be a function, for example a statistical test, but you aren't sure what it is called in R, or what functions may be available.

> <hands-on-title>Search for a function</hands-on-title>
>
> 1. Type `help.search('chi-Squared test')`
> 2. Check the **Help** panel
{: .hands_on}

A list of potential interesting function related to "chi-Squared test" are listed. You can click on one of them to see the help of it. Remember to put your search query in quotes inside the function's parentheses.

> <question-title>Search for R functions</question-title>
>
> Search the R functions for the following statistical
> functions
>
> 1. Student t-test
> 2. mixed linear model
>
> > <solution-title></solution-title>
> >
> > While your search results may return several tests, we list a few you might
> > find:
> > 1. Student's t-Test: `stats::t.test`
> > 2. mixed linear model: `stats::lm.glm`
> >
> {: .solution}
{: .question}

We will not discuss now, where to look for the libraries and packages that contain functions you want to use. For now, be aware that two important ones are:
1. [CRAN](https://cran.r-project.org/): the main repository for R
2. [Bioconductor](http://bioconductor.org/): a popular repository for bioinformatics-related R packages

## RStudio contextual help

Here is one last bonus we will mention about RStudio. It's difficult to remember all of the arguments and definitions associated with a given function.

> <hands-on-title>Search for a function</hands-on-title>
>
> 1. Type `lm` in the **Script** panel
> 2. Hit <kbd>Tab</kbd>
>
>    RStudio displays functions and associated help
>
>    ![rstudio contextual help](../../images/rstudio/studio_contexthelp1.png)
>
> 3. Select `lm` function using the arrows
> 4. Hit <kbd>Enter</kbd>
> 4. Hit <kbd>Tab</kbd> again inside the parentheses
>
>    RStudio shows the function's arguments and provide additional help for each of these arguments:
>
>    ![rstudio contextual help](../../images/rstudio/studio_contexthelp2.png)
>
{: .hands_on}


## Stopping RStudio

RStudio will keep running until you stop it, so you can always come back to your analysis later. However, once you are finished with your analysis,
you should save the work you did within RStudio by exporting any files you created back to your Galaxy history, as well as the log describing all the commands you ran.
Then you can safely shut down RStudio.

{% snippet faqs/galaxy/interactive_tools_rstudio_stop.md  %}

# Interaction between RStudio and Galaxy

Getting data in and out from Galaxy

## Import Data from the Galaxy History to RStudio

To import a dataset from the history into RStudio, you need to get the path to that file. To do so, you can use `gx_get()` function with the dataset `id` (number in the Galaxy history) . For example, if you want to import a dataset with history ID 7 to their RStudio, you can get the path to the file by:

```
gx_get(7)
```

It is important to know that the `gx_get()` function copies the data from the Galaxy history to the RStudio session and returns the path to the copied file. You are supposed to use a proper R function to read the file. For example, you can pass the path to a function that reads tables such as `read_table` or `read_tsv`. Let's assume that dataset 7 in the history is a tab-separated table (TSV) and you want to read it into your RStudio. You can do it as follows:

```
table_name <- read.table(gx_get(7))
```

## Export Data from the RStudio

You can export the R History and all objects from RStudio to Galaxy as follows (`analysis_17.01.2025` is an arbitrary name):

```
gx_save(session_name = "analysis_17.01.2025")
```

This data object can be loaded to R.
If you want to export just one file from your R environment to your Galaxy history, you should first write the object from the memory to a file and then use the path to that file to export it. For example, if you want to export a table called `results` from RStudio to Galaxy you can do as follows:

```
write.csv(results, "./result.csv", row.names = FALSE) # Do you want to save the row.names or not?
gx_put("result.csv")
```


topic: galaxy-interface
tutorial: ncbi-sarf
title: SRA Aligned Read Format to Speed Up SARS-CoV-2 data Analysis
difficulty: intermediate
time_estimation: 30m
requirements_json: []
tags_json: []
content_hash: 2ae1a8f06e07d0ad0dc3157273fbd8e9
last_modified: 2026-05-15T22:28:13.341751
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/ncbi-sarf/tutorial.html

content:
Traditionally, after a list of run accessions has been filtered on the NCBI website, the accessions are used to download and extract fastq using the SRA toolkit to enter into the next steps of the workflow. A newer compressed data type, generated from raw submitted data containing SARS-CoV-2 sequence, is also accessible to Galaxy users from SRA in the Cloud.

SRA Aligned Read Format (SARF) provides further output options beyond basic fastq format, for example:

1. contigs created from the raw reads in the run
2. reads aligned back to the contigs
3. reads with placeholder quality scores
4. VCF files can also be downloaded for these records relative to the SARS-CoV-2 RefSeq record

- These formats can speed up workflows such as assembly and variant calling.
- This data format is still referenced by the Run accession and accessed using the SRA toolkit.
- This workshop describes the SARF data objects along with associated searchable metadata, and demonstrates a few ways to enter them into traditional workflows.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction

The aim of this tutorial is to introduce you to some of SRA's new SARS-CoV-2 cloud resources and data formats, then show you how to filter for Runs of interest to you and access that data in your format of choice in Galaxy to use in your analysis pipeline.

## SRA Aligned Read Format

All data submitted to SRA is scanned with our SARS-CoV-2 Detection Tool which uses a Kmer-based approach to identify Runs with Coronaviridae content. The initial scope of the project is limited to those runs deposited in SRA with at least 100 hits for SARS-CoV-2 via the SARS-CoV-2 Detection Tool, a read length of at least 75, and generated using the Illumina platform.

1. For these Runs, Saute was used to assemble contigs via guided assembly, with the SARS-CoV-2 refseq genomic sequence (NC_045512.2) used as the guide.

2. If contigs were successfully assembled, reads were mapped back to the contigs and coverage calculated. These contigs with the reads mapped back and with quality scores removed (to keep the object size small) are the aligned read format files.

3. The SRA toolkit can be used to dump just the contigs in fasta format, the reads aligned to the contigs in sam format or the raw reads in fastq format with placeholder quality scores.

4. The contigs were also assessed via megablast against the nucleotide blast database and the results made available for search.

5. In addition, to support investigation of viral evolution during the pandemic and after the introduction of vaccines, variants are identified relative to the SARS-CoV-2 RefSeq record for each processed run using BCFTools.

The SRA aligned reads, the VCF files, the results of these analyses (such as BLAST and VIGOR3 annotation), and the associated BioSample and sequencing library metadata are available for free access from cloud providers.


> <comment-title></comment-title>
>
> These data can be dumped in `sam` format using the `sam-dump` tool in the SRA Toolkit, but this function doesn't work within Galaxy yet.
> We hope to include that functionality in a future update.
>
{: .comment}

# Workflow Diagram

![Workflow Diagram for this Tutorial](../../images/SARF_Workflow_Overview.png "Workflow Diagram for this Tutorial")


# Finding SRA SARS-CoV-2 Runs of Interest

Metadata for SARS-CoV-2 submissions to the SRA includes submitted sample and library information, BLAST results, descriptive contig statistics, and variation and annotation information.  These metadata are updated daily and made available to query in the cloud using [Google's BigQuery](https://www.ncbi.nlm.nih.gov/sra/docs/sra-bigquery/) or [Amazon's Athena](https://www.ncbi.nlm.nih.gov/sra/docs/sra-athena/) services. However, the raw underlying information is also provided as a group of json files that can be **downloaded for free from the Open Data Platform** without logging in to the cloud. These json files can be imported to Galaxy and queried there to find **Runs of interest**.

> <comment-title></comment-title>
>
> Some of these tables include complex data fields (array of values) that don't have a clean analogue in a classic SQL database or table and these can't be easily queried in Galaxy currently. If you require access to [cloud tables](https://www.ncbi.nlm.nih.gov/sra/docs/aligned-metadata-tables/) or fields not available in Galaxy we recommend accessing those natively in [BigQuery](https://www.ncbi.nlm.nih.gov/sra/docs/sra-bigquery/) or [Athena](https://www.ncbi.nlm.nih.gov/sra/docs/sra-athena/).
>
{: .comment}


We will import the JSON files into Galaxy to query them directory, however the files are split up for efficient querying in the cloud and updated daily, so we first need to get the most up-to-date list of files so we can import those to Galaxy. We'll just be using a couple of tables in this training, but the other tables can be imported in the same way, using the index files below.


> <comment-title></comment-title>
>
> These metadata files are updated daily around 5:30pm EST. If you try to access the data around this time but encounter an error, trying again a short while later should resolve the issue. [Time Zone Converter](https://www.thetimezoneconverter.com)
>
{: .comment}

{% assign servers = nil | list_usegalaxy_servers_shuffle %}

> <hands-on-title>Loading SRA Aligned Read Format (SARF) Object Metadata URLs into Galaxy</hands-on-title>
>
>This step needs to be repeated at the beginning of an analysis to refresh the metadata to the latest daily version.
>
> 1. Go to your Galaxy instance of choice such as one of the {% for server in servers %}[{{ server.name }}]({{ server.url }}){% if forloop.last %}{% else %}, {% endif %}{% endfor %} or any other.
>
> 2. Create a new history
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 3. Rename your history, *e.g.* "NCBI SARF"
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 4. **Click** the upload icon toward the top left corner.
>
>    By default the familiar simple upload dialog should appear. This dialog has more advanced options as different tabs across the top of this dialog though.
>
> 5. **Click** `Rule-based` as shown below.
>
>    ![screenshot of the rule based uploader modal, it is empty](../../images/rules/rules_example_1_1_rules_landing.png "Be sure that you are uploading data as Datasets, and that it will be loaded from a 'Pasted Table'")
>
>    1. Copy/**paste** the URLs into the provided box:
>
>       ```
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/contigs.filelist
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/annotated_variations.filelist
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/blastn.filelist
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/metadata.filelist
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/peptides.filelist
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/tax_analysis.filelist
>       https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/variations.filelist
>       ```
>
>    2. Click **Build**
>
>    3. From **Column** menu select `Basename of Path of URL`
>       - *"From Column?"*: `A`
>
>    4. From **Rules** menu select `Add / Modify Column Definitions`
>        - Click `Add Definition` button and select `URL`
>          - *"URL"*: `A`
>        - Click `Add Definition` button and select `Name` (not name tag!)
>          - *"Name"*: `B`
>
>    5. **Click** `Apply`. You should see a table with two columns, the left being the URL column, and the right being the Name column with just the filename.
>
>       You are now ready to start the upload.
>
>    6. **Click** the `Upload` button
>
{: .hands_on}

With that you should have 7 different files in your history. If you examine the files you'll see that they each have a list of filenames like the following table, with either today or yesterday's date in the filename, depending on your timezone offset from NCBI's offices. [Time zone converter](

```
2021-05-27.000000000000.json.gz
2021-05-27.000000000001.json.gz
2021-05-27.000000000002.json.gz
2021-05-27.000000000003.json.gz
2021-05-27.000000000004.json.gz
2021-05-27.000000000005.json.gz
2021-05-27.000000000006.json.gz
2021-05-27.000000000007.json.gz
```

> <hands-on-title>Loading SRA Aligned Read Format (SARF) Contig Metadata into Galaxy</hands-on-title>
>
> Next we will convert this list of filenames to the HTTP URLs for easy import into Galaxy.
>
> 1. **Open** the `Rule-based` upload tab again, but this time:
>    1. *"Upload data as"*: `Collection(s)`
>    2. *"Load tabular data from"*: `History Dataset`
>    3. *"Select dataset to load"*: `contigs.filelist`
>
>    4. **Click** `Build` to bring up the rule builder.
>
>    5. Make the following changes in the Rule Builder
>
>       - From **Column** menu select `Fixed Value`
>           - *"Value"*: `https://storage.googleapis.com/nih-sequence-read-archive/SARS_COV_2/contigs/`
>           - Apply
>       - From **Column** menu select `Concatenate Columns`
>           - *"From Column"*: `B`
>           - *"From Column"*: `A`
>           - Apply
>       - From **Rules** menu, select `Add / Modify Column Definitions`
>           - `Add Definition`, `URL`, Select Column `C`
>           - `Add Definition`, `List Identifier(s)`, Column `A`
>           - Apply
>
>    6. Name the collection `contigs.json`
>
>    7. Click the `Upload` button.
>
>    **Once those download jobs have all turned green in the history list, we'll concatenate these into a single file.**
>
> 2. {% tool [Concatenate datasets](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.0) %} tail to head (cat) files with the following parameters:
>
>    - {% icon param-collection %} *"Datasets to Concatenate"*: `contigs.json` collection
>    - Click Run Tool
>
> 3. Rename this history item to `contigs.single.json`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="contigs.single.json" %}
>
> 4. {% tool [JQ](toolshed.g2.bx.psu.edu/repos/iuc/jq/jq/1.0) %} process JSON files with the following parameters:
>
>    - {% icon param-file %} *"JSON Input"*: `contigs.single.json`
>    - *"jq filter"*: `[.[]]`
>    - *"Convert output to tabular"*: `yes`
>    - Click Run Tool
>
> 5. Rename this file `contigs.tsv`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="contigs.tsv" %}
>
{: .hands_on}

> <comment-title></comment-title>
>
> The conversion of the json files to tabular format takes some time, this is a good time to go make some tea.
>
{: .comment}

# Query SARF Metadata

Now that a table has been generated, we will query the table to find the runs of interest.  It is a good idea to save the table for future queries on the same dataset.  Rerunning the import steps above without filtration will provide a different set of metadata each day.

> <hands-on-title>Query the SRA Metadata Table using SQLite</hands-on-title>
>
> Next we'll query this metadata using the `Query tabular` tool to get a list of all Runs containing contigs of greater than 20,000 nucleotides and average coverage of at least 100X.
>
> 1. Run {% tool [Query Tabular using sqlite sql](toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.0.0) %} with the following parameters:
>
>    > <tip-title>Can't find it?</tip-title>
>    > If you're not using the [GTN-in-Galaxy view](https://docs.galaxyproject.org/en/latest/admin/special_topics/gtn.html), you can search for 'sql' to find it.
>    {: .tip}
>
>    - In *"Database Table"*:
>      - {% icon param-repeat %} *"Insert Database Table"*
>        - {% icon param-file %} *"Tabular Dataset for Table"*: `contigs.tsv`
>        - In *"Table Options"*
>          - In *"Table Options"*
>             - *"specify name for table"*: `SARS_contigs`
>             - *"Specify Column Names"*: `name,run,coverage,tax_id,hits,length,md5`
>    - *"SQL Query to generate tabular output"*:
>
>       ```sql
>       SELECT DISTINCT run
>       FROM SARS_contigs
>       WHERE length > 20000 AND coverage > 100 AND run like '%SRR%'
>       ORDER BY run ASC
>       LIMIT 10
>       ```
>    - *"include query result column headers"*: `no`
>
>    > <tip-title>Save SQlite Database for future queries</tip-title>
>    >
>    > If you plan to do multiple queries on the same SQL database or want to skip preprocessing the metadata for future work, it may be useful to set
>    >
>    >  - {% icon param-repeat %} *"Save the sqlite database in your history"* to `Yes`
>    >
>    {: .tip}
>
> 2. Click **Run Tool** and rename the output file to `Run_list`
>
>    {% snippet faqs/galaxy/datasets_rename.md name="Run_list" %}
>
>    > <tip-title>Need fastq?</tip-title>
>    > Use `Run_list` to bring in Fastq files with Submitted Quality Scores (+BQS format)
>    {: .tip}
>
>    > <tip-title>Column Headers for the Other Metadata Tables</tip-title>
>    > We are not going to bring in the other metadata tables in this tutorial. Here is a list of column headers for contigs and the other tables.
>    > You can find full definitions for these columns here:
>    >
>    > [https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud-based-examples/](https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud-based-examples/)
>    >
>    > [https://www.ncbi.nlm.nih.gov/sra/docs/aligned-metadata-tables/](https://www.ncbi.nlm.nih.gov/sra/docs/aligned-metadata-tables/)
>    >
>    > **contigs**
>    > ```
>    > name,run,coverage,tax_id,hits,length,md5
>    > ```
>    > **annotated_variations**
>    > ```
>    > run,chrom,pos,id,ref,alt,qual,filter,info,format,sample_a,ac,an,bqb,dp,dp4,dp4_1,dp4_2,dp4_3,dp4_4,idv,imf,mq,mq0f,mqb,mqsb,rpb,sgb,vdb,g_gt,g_pl,g_pl_1,g_pl_2,g_dp,g_ad,g_ad_1,g_ad_2,protein_position,ref_codon,alt_codon,ref_aa,alt_aa,protein_name,protein_length,variation
>    > ```
>    > **blastn**
>    > ```
>    > acc,qacc,staxid,sacc,slen,length,bitscore,score,pident,sskingdom,evalue,ssciname
>    > ```
>    > **metadata**
>    > ```
>    > acc,assay_type,center_name,consent,experiment,sample_name,instrument,librarylayout,libraryselection,librarysource,platform,sample_acc,biosample,organism,sra_study,releasedate,bioproject,mbytes,loaddate,avgspotlen,mbases,insertsize,library_name,biosamplemodel_sam,collection_date_sam,geo_loc_name_country_calc,geo_loc_name_country_continent_calc,ena_first_public_run,ena_last_update_run,sample_name_sam,datastore_filetype,datastore_provider,datastore_region,attributes,jattr
>    > ```
>    > **peptides**
>    > ```
>    > name,contig,mat_peptide,run,location,gene,product,ref_db,ref_id,sequence
>    > ```
>    > **tax_analysis**
>    > ```
>    > run,contig,tax_id,rank,name,total_count,self_count,ilevel,ileft,iright
>    > ```
>    > **variations**
>    > ```
>    > run,chrom,pos,id,ref,alt,qual,filter,info
>    > ```
>    >
>    >
>    {: .tip}
>
{: .hands_on}

> <tip-title>Download Fastq with Quality Scores</tip-title>
>
> If you would like to dump the raw, underlying data in fastq format with the original quality scores, you can stop here and use the {% tool [Faster Download and Extract Reads in FASTQ](toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fasterq_dump/2.11.0+galaxy0) %} tool with the following parameters:
>
> - *"input type"*: `list of SRA accessions, one per line`
> - {% icon param-file %} *"sra accession list"*: `Run_list`
>
{: .tip}

> <tip-title>Importing a list of SRR from Athena or BigQuery</tip-title>
>
> If you opted to conduct your metadata search in the cloud using AWS Athena or GCP BigQuery instead of importing the json file to Galaxy, you can save a list of your Run accessions from that search result and import that file as the `Run_list` to proceed with the rest of this tutorial.
{: .tip}

# Importing SARFs of Interest

Now that we have assembled a list of Runs that have contigs we are interested in, we'll construct the path to the **SARFS** in the cloud and import those to Galaxy so we can work with them.

> <hands-on-title>Importing SARFs of Interest</hands-on-title>
>
> 1. Upload Data
>
>    1. **Open** the `Rule-based` upload tab again, but this time:
>       - *"Upload data as"*: `Collection(s)`
>       - *"Load tabular data from"*: `History Dataset`
>       - *"Select dataset to load"*: `Run_list`
>
>    2. **Click** `Build` to bring up the rule builder.
>
>    3. Make the following changes in the Rule Builder
>
>       - From **Column** menu select `Using a Regular Expression`
>           - Check *"Create column from expression replacement"*
>           - *"Regular Expression"*: `(.*)`
>           - *"Replacement Expression"*: `https://sra-pub-sars-cov2.s3.amazonaws.com/RAO/\1/\1.realign`
>           - Apply
>       - From **Rules** menu select `Add / Modify Column Definitions`
>          - Click `Add Definition` button and select `URL`
>            - *"URL"*: `B`
>          - Click `Add Definition` button and select `List Identifier`
>            - *"List Identifier"*: `A`
>            - Apply
>
>    4. Name the output collection `sarf_path` before clicking **Upload**
>
>       > <comment-title></comment-title>
>       >
>       > Please note that there can be some lag in availability of SARF/VCF files in the cloud (particularly for newly submitted data). So it's possible to get a download error for a file that isn't yet present in the cloud. In these cases waiting ~24 hours will generally resolve the issue and allow you to access the file.
>       >
>       {: .comment}
>
> 2. **Now we will use the SRA toolkit to retrieve the contigs in fasta format**
>
>    Run {% tool [Download and Extract Reads in FASTA/Q format from NCBI SRA](toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fastq_dump/2.11.0+galaxy0) %} with the following parameters:
>
>    - *"Select input type"*: `SRA Archive in current history`
>    - {% icon param-collection %}*"sra archive"*: `sarf_path`
>    - In "Advanced Options":
>      - *"Table name within cSRA object"*: `REFERENCE`
>    - Click Run Tool
>
>    The resulting dataset includes the contigs generated from these runs with placeholder `?` for quality scores
>
>    - Rename this collection to `sarf_contigs`
>
>    Run {% tool [Fastq to Fasta converter](toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5) %}
>
>    - {% icon param-collection %}*"FASTQ file to convert"*: `sarf_contigs` (note: in the video this did not get renamed)
>    - Click Run Tool
>
>    The resulting dataset includes the contigs generated from these Runs in fasta format
>
>    > <tip-title>Fastq format option</tip-title>
>    > If you prefer to dump the raw reads in fastq format with placeholder quality scores, leave the `Table name within cSRA object` field blank.
>    >
>    {: .tip}
{: .hands_on}

# Importing VCFs for SARS-Cov-2 Runs

This example starts with the same `Run_list` generated for importing SARFs.

A `Run_list` could also be imported after querying metadata in the cloud using  [Google's BigQuery](https://www.ncbi.nlm.nih.gov/sra/docs/sra-bigquery/) or [Amazon's Athena](https://www.ncbi.nlm.nih.gov/sra/docs/sra-athena/) services. Metadata about these runs includes submitted sample and library information, BLAST results, descriptive contig statistics, and variation and annotations. See the tutorial video for a short demo on how to search and download `Run_list` from the cloud.


> <hands-on-title>Importing VCFs of Interest</hands-on-title>
>
> 1. Upload Data
>
>    1. **Open** the `Rule-based` upload tab again, but this time:
>       - *"Upload data as"*: `Collection(s)`
>       - *"Load tabular data from"*: `History Dataset`
>       - *"Select dataset to load"*: `Run_list`
>
>    2. **Click** `Build` to bring up the rule builder.
>
>    3. Make the following changes in the Rule Builder
>
>       - From **Column** menu select `Using a Regular Expression`
>           - Check *"Create column from expression replacement"*
>           - *"Regular Expression"*: `(.*)`
>           - *"Replacement Expression"*: `https://sra-pub-sars-cov2.s3.amazonaws.com/VCF/\1/\1.vcf`
>           - Apply
>       - From **Rules** menu select `Add / Modify Column Definitions`
>            - *"URL"*: `B`
>            - *"List Identifier"*: `A`
>            - Apply
>
>    4. Name the output collection `VCFs` before clicking **Upload**
>
>       > <comment-title></comment-title>
>       >
>       > Please note that there can be some lag in availability of SARF/VCF files in the cloud (particularly for newly submitted data). So it's possible to get a download error for a file that isn't yet present in the cloud. In these cases waiting ~24 hours will generally resolve the issue and allow you to access the file.
>       >
>       {: .comment}
>
> 2. **Use VCFs in Another Galaxy Tool**. Once you have imported the VCF files, you can use them in your standard pipeline- here we will annotate them with SnpEff.
>
>    Run {% tool [SnpEff eff: annotate variants for SARS-CoV-2](toolshed.g2.bx.psu.edu/repos/iuc/snpeff_sars_cov_2/snpeff_sars_cov_2/4.5covid19) %}  with the following parameters:
>
>    - {% icon param-collection %} *"Sequence changes (SNPs, MNPs, InDels)"*: the `VCFs` collection we just created
>
>    > <comment-title></comment-title>
>    >
>    > Please note that there are 2 Snepff tools, please choose the one for SARS-CoV-2
>    >
>    {: .comment}
>
{: .hands_on}

# Feedback for NCBI

If you enjoyed this tutorial, please consider filling out this feedback link for NCBI: [https://nlmenterprise.co1.qualtrics.com/jfe/form/SV_0jQct4IQOgfYYaq](https://nlmenterprise.co1.qualtrics.com/jfe/form/SV_0jQct4IQOgfYYaq)

Note: the survey will stay open until July 31, 2021 (it may say it expires June 30, but we will extend the deadline at the end of the month)

(There is another survey link below for Galaxy).

# Other NCBI Resources
* [Getting started with SRA in the Cloud](https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud/)
* [NCBI SARS-CoV-2 Resources](https://www.ncbi.nlm.nih.gov/sars-cov-2/)
* [STAT tool](https://www.ncbi.nlm.nih.gov/sra/docs/sra-taxonomy-analysis-tool/)
* [SRA Aligned Read Format](https://www.ncbi.nlm.nih.gov/sra/docs/sra-aligned-read-format/)
* [VCF generation](https://www.ncbi.nlm.nih.gov/sra/docs/sars-cov-2-variant-calling/)

If you have questions or feedback you can email the SRA helpdesk: [sra@ncbi.nlm.nih.gov](mailto:sra@ncbi.nlm.nih.gov)

# Acknowledgements

 The authors would like to acknowledge the SRA Product Team at NCBI and the Community Outreach staff at Galaxy for their assistance on this tutorial.

# Affiliations

 Adelaide Rhodes & Jon Trow - Computercraft assigned to NCBI/NLM/NIH

tools_json:
["Concatenate datasets", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.0", "JQ", "toolshed.g2.bx.psu.edu/repos/iuc/jq/jq/1.0", "Query Tabular using sqlite sql", "toolshed.g2.bx.psu.edu/repos/iuc/query_tabular/query_tabular/3.0.0", "Faster Download and Extract Reads in FASTQ", "toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fasterq_dump/2.11.0+galaxy0", "Download and Extract Reads in FASTA/Q format from NCBI SRA", "toolshed.g2.bx.psu.edu/repos/iuc/sra_tools/fastq_dump/2.11.0+galaxy0", "Fastq to Fasta converter", "toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/fastq_to_fasta_python/1.1.5", "SnpEff eff: annotate variants for SARS-CoV-2", "toolshed.g2.bx.psu.edu/repos/iuc/snpeff_sars_cov_2/snpeff_sars_cov_2/4.5covid19"]


topic: galaxy-interface
tutorial: search
title: Searching Your History
difficulty: introductory
time_estimation: 5m
content: {% include topics/galaxy-interface/tutorials/search/search.md %}
tools_json: []
requirements_json: []
tags_json: []
content_hash: 5e6747bf6f48c44be845dfb2c0eea99d
last_modified: 2026-05-15T22:28:13.352716
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/search/tutorial.html


topic: galaxy-interface
tutorial: intermine
title: InterMine integration with Galaxy
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["gmod", "intermine"]
content_hash: abaedc0bcb645e21ff225aa3a951372c
last_modified: 2026-05-15T22:28:13.337394
zenodo_link: https://zenodo.org/record/3407174
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/intermine/tutorial.html

content:
InterMine ({% cite Smith2012 %}) is a well-establish platform to integrate and access life sciences data.
It provides the integrated data via a web interface and RESTful web services.

Other organizations download and deploy InterMine on their servers:
there are more than 30 instances over the world (registered at [registry.intermine.org](http://registry.intermine.org)), covering many organism,
including human data, model animals, plants and drug targets.

InterMine has been integrated with Galaxy: the InterMine tool server in Galaxy allows
to import the data returned by any InterMine search and viceversa, using the InterMine Interchange format
it's possible to export a list of identifiers from Galaxy into any InterMine instance of your choice.

Learn more in this tutorial.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Import data from InterMine

> <hands-on-title>Import</hands-on-title>
> Search Galaxy for `InterMine` (not case sensitive; `intermine` is fine too), and click on **InterMine Server** under **Get Data**.
>
> 1. {% tool [InterMine Server](intermine) %}
>
> 2. This will redirect you to the InterMine registry, which shows a full list of InterMines and the various organisms they support. Find an InterMine that has the organism type you’re working with, and click on it to redirect to that InterMine.
>
> 3. Once you arrive at your InterMine of choice, you can run a query as normal - this could be a search, a list results page, a template, or a query in the query builder. Eventually you’ll be presented with an InterMine results table.
>
> 4. Click on **Export** (top right). This will bring up a modal window.
> 5. Select **Send to Galaxy** and double-check the *"Galaxy Location"* is correct.
> 6. Click on the **Send to Galaxy** button on the bottom right of the pop-up window.
>
>    > <tip-title>Enable popups</tip-title>
>    >
>    > If you get an error when you click on the **Send to Galaxy** button, please make sure to allow popups and try again.
>    {: .tip}
>
{: .hands_on}
You have now exported your query results from InterMine to Galaxy.


# Export identifiers into InterMine

## Get data

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import some fly data from [Zenodo](https://zenodo.org/record/3407174) or from the data library
>
>    ```
>    https://zenodo.org/record/3407174/files/GenesLocatedOnChromosome4.tsv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
> 2. Rename the dataset to `GenesLocatedOnChromosome4`
>
>    {% snippet faqs/galaxy/datasets_rename.md %}
>
> 3. Inspect the data
>
{: .hands_on}

The dataset contains the secondary identifier and the symbol of the *Drosophila melanogaster* genes and their location on the chromosome 4

> <question-title></question-title>
>
> Do the data contain the type, e.g `Protein` or `Gene`?
>
> > <solution-title></solution-title>
> >
> > No, they don't. So we have to specify it, when we create the InterMine Interchange file
> >
> {: .solution}
>
{: .question}

## Create InterMine Interchange dataset

We will use **Create InterMine Interchange Dataset** {% icon tool %} in order to generate an intermediate file which will be used to send the identifiers (e.g. gene identifiers) to InterMine. This file requires the identifier's type (e.g. `Gene`), the identifier (e.g `WBGene00007063`) and, optionally, the organims's name.

> <hands-on-title>Generate InterMine file</hands-on-title>
>
> 1. {% tool [Create InterMine Interchange dateset](toolshed.g2.bx.psu.edu/repos/iuc/intermine_galaxy_exchange/galaxy_intermine_exchange/0.0.1) %} with the following parameters:
>    - {% icon param-file %} *"Tabular file"*: select the `GenesLocatedOnChromosome4` dataset which contains some fly's genes
>    - *"Feature Type Column"*: `Column: 1`
>    - *"Feature Type"*: `Gene`
>    - *"Feature Identifier column"*: `Column: 2`
>
>    > <comment-title></comment-title>
>    > - In this example, because the `GenesLocatedOnChromosome4` dataset does not contain the type we have to specify it, in the *"Feature Type"*
>    > - *"Feature Type"*: this is type of the identifiers you are exporting to InterMine, in this example `Gene`. It must be a class in the InterMine data model.
>    > - *"Feature Identifier column"*: select a column from the input file which contains the identifier. We have selected Column 2, which contains the gene symbol.
>    > - *"Feature Identifier"*: This could be, as an example, a gene symbol like `GATA1` or another other identifier, e.g. `FBGN0000099` or perhaps a  protein accession. In our example we do not have to edit anything because the values for this field are contained in the `GenesLocatedOnChromosome4` dataset, in *Column 2*.
>    > - *"Organism Name column"*: select a column from the input file which contains the organism's name, if you have multiple organisms in the same dataset.
>    > - *"Organism Name"*: alternatively you can directly provide the organism's name. The organims' name is not mandatory, but is good to provide if it is known. It does not have to be precise
>    {: .comment}
>
> 2. Click on **Run Tool**
>
{: .hands_on}

## Send identifiers to InterMine

Once the generation of the interchange dataset has been completed, open the green box related to **Create InterMine Interchange on data**.

> <hands-on-title>Send data</hands-on-title>
>
> 1. Click on view intermine at **Registry** to be redirected to the InterMine registry, which shows a full list of InterMines and the various organisms they support.
> 2. Find an InterMine that has the organism type you’re working with, in our case FlyMine, and click on the **Send to** green button to export the identifiers to.
>    3. You are redirected to FlyMine, in the List Analysis page showing the identifiers you have just exported from Galaxy.
>
{: .hands_on}

# Conclusion

You have now exported your identifiers from Galaxy to InterMine.

tools_json:
["InterMine Server", "intermine", "Create InterMine Interchange dateset", "toolshed.g2.bx.psu.edu/repos/iuc/intermine_galaxy_exchange/galaxy_intermine_exchange/0.0.1"]


topic: galaxy-interface
tutorial: upload-rules-advanced
title: Rule Based Uploader: Advanced
difficulty: advanced
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["collections", "tags"]
content_hash: 7395aa35f012a9be73aa31543d3a0492
last_modified: 2026-05-15T22:28:13.368595
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/upload-rules-advanced/tutorial.html

content:
This builds on the previous Rule Based Uploader tutorial to cover even more advanced topics.

> ###  {% icon comment %} Audience
> This tutorial assumes a basic knowledge of using dataset collections in Galaxy but doesn't assume any particular knowledge of biology or bioinformatics. If you have not used collections with Galaxy previously, please check out the [using dataset collections]({% link topics/galaxy-interface/tutorials/collections/tutorial.md %}) tutorial.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Building URLs from Accession Information

In the previous examples the metadata we started with already contained URLs. In some cases such URLs will not be present in the available metadata and may need to be constructed dynamically from identifiers.

For this multiomics example, we will start with a uniprot query and build URLs from accession numbers contained within the supplied tabular data. Consider the uniprot query [https://uniprot.org/uniprot/?query=proteome:UP000052092+AND+proteomecomponent:"Genome"](https://uniprot.org/uniprot/?query=proteome:UP000052092+AND+proteomecomponent:%22Genome%22), pictured below.

![UniProt table screenshot](../../images/rules/uniprot_url.png)

In order to make this data useful for the Rule Builder, we need to turn these accessoin IDs into URLs.

> <hands-on-title>Turn Uniprot Accession IDs into URLs</hands-on-title>
>
> 1. Open [Uniprot](https://www.uniprot.org/uniprot/E7C0H6)
> 1. **Click** on one of the entries in the table (e.g. `E7C0H6`)
> 2. **Select** `Format` from the top menu
> 3. **Click** `FASTA (canonical)`
>
>    ![Link to access FASTA file](../../images/rules/uniprot_fasta.png)
>
> 4. Your browser will redirect to the fasta file
>
>    ![Viewing the FASTA file](../../images/rules/uniprot_fasta_url.png)
>
> 5. We can deduce that the FASTA files for the other accession IDs will be available from URLs of the form `https://www.uniprot.org/uniprot/{identifier}.fasta`
> <!-- https://rest.uniprot.org/uniprotkb/search?format=fasta&includeIsoform=true&query=accession:E7C0H6  is also an option, for future editors. -->
{: .hands_on}
{:#example-3-metadata}

We will use this information to build a collection of FASTA files from our query.

> <hands-on-title>Build Collection from Uniprot Accession IDs</hands-on-title>
>
> 1. Open the Rule Builder
>    - *"Upload data as"*: `Collection(s)`
>    - *"Load tabular data from"*: `Pasted Table`
>    - **Paste** the following table:
>
>      ```
>      Entry	Entry name	Status	Protein names	Gene names	Organism	Length
>      E7C0H6	E7C0H6_9PAPI	unreviewed	Major capsid protein L1	L1	Equus caballus papillomavirus 3	498
>      E7C0H0	E7C0H0_9PAPI	unreviewed	Protein E6	E6	Equus caballus papillomavirus 3	150
>      E7C0H5	E7C0H5_9PAPI	unreviewed	Minor capsid protein L2	L2	Equus caballus papillomavirus 3	498
>      E7C0H1	E7C0H1_9PAPI	unreviewed	Protein E7		Equus caballus papillomavirus 3	93
>      E7C0H3	E7C0H3_9PAPI	unreviewed	Regulatory protein E2	E2	Equus caballus papillomavirus 3	421
>      E7C0H4	E7C0H4_9PAPI	unreviewed	Putative E4 early protein (Fragment)		Equus caballus papillomavirus 3	175
>      E7C0H2	E7C0H2_9PAPI	unreviewed	Replication protein E1 (EC 3.6.4.12) (ATP-dependent helicase E1)	E1	Equus caballus papillomavirus 3	621
>      ```
>
>    - Click `Build`
>   ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_2_initial_rules.png)
>
> 2. Let's apply some initial filtering to our data:
>    - From **Filter** menu, select `First or Last N Rows`
>        - Filter the first row
>    - From **Rules** menu, select  `Remove Columns`
>        - Remove columns B, C, E, F, and G.
>    - From **Rules** menu, select  `Add / Modify Column Definitions`
>        - Click `Add Definition`, `Info`, Column `B`
>
>          This is a block of text that will appear in the history panel when the dataset is expanded.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_3_basic_rules.png)
>
>    These datasets appear in a seemingly random order, it will be easier to manage things in the history panel if we sort this data first.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_4_sort.png)
>
> 3. From **Rules** menu, select `Sort`
>     - *"From Column"*: `A`
>
>    Next is the key step, we will build a URL from the pattern we described above using the accession ID in column A.
>
> 4. From **Column** menu, select `Using a Regular Expression`
>    - *"From Column"*: `A`
>    - Select `Create column from expression replacement`
>    - *"Regular Expression"*: `.*`
>    - *"Replacement Expression"*: `https://www.uniprot.org/uniprot/\0.fasta`
>    - Click `Apply`
>
>      ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_5_url.png)
>
>    > <comment-title>Regular expression explained</comment-title>
>    > In this regular expression, `.*` will capture the entire accession ID.
>    >
>    > In the replacement expression, `https://www.uniprot.org/uniprot/\0.fasta`, the `\0` will be replaced by the captured
>    > regular expression (the accession ID) in the resulting column values.
>    {: .comment}
>
>    The new column C with the URL we built should appear as shown below.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_6_url_built.png)
>
>    Finally, let us set our column definitions:
>
> 5. From **Rules** menu, select `Add / Modify Column Definitions`
>     - `Add Definition`, `List Identifier(s)`, Select Column `A`
>     - `Add Definition`, `URL`, Column `C`
>
> 6. Finalize the collection:
>     - *"Name"*: for example `UP000052092`
>     - *"Type"*: `fasta`
>     - Click `Build`
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_7_mapping_extension_and_name.png)
>
>    > <comment-title>JSON Rule Definitions</comment-title>
>    > This example is ready to go, but before clicking "Build" it may be interesting to check out the rules Galaxy is following to clean and import your data. Click the little Wrench icon at the top of the list of rules. The result is a bunch of JavaScript object notation (JSON) text that you should never need to worry about but that you can build or modify by hand if you find it useful. We will use it the next example to quickly restore the list builder back to this state.
>    >
>    > ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_4_8_source.png)
>    >
>    > This could additionally be copied and pasted if you need to do the same set of operations on multiple metadata inputs that are similarly formatted.
>    {: .comment}
> 7. Click `Upload` and wait for your list of FASTA files to appear.
{: .hands_on}


# Building Matched Collections

This example will demonstrate creating multiple collections at the same time. We will use the same metadata generated from UniProt as the last example but we will build two collections with matching list identifiers - one collection of FASTA files and one collection of GFF files. This will also demonstrate reading the collection name and the target datatype from the metadata itself - important techniques if generating multiple collections with different names and datatypes.

For this example we will re-use the metadata from the [previous example](#example-3-metadata).

> <hands-on-title>Build a matched collection</hands-on-title>
>
> 1. **Open** the Rule Builder
>    - *"Upload data as"*: `Collection(s)`
>    - *"Load tabular data from"*: `Pasted Table`
>    - Paste the table from the previous exercise:
>
>      ```
>      Entry	Entry name	Status	Protein names	Gene names	Organism	Length
>      E7C0H6	E7C0H6_9PAPI	unreviewed	Major capsid protein L1	L1	Equus caballus papillomavirus 3	498
>      E7C0H0	E7C0H0_9PAPI	unreviewed	Protein E6	E6	Equus caballus papillomavirus 3	150
>      E7C0H5	E7C0H5_9PAPI	unreviewed	Minor capsid protein L2	L2	Equus caballus papillomavirus 3	498
>      E7C0H1	E7C0H1_9PAPI	unreviewed	Protein E7		Equus caballus papillomavirus 3	93
>      E7C0H3	E7C0H3_9PAPI	unreviewed	Regulatory protein E2	E2	Equus caballus papillomavirus 3	421
>      E7C0H4	E7C0H4_9PAPI	unreviewed	Putative E4 early protein (Fragment)		Equus caballus papillomavirus 3	175
>      E7C0H2	E7C0H2_9PAPI	unreviewed	Replication protein E1 (EC 3.6.4.12) (ATP-dependent helicase E1)	E1	Equus caballus papillomavirus 3	621
>      ```
>
> 2. Instead of manually creating the rules this time, we will import an existing set of rules. The easiest way to do this is to click the history icon {% icon galaxy-rulebuilder-history %}. Clicking this will drop down a list of the 10 most recent rule sets you've used. Select the most recent one to reuse the rules from the last exercise.
>
>    You should now see the rules you created in the last example.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_5_3_initial_rules.png)
>
>    > <comment-title>JSON Editor</comment-title>
>    >
>    > Another way to do this is to open the JSON Editor, by clicking the wrench icon {% icon tool %} to the left of the history icon.  This will open up the JSON that gets created as you modify rules.  You can copy and paste JSON here or modify it directly. When you're working in this way, be sure to write good JSON (closing brackets, fields that make sense, etc.), otherwise you will get an error.
>    {: .comment}
>
>    If you don't see the rule set we used in the last  exercise anymore, here is the JSON that you can paste directly into the JSON Editor:
>
>    ```json
>    {"rules":[{"type":"add_filter_count","count":"1","which":"first","invert":false},{"type":"remove_columns","target_columns":[1,2,4,5,6]},{"type":"sort","target_column":0,"numeric":false},{"type":"add_column_regex","target_column":0,"expression":".*","replacement":"http://www.uniprot.org/uniprot/\\0.fasta"}],"mapping":[{"type":"info","columns":[1]},{"type":"list_identifiers","columns":[0],"editing":false},{"type":"url","columns":[2]}],"extension":"csfasta"}
>    ```
>
>    This next part may seem a bit silly at first but we are going to add some columns with fixed values into the builder. When we split up the columns at a later step this will make sense.
>
> 3. From **Column** menu, select  `Fixed Value`
>
>    - *"Value"*: `fasta`
>    - Click `Apply`
>
>    This value will eventually be used for the datatype of the file.
>
> 4. **Repeat** this process with:
>
>    - *"Value"*:`UP000052092 FASTA`
>
>    Next we will repeat the process of adding URL, name, and datatype columns but for GFF files.
>
> 5. From **Column**, select `Using a Regular Expression`
>
>    - *"From Column"*: `A`
>    - Select `Create column from expression replacement`
>    - *"Regular Expression"*: `.*`
>    - *"Replacement Expression"*: `https://www.uniprot.org/uniprot/\0.gff`.
>
> ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_5_5_url.png)
>
>    Next we will add two more columns
> 6. From **Column** menu, select  `Fixed Value`
>     - *"Value"*: `gff3`
>     - Click `Apply`
>
>    This value will eventually be used for the datatype of the file.
>
> 7. **Repeat** this process with:
>     - *"Value"*: `UP000052092 GFF3`
>
>    Your very large list of rules should now look like the following screenshot.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_5_6_presplit.png)
>
>    Notice we have two URLs, two collection names, and two datatype extensions for each accession ID we started with. Like in the example where we split the columns, here we will split these up to describe multiple collections.
> 7. From **Rules** menu, select  `Split Columns`
>    - *"Odd Row Column(s)"*: `C`, `D`, and `E` (the fasta columns)
>    - *"Even Row Column(s)"*: `F`, `G`, and `H` (the gff3 columns)
>
>    This will take the row consisting of the columns `ABCDEFGH` and build two rows, one with `ABCDEF` and the other with `ABFGH`
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_5_7_split_columns.png)
>
> 8. Click `Apply` and you should be returned the list of rules.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_5_8_split.png)
>
>    Finally, we need to add some more column definitions for these new columns we just created:
>
> 9. From **Rules** menu, select `Add / Modify Column Definitions`
>     - `Add Definition`, `List Identifier`, Column `A`
>     - `Add Definition`, `Info`, Column `B`
>     - `Add Definition`, `URL`, Column `C`
>     - `Add Definition`, `Type`, Column `D`
>     - `Add Definition`, `Collection Name`, Column `E`
>
>    Notice when these values are being generated from the metadata the option to specify them manually from the **type** and **collection name** boxes from the bottom of the form disappear.
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_5_9_mapping.png)
>
> 10. Click `Upload`
>
>     Galaxy should make two collections - one containing FASTA files and one containing GFF3 files.
>
{: .hands_on}


# Building Nested Lists

In this example, we will be building a nested list using data from [SRA](https://www.ncbi.nlm.nih.gov/sra). This is a more sophisticated structure for organizing datasets in Galaxy. In the above examples we organized datasets into simple lists with a single "list identifier" describing the files in the collection. Galaxy allows lists to be organized into nested lists - where each level of the list has an identifier.

If two such identifiers are present, this is a list of lists (called ``list:list`` in the workflow editor). In such a structure the outer identifiers (or first level of identifiers) may describe sample names and the inner identifiers (or second level) may describe replicates. Alternative the outer identifiers may describe conditions and the inner identifiers may describe individual samples. The structure of such collections should ideally be dictated by the study design.

If certain parts of your analysis require benefit from datasets being nested this way while other parts require feeding you data to a Galaxy tool all together without such a structure, then it is probably best to go ahead and build nested lists at the start of your analysis and then use the "Flatten Collection" tool on the resulting collection or a derivative collection to get the flat structure needed by certain tools in certain parts of your analysis.

For this example, we will describe analyzing the metadata of the [SRA project PRJNA355367](https://www.ncbi.nlm.nih.gov/sra?term=PRJNA355367&cmd=DetailsSearch). Unlike the other examples, these SRA files are relatively large and not ideal for training purposes. So we've pre-downloaded the project metadata and replaced all the links with simple text files that should download really quickly - the [result is here](./PRJNA355367.tsv).

So use either the SRA exporter tool or download the CSV file with fake URLs. If you download the data from the SRA exporter tool, select only the first 12 columns from the data (up to the column labeled "Library Name") and copy the resulting data to your clipboard.


> <hands-on-title>Building Nested Lists</hands-on-title>
>
> 1. Open the Rule Builder
>    - *"Upload data as"*: `Collection(s)`
>    - *"Load tabular data from"*: `Pasted Table`
>    - Paste the table
>
>    > <solution-title>Show Table</solution-title>
>    >
>    > Below is the table with pre-downloaded example data mentioned above:
>    >
>    > ```
>    > Run	ReleaseDate	LoadDate	spots	bases	spots_with_mates	avgLength	size_MB	AssemblyName	download_path	Experiment	LibraryName
>    > SRR5363633	2017-03-21 11:40:07	2017-03-21 11:29:13	9942851	1993124486	9791175	200	809	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363633	SRX2659078	wtPA14
>    > SRR5363634	2017-03-21 13:19:08	2017-03-21 13:17:11	1158307	232298840	1141697	200	81	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363634	SRX2659079	con10
>    > SRR5363635	2017-03-21 13:19:08	2017-03-21 13:16:39	1270067	254490549	1249658	200	89	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363635	SRX2659080	con9
>    > SRR5363636	2017-03-21 13:19:08	2017-03-21 13:16:56	1023725	205709452	1013013	200	74	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363636	SRX2659081	con8
>    > SRR5363637	2017-03-21 11:37:06	2017-03-21 11:28:00	3179821	638410351	3141113	200	216	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363637	SRX2659082	con7
>    > SRR5363638	2017-03-21 12:53:08	2017-03-21 12:04:58	3359498	675138344	3325080	200	288	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363638	SRX2659083	con6
>    > SRR5363639	2017-03-21 13:19:08	2017-03-21 13:17:25	2074389	416191247	2046343	200	145	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363639	SRX2659084	con5
>    > SRR5363640	2017-03-21 13:19:08	2017-03-21 13:17:46	2692438	540402576	2658118	200	187	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363640	SRX2659085	con4
>    > SRR5363641	2017-03-21 13:19:08	2017-03-21 13:17:14	2271913	456015568	2243122	200	161	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363641	SRX2659086	con3
>    > SRR5363642	2017-03-21 13:19:08	2017-03-21 13:16:58	2164949	434399488	2136063	200	151	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363642	SRX2659087	con2
>    > SRR5363643	2017-03-21 11:37:06	2017-03-21 11:27:34	1999861	401313700	1973567	200	140	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363643	SRX2659088	con1
>    > SRR5363644	2017-03-21 11:44:06	2017-03-21 11:36:06	13473203	2704705108	13306228	200	1076	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363644	SRX2659089	str10
>    > SRR5363645	2017-03-21 11:37:06	2017-03-21 11:27:24	1801962	362038363	1782599	200	130	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363645	SRX2659090	str9
>    > SRR5363646	2017-03-21 11:36:06	2017-03-21 11:26:22	39113	7861711	39106	200	4	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363646	SRX2659091	str8
>    > SRR5363647	2017-03-21 11:36:06	2017-03-21 11:26:43	1255569	252289127	1242359	200	90	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363647	SRX2659092	str7
>    > SRR5363648	2017-03-21 11:36:06	2017-03-21 11:26:42	903374	181362535	892308	200	64	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363648	SRX2659093	str6
>    > SRR5363649	2017-03-21 11:37:06	2017-03-21 11:27:05	2059145	413598822	2035919	200	140	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363649	SRX2659094	str5
>    > SRR5363650	2017-03-21 11:37:06	2017-03-21 11:27:37	1685689	338152407	1662375	200	119	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363650	SRX2659095	str4
>    > SRR5363651	2017-03-21 11:36:06	2017-03-21 11:26:51	698470	140689157	694502	201	48	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363651	SRX2659096	str3
>    > SRR5363652	2017-03-21 11:37:06	2017-03-21 11:27:53	755393	152119594	750753	201	52	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363652	SRX2659097	str2
>    > SRR5363653	2017-03-21 11:37:06	2017-03-21 11:27:41	1651613	331529685	1630882	200	119	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363653	SRX2659098	str1
>    > SRR5363654	2017-03-21 11:37:06	2017-03-21 11:27:37	1693765	340003252	1672625	200	118	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363654	SRX2659099	gen10
>    > SRR5363655	2017-03-21 11:37:06	2017-03-21 11:28:30	2173449	436124391	2144642	200	153	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363655	SRX2659100	gen9
>    > SRR5363656	2017-03-21 11:43:06	2017-03-21 11:33:05	9625215	1933561588	9519090	200	778	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363656	SRX2659101	gen8
>    > SRR5363657	2017-03-21 11:37:06	2017-03-21 11:27:38	1106350	222126014	1092932	200	79	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363657	SRX2659102	gen7
>    > SRR5363658	2017-03-21 11:37:07	2017-03-21 11:27:07	565475	113765064	560920	201	42	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363658	SRX2659103	gen6
>    > SRR5363659	2017-03-21 11:37:07	2017-03-21 11:28:20	1320926	265553033	1308329	201	96	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363659	SRX2659104	gen5
>    > SRR5363660	2017-03-21 11:37:07	2017-03-21 11:28:18	949047	190786413	939940	201	70	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363660	SRX2659105	gen4
>    > SRR5363661	2017-03-21 11:39:07	2017-03-21 11:29:11	1609659	323469152	1593027	200	116	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363661	SRX2659106	gen3
>    > SRR5363662	2017-03-21 11:37:07	2017-03-21 11:28:20	1244105	249942443	1230587	200	90	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363662	SRX2659107	gen2
>    > SRR5363663	2017-03-21 11:44:06	2017-03-21 11:34:00	13546763	2714292352	13327595	200	1112	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363663	SRX2659108	gen1
>    > SRR5363664	2017-03-21 11:37:07	2017-03-21 11:28:35	1995417	400617217	1971115	200	140	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363664	SRX2659109	pit9
>    > SRR5363665	2017-03-21 11:39:07	2017-03-21 11:29:18	2098280	421450493	2074524	200	146	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363665	SRX2659110	pit10
>    > SRR5363666	2017-03-21 11:39:07	2017-03-21 11:28:56	1624164	326200210	1605561	200	115	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363666	SRX2659111	pit8
>    > SRR5363667	2017-03-21 11:39:07	2017-03-21 11:28:56	1765896	355178220	1750742	201	122	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363667	SRX2659112	pit7
>    > SRR5363668	2017-03-21 11:39:07	2017-03-21 11:29:36	834144	167643900	825708	200	59	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363668	SRX2659113	pit6
>    > SRR5363669	2017-03-21 11:42:06	2017-03-21 11:31:07	9524989	1913263017	9418320	200	780	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363669	SRX2659114	pit5
>    > SRR5363670	2017-03-21 11:39:07	2017-03-21 11:28:48	1196456	240479817	1184548	200	84	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363670	SRX2659115	pit4
>    > SRR5363671	2017-03-21 11:39:07	2017-03-21 11:29:21	1703068	341745049	1680568	200	119	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363671	SRX2659116	pit3
>    > SRR5363672	2017-03-21 11:40:07	2017-03-21 11:29:21	1752519	352259586	1735224	201	122	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363672	SRX2659117	pit2
>    > SRR5363673	2017-03-21 11:37:07	2017-03-21 11:28:40	1702102	341794907	1682027	200	121	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363673	SRX2659118	pit1
>    > SRR5363674	2017-03-21 11:37:07	2017-03-21 11:28:41	2186337	438861929	2158858	200	155	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363674	SRX2659119	car10
>    > SRR5363675	2017-03-21 11:37:07	2017-03-21 11:28:37	1806341	362679360	1784566	200	128	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363675	SRX2659120	car9
>    > SRR5363676	2017-03-21 11:37:07	2017-03-21 11:28:29	1492644	299621613	1473926	200	106	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363676	SRX2659121	car8
>    > SRR5363677	2017-03-21 11:40:07	2017-03-21 11:29:02	2139706	429729175	2115063	200	151	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363677	SRX2659122	car7
>    > SRR5363678	2017-03-21 11:40:07	2017-03-21 11:29:13	1787521	358790286	1764881	200	124	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363678	SRX2659123	car6
>    > SRR5363679	2017-03-21 11:38:06	2017-03-21 11:28:48	1500378	300767719	1477539	200	106	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363679	SRX2659124	car5
>    > SRR5363680	2017-03-21 11:38:06	2017-03-21 11:28:54	879242	176495860	868255	200	63	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363680	SRX2659125	car4
>    > SRR5363681	2017-03-21 11:40:07	2017-03-21 11:30:02	1852256	371663665	1827605	200	130	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363681	SRX2659126	car3
>    > SRR5363682	2017-03-21 11:40:07	2017-03-21 11:29:40	996680	199979148	983324	200	72	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363682	SRX2659127	car2
>    > SRR5363683	2017-03-21 11:40:07	2017-03-21 11:29:49	1089078	218680766	1076092	200	79	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363683	SRX2659128	car1
>    > SRR5363684	2017-03-21 11:40:07	2017-03-21 11:29:57	1682993	337921519	1662784	200	118	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363684	SRX2659129	cef10
>    > SRR5363685	2017-03-21 11:40:07	2017-03-21 11:29:46	1274849	255689623	1256747	200	92	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363685	SRX2659130	cef9
>    > SRR5363686	2017-03-21 11:40:07	2017-03-21 11:29:22	1218188	244568636	1203299	200	87	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363686	SRX2659131	cef8
>    > SRR5363687	2017-03-21 11:40:07	2017-03-21 11:30:06	2027741	406979584	2001785	200	142	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363687	SRX2659132	cef7
>    > SRR5363688	2017-03-21 11:40:07	2017-03-21 11:29:38	998237	200918449	991068	201	72	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363688	SRX2659133	cef6
>    > SRR5363689	2017-03-21 11:40:07	2017-03-21 11:29:26	518920	104507529	515814	201	41	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363689	SRX2659134	cef5
>    > SRR5363690	2017-03-21 11:44:06	2017-03-21 11:33:47	11530820	2311357478	11354074	200	982	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363690	SRX2659135	cef4
>    > SRR5363691	2017-03-21 11:40:07	2017-03-21 11:29:41	516125	103920037	512793	201	41	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363691	SRX2659136	cef3
>    > SRR5363692	2017-03-21 11:40:07	2017-03-21 11:30:11	890572	179326220	884948	201	65	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363692	SRX2659137	cef2
>    > SRR5363693	2017-03-21 11:40:07	2017-03-21 11:30:04	978135	196865238	971039	201	74	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363693	SRX2659138	cef1
>    > SRR5363694	2017-03-21 11:40:07	2017-03-21 11:29:57	1959249	393031721	1932180	200	138	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363694	SRX2659139	imi2
>    > SRR5363695	2017-03-21 11:40:07	2017-03-21 11:29:43	1074984	215595123	1059636	200	76	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363695	SRX2659140	imi1
>    > SRR5363696	2017-03-21 11:41:07	2017-03-21 11:29:48	1288979	258545055	1270889	200	92	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363696	SRX2659141	dor10
>    > SRR5363697	2017-03-21 11:42:06	2017-03-21 11:31:06	2908416	583478795	2868638	200	200	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363697	SRX2659142	dor9
>    > SRR5363698	2017-03-21 11:49:07	2017-03-21 11:37:59	15442120	3097410993	15225523	200	1224	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363698	SRX2659143	dor8
>    > SRR5363699	2017-03-21 11:40:07	2017-03-21 11:30:00	1718167	344971206	1697410	200	123	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363699	SRX2659144	dor7
>    > SRR5363700	2017-03-21 11:41:07	2017-03-21 11:29:57	1739828	349655178	1722126	200	125	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363700	SRX2659145	dor6
>    > SRR5363701	2017-03-21 11:45:06	2017-03-21 11:35:22	18584730	3732962863	18375556	200	1453	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363701	SRX2659146	dor5
>    > SRR5363702	2017-03-21 11:44:06	2017-03-21 11:35:13	12158676	2438787680	11987886	200	981	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363702	SRX2659147	dor4
>    > SRR5363703	2017-03-21 11:44:06	2017-03-21 11:34:44	11391713	2283748728	11219804	200	916	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363703	SRX2659148	dor3
>    > SRR5363704	2017-03-21 11:42:06	2017-03-21 11:32:45	6155766	1236497552	6086854	200	513	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363704	SRX2659149	dor2
>    > SRR5363705	2017-03-21 11:41:07	2017-03-21 11:30:25	1002355	201209957	989835	200	72	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363705	SRX2659150	dor1
>    > SRR5363706	2017-03-21 11:41:07	2017-03-21 11:30:25	752409	151419028	746795	201	59	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363706	SRX2659151	cip10
>    > SRR5363707	2017-03-21 11:44:06	2017-03-21 11:33:49	10594069	2127067244	10466132	200	870	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363707	SRX2659152	cip9
>    > SRR5363708	2017-03-21 11:41:07	2017-03-21 11:30:31	1431059	287670501	1417182	201	105	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363708	SRX2659153	cip8
>    > SRR5363709	2017-03-21 11:41:07	2017-03-21 11:30:41	1208297	242791534	1195595	200	88	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363709	SRX2659154	cip7
>    > SRR5363710	2017-03-21 11:41:07	2017-03-21 11:30:35	1370364	275356971	1355960	200	100	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363710	SRX2659155	cip6
>    > SRR5363711	2017-03-21 11:42:06	2017-03-21 11:31:24	1593545	320335049	1578109	201	116	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363711	SRX2659156	cip5
>    > SRR5363712	2017-03-21 11:41:07	2017-03-21 11:31:00	1332160	267714312	1318490	200	99	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363712	SRX2659157	cip4
>    > SRR5363713	2017-03-21 11:42:06	2017-03-21 11:31:02	1893119	379943269	1868718	200	134	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363713	SRX2659158	cip3
>    > SRR5363714	2017-03-21 11:42:06	2017-03-21 11:31:01	1839276	369651139	1820660	200	129	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363714	SRX2659159	cip2
>    > SRR5363715	2017-03-21 11:26:06	2017-03-21 10:59:33	712060	142833136	702138	200	51	assembly	https://raw.githubusercontent.com/jmchilton/galaxy-examples/master/sra_PRJNA355367/SRR5363715	SRX2659160	cip1
>    > ```
>    {: .solution}
>
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_1_paste.png)
>
>    The resulting table should look something like the following:
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_2_rules_landing.png)
>
>    Navigate to the end of the table and notice that column J is the URL target we are hoping to download for each file.
>
> 2. As in the previous exercises, please:
>     - Set this `Column definition` for the URL target
>     - Strip the header row
>
>       > <solution-title></solution-title>
>       > From **Filter** menu, select `First or Last N Rows`
>       >   - Filter first Row
>       >
>       >  From **Rules** menu, select  `Add / Modify Column Definitions`
>       >   - `Add Definition`, `URL`, Column `J`.
>       >   ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_3_end_of_table.png)
>       {: .solution}
>
>    For the analysis we wish to do, we want to group these files based on the type indicated in column `L` (LibraryName) shown below.
>    The source data though adds numbers to the library type to generate the `LibraryName`, so we need to strip those out to use the type
>    as an identifier for grouping the datasets. To do this, use the regex column adder rule again.
>
> 3. From **Column** menu, select `Using a Regular Expression`
>     - *"From Column"*: `L`
>     - Select `Create columns matching expression groups`
>     - *"Regular Expression"*: `([^\d]+)\d+`.
>     - *"Number of Groups"*: `1`
>
>    The result looks like:
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_4_regex.png)
>
>    > <comment-title>Regular expression explained</comment-title>
>    > In this regular expression, `\d` means any digit, so `[^...]` means match anything that is not inside the brackets.
>    > So together `[^\d]+` means match one or more, non digits at the start of the column and the `()` around that means capture
>    > them into a group. We've add `\d+` at the end of the expression but it isn't grouped so we are effectively ignoring the
>    > digits at the end as we had hoped.
>    {: .comment}
>
>    Now we have two columns we need to assign list identifiers for, the new column `M` for the first, outer identifier and the
>    first column `A` for the inner, second identifier.
>
> 4. From **Rules** menu, select `Add / Modify Column Definitions`
>     - `Add Definition`, `List Identifier(s)`, Column `M`
>     - Click on `... Assign Another Column`
>     - Select column `A`
>
>        > <comment-title>Re-ordering columns</comment-title>
>        > If you make a mistake in the order you select columns in you can simple use the up and down arrows to re-arrange the list
>        {: .comment}
>
>    The result should look something like this:
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_5_multiple_identifiers_edit.png)
>
> 5. Click `Apply` to return to the rule preview screen and notice there are two column listed for the list identifier definition
>   ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_6_multiple_identifiers.png)
>
> 6. Finalize your collection:
>    - *"Type"*: `txt`
>    - Name your Collection
>    - Click `Upload`
>
>    Note: set *"Type"* to `sra` if you used real data from SRA instead of the table from exercise.
>   ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_example_6_7_named.png)
>
{: .hands_on}

# Apply Rules to Existing Collections

To start this example, we will first upload a simple, flat collection of data. The data files we will use
will be the same as those used by the [DESeq2 Vignette](https://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html)
from the [Pasilla Bioconductor Package](https://bioconductor.org/packages/release/data/experiment/html/pasilla.html).

> <hands-on-title>Applying Rules to Existing Collections</hands-on-title>
>
> 1. Open the Rule Builder
>    - *"Upload data as"*: `Collection(s)`
>    - *"Load tabular data from"*: `Pasted Table`
>    - Paste the following table:
>
>    ```
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/treated1fb.txt treated_single_1
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/treated2fb.txt treated_paired_2
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/treated3fb.txt treated_paired_3
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/untreated1fb.txt untreated_single_4
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/untreated2fb.txt untreated_single_5
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/untreated3fb.txt untreated_paired_6
>    https://raw.githubusercontent.com/jmchilton/galaxy/apply_rules_tutorials/test-data/rules/untreated4fb.txt untreated_paired_7
>    ```
>
>    Next, we set up the column types:
>
> 2. From **Rules** menu, select  `Add / Modify Column Definitions`
>    - `Add Definition`, `URL`, column `A`
>    - `Add Definition`, `List Identifier(s)`, column `B`
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_2_input_rules.png)
>
> 3. Finalize the collection:
>    - *"Type"*: `txt`
>    - Give the collection a name.
>    - Click `Build`
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_4_input_list.png)
>
>    The first thing we will do to this new collection is add some levels or depth to its structure.
>    Lets assume we want to group it into "treated" and "untreated" lists and "paired" and "single"
>    sublists below that. We can do this with the `Apply Rules` collection operation tool, which will
>    likely be under the `Collection Operations` tool menu in your Galaxy interface.
>
> 4. Open the **Apply Rules** tool
>
>    The very simple interface should look something like this:
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_5_apply_rules_landing.png)
>
>    This interface simply lets one pick a collection to operate on and then launch the rule builder
>    window to work to describe and preview manipulating the metadata of that collection.
>
> 5. **Apply Rules** {% icon tool %} with the following parameters:
>    - *"Input Collection"*: the collection we just uploaded
>    - Click the `Edit` button
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_6_apply_rules_init_flat.png)
>
>    When a flat collection is used with this tool, the rule builder will initialize a default rule to pull
>    the list identifier out for each item of the collection as shown above. Next we will use regular expressions
>    to build two new columns, these columns will group the datasets into "treated" and "untreated" sublists and
>    then "single" and "paired" sublists of that. This rule is found under the `Column` menu.
>
>    We'll transform column A into some more useful columns
>
> 6. From **Column**, select `Using a Regular Expression`
>    - *"Column"*: `A`
>    - Select `Create columns matching expression groups`
>    - *"Regular Expression"*: `(.*)_(.*)_.*`
>    - *"Number of Groups"*: `2`
>
>    > <comment-title>Regular expression explained</comment-title>
>    >  Here ``.*`` means match any number of any character - so basically match anything.
>    >  The parentheses around ``.*`` means form a "group" from whatever is matched.
>    >  The ``_`` describes the literal ``_`` values in the identifier we are matching. The result is that
>    > everything before the first ``_`` will be matched as the first group and everything between the ``_``
>    > characters will be matched as the second group. Click to apply this rule and two new columns should be created.
>    >
>    > We have defined two matching groups (the `(.*)` in the regex), so the number of groups is set to `2`
>    {: .comment}
>
> 7. From  **Rules**, select `Add / Modify Column Definitions`
>    - `Add Definition`, `List Identifier(s)`, Column `B`
>    - `Assign another column`, column `C`
>    - `Assign another column`, column `A`
>    - Click `Save`
>
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_7_apply_rules_add_depth.png)
>
>    The resulting collection should have two new levels of depth for "untreated" vs "treated" and "paired" vs "single" as shown below:
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_8_nested.png)
>
>    Placing the "single" and "paired" sublists inside the "treated" and "untreated" lists was a bit arbitrary and depending on the
>    workflow these may need to be inverted at different parts of an analysis. The `Apply Rules` tool can be used to invert these levels
>    if that is needed.
>
> 8. **Apply Rules** {% icon tool %} with the following parameters:
>    - *"Input Collection"*: the new nested list we created in the previous step
>    - Click "Edit".
>
>     ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_9_apply_rules_init_nested.png)
>
>    Notice when loading a nested collection into the rule builder, there is a column for each layer of the list for each element.
>    One can pretty easily invert the outer two layers of the list by simply assigning the list identifiers in a new order. So select to assign "List Identifiers" and this time assign them to columns "B", "A", and "C" - in that order.
>
> 9. From **Rules**, select `Add / Modify Column Definitions`
>    - `Add Definition`, `List Identifier(s)`, Column `B`
>    - `Assign another column`, Column `A`
>    - `Assign another column`, Column `C`
>    - Click `Save`
>
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_10_apply_rules_inverted.png)
>
>    The resulting collection should be inverted.
>
>    ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_11_inverted.png)
>
>    In addition to structural re-organizations of a collection, the `Apply Rules` tool can be used to
>    filter elements out of the collection.
>
> 10. **Apply Rules** {% icon tool %} with the following parameters:
>     - *"Input Collection"*: the original flat list created
>     - Click `Edit`
>     - From **Filter** menu, select `Using a Regular Expression`
>       - *"From Column"*: `A`
>       - *"Regular Expression"*: `.*_single_.*`
>     - From **Rules** menu, select `Add / Modify Column Definitions`
>       - `Add Definition`, `List Identifier(s)`, Column `A`
>
>     ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_12_apply_rules_filter.png)
>
>     The resulting collection should be a filtered version of the original list.
>
>     ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_13_filtered.png)
>
>     Structural re-organizations of collections can also be combined with filtering. To demonstrate this, reopen the original flat list created for this example again in the rule builder of the `Apply Rules` tool. Use the same regular expression as last time to filter the result but also add a column for "treated" and "untreated" list identifiers.
>
> 11. **Apply Rules** {% icon tool %} with the following parameters:
>     - From **Filter** menu, select `Using a Regular Expression`
>       - *"From Column"*: `A`
>       - *"Regular Expression"*: `.*_single_.*`
>     - From **Column** menu, select `Using a Regular Expression`
>       - *"From Column"*: `A`
>       - Select `Create columns matching expression groups`
>       - *"Regular Expression"*:  `(.*)_single_.*`
>       - *"Number of Groups"*: `1`
>     - From **Rules** menu, select  `Add / Modify Column Definitions`
>       - `Add Definition`, `List Identifier(s)`. Column `B`
>       - `Assign another column`, Column `A`
>
>     ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_14_apply_rules_filtered_and_nested.png)
>
>     The resulting collection should be a filtered to only include the "single" data and broken into "treated" and "untreated" sublists.
>
>     ![screenshot](https://galaxy-tests.s3.amazonaws.com/galaxy-gtn-screenshots/local/rules_apply_rules_example_4_15_filtered_and_nested.png)
{: .hands_on}


topic: galaxy-interface
tutorial: galaxy-intro-jupyter
title: Use Jupyter notebooks in Galaxy
difficulty: introductory
time_estimation: 1H
tools_json: ["JupyterLab", "interactive_tool_jupyter_notebook"]
requirements_json: ["type: internal", "history"]
tags_json: []
content_hash: d184069f01a06723627d311a49c41e7b
last_modified: 2026-05-27T09:01:48.661640
zenodo_link: https://zenodo.org/record/15263830
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/galaxy-intro-jupyter/tutorial.html

content:
In this tutorial, we are going to explore the basics of using JupyterLab in Galaxy. We will use the Gapminder data as a test set to get the hang of Jupyter notebooks. The [python-novice-gapminder-data.zip](https://swcarpentry.github.io/python-novice-gapminder/files/python-novice-gapminder-data.zip) file is publicly available. This tutorial can also be used as an initial setup for the Software Carpentries training [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/instructor/index.html).


> <agenda-title></agenda-title>
>
> In this tutorial, we will see :
>
> 1. TOC
> {:toc}
>
{: .agenda}


# What is Jupyter?

[Jupyter](http://jupyter.org/) is an interactive environment that mixes explanatory text, command line and output display for an interactive analysis environment. Its implementation in Galaxy facilitates the performance of additional analyses if there are no tools for it.

These notebooks allow you to replace any in-house script you might need to complete your analysis. You don't need to move your data out of Galaxy. You can describe each step of your analysis in the markdown cells for an easy understanding of the processes, and save it in your history for sharing and reproducibility. In addition, thanks to [Jupyter magic commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html), you can use several different languages in a single notebook.

![Jupyter notebook](../../images/notebook_overview.png)

You can find the complete manual for Jupyter commands [on Read the Docs](http://jupyter-notebook.readthedocs.io/en/stable/).

# Use Jupyter notebook in Galaxy

## Import data
To manipulate data, we first upload the [python-novice-gapminder-data.zip](https://swcarpentry.github.io/python-novice-gapminder/files/python-novice-gapminder-data.zip) folder into your Galaxy history. To add the files, you can either upload them locally from your computer or use Zenodo.
To add the files, you can either upload them locally from your computer or use Zenodo.

> <hands-on-title>Data upload using Zenodo</hands-on-title>
>
> 1. Create a new history for this Jupyter notebook exercise
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Import the following tabular file from [Zenodo]({{ page.zenodo_link }}):
>
>    ```text
>    {{ page.zenodo_link }}/files/gapminder_all.csv
>    {{ page.zenodo_link }}/files/gapminder_gdp_africa.csv
>    {{ page.zenodo_link }}/files/gapminder_gdp_americas.csv
>    {{ page.zenodo_link }}/files/gapminder_gdp_asia.csv
>    {{ page.zenodo_link }}/files/gapminder_gdp_europe.csv
>    {{ page.zenodo_link }}/files/gapminder_gdp_oceania.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>
> 3. Make sure the files are imported as `CSV` by expanding the box of each imported file in your history and check the format.
> 
> {% snippet faqs/galaxy/datasets_change_datatype.md datatype="csv" %}
>
{: .hands_on}



## Open the JupyterLab environment

Opening up your {% tool [JupyterLab](interactive_tool_jupyter_notebook) %}:

{% snippet faqs/galaxy/interactive_tools_jupyter_launch.md %}

You should now be looking at a page with the JupyterLab interface:

![Jupyterlab default session](../../images/jupyterlab_default_session.png)

As shown in the figure above, JupyterLab interface is made of 3 main areas:
- The menu bar at the top
- The left side bar with, in particular, the **File Browser**
- The main work area in the central panel


## Start your first notebook

Now that we are ready to start exploring JupyterLab, let's open a python Notebook. There will be one pre-opened notebook available in the file browser on the left side called **ipython_galaxy_notebook.ipynb**. For this training, however, we will open a new Jupyter notebook, select a kernel and give it a name.

> <hands-on-title>Start a notebook</hands-on-title>
> 
> 1. Open a new Jupyter notebook 
> 
>    There is more than one option to open a Jupyter notebook. One option is:   
>  
>    - Click on **File** in the top menu bar
>    - Select **New** -> **Notebook**
>    - Choose the kernel `Python 3 (ipykernel)` from the dropdown menu and click **select**
>
> 2. Change the kernel (only if you run `JupyterLab` with Galaxy Version 1.0.1)
>
>    If the kernel `Python [conda env:python-kernel-3.12]` was chosen in the previous step, it should appear in the upper right corner of the notebook file. If not, this is the location where the kernel can be switched. 
>  
>    - Click on the field that displays the current kernel e.g. `Python [conda env:base]`
>    - Now select the kernel `Python [conda env:python-kernel-3.12]` from the drop down menu
>
> 3. Name the Jupyter notebook
>
>    There are several options how to name or rename your Jupyter Notebook. One way is to: 
>  
>    - Click on **File** and select **Save File As...**
>    - Enter your file name e.g. `first_galaxy_notebook.ipynb`. Note that your file needs to end with `.ipynb`
>    - Click on **Save**
>
{: .hands_on}

## Install and import libraries using Conda

Some dependencies or programming libraries may not be available in the kernel your Jupyter Notebook is using. If that's the case, you can install or update these libraries using **Conda**.


> <hands-on-title>Install from a Conda recipe</hands-on-title>
>
> 1. Click on a cell of your notebook to edit it (verify that it is defined as a `Code` cell)
> 2. Enter the following lines : 
>    ```text
>!conda install -y pandas
>!conda install -y seaborn
>    ```
> - The `!` indicates you are typing a bash command line (alternatively, you can add the line `%%bash` at the beginning of your cell. In that case, the whole cell will be run as bash commands.)
>   - The `-y` option allows the installation without asking for confirmation  (The confirmation is not managed well by notebooks)
> 3. `shift`+`return` to run the cell or click on the run cell button.
>
{: .hands_on}

If you wish to follow the Software Carpentries training [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/instructor/index.html) after finishing this Jupyter Notebook introduction, you should install the following Python libraries.

> <hands-on-title>Install Python libraries for a Python introduction</hands-on-title>
>
> 1.  Copy the following install commands and paste them into an empty cell of your Notebook
>    ```text
>!conda install -y math
>!conda install -y matplotlib
>!conda install -y glob
>!conda install -y pathlib
>    ```
>
> 2. press `shift`+`return` to run the cell or click on the run cell button.
>
{: .hands_on}

Now you will be able to import these Python libraries and use them with your library code. 

> <hands-on-title>Import Python libraries</hands-on-title>
>
> 1. Click on a cell of your notebook to edit it (verify that it is defined as a `Code` cell)
>
> 2. Enter the following lines : 
>    ```python
>import pandas as pd
>import seaborn as sns
>from IPython.display import display
>import matplotlib.pyplot as plt
>    ```
> 3. `shift`+`return` to run the cell or click on the run cell button.
>
{: .hands_on}



## Import data 
If you want to include datasets from your history into your  Jupyter notebook, you can import them  using the `get("[file_number]")` command, with the number of your dataset in the history (If you are working on a collection, unhide datasets to see their numbers).
You can use the `gapminder_gdp_europe.csv` file. Using the `get("[file_number]")` function will retrieve or get the location where the dataset from your galaxy history (with the `file_number`) is stored. Within your Python program this file can be read using the location retrieved the `get()` function.

![History with dataset to import](../../images/jupyterlab_history.png)

Looking at the history in the image you would need to call `get("7")` to get the file path of the `gapminder_gdp_europe.csv` file.

> <hands-on-title>Import a file location from your history</hands-on-title>
>
> Save the import file location to a variable (`file_import`) name with the `get()` function.
>    ```python
>file_import = get("[file_number]")
>    ```
>    - The files are referenced in Jupyter by their number in the history.
>    - The variable `file_import` now stores the location where your file can be imported from.
>
{: .hands_on}

If you wish to follow the python training later you should import all of the gapminder datasets now. The `file_import` variables can be treated like the path with the tutorial. 

> <hands-on-title>Import the training data locations from your history</hands-on-title>
>
> 1. Find the `file_numbers` of all of the gapminder datasets (`gapminder_all.csv`, `gapminder_gdp_africa.csv`, `gapminder_gdp_americas.csv`, `gapminder_gdp_asia.csv`, `gapminder_gdp_europe.csv`, `gapminder_gdp_oceania.csv`) with your galaxy history
> 2. Load the datasets `file_import`into your Ipython notebook:
>    ```python
>gapminder_all_import = get("[file_number]")
>gapminder_gdp_africa_import = get("[file_number]")
>gapminder_gdp_americas_import = get("[file_number]")
>gapminder_gdp_asia_import = get("[file_number]")
>gapminder_gdp_europe_import = get("[file_number]")
>gapminder_gdp_oceania_import = get("[file_number]")
>    ```
>    - The files are referenced in Jupyter by their number in the history.
>    - The variable `file_import` now stores the location where your file can be imported from
>
{: .hands_on}



## Graph Display in Jupyter

In this tutorial, we are going to plot a distribution graph of our data. For this, we will first need to load one of our tabular data files. You can use the `gapminder_all.csv` file.

> <hands-on-title>Load a file from your history</hands-on-title>
>
> 1. Open the dataset as a pandas DataFrame with the function.
>    ```python
>dataframe = pd.read_csv(file_import)
>    ```
>    - The files are referenced in Jupyter by their number in the history.
>
{: .hands_on}


> <hands-on-title>Draw a distribution plot</hands-on-title>
>
> 1. Create your figure with the command 
>    ```python
>fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(15, 10))
>    ```
>   -  `nrows=1, ncols=1` means you will have one plot in your figure (one row and one column)
>   -  `figsize` parameter determines the size of the figure
> 2. Draw the distribution plot of the second column of our dataset with the command
>    ```python
>sns.histplot(data=dataframe, x='gdpPercap_1997',  kde=True, ax=ax)
>    ```
> ![Distribution plot in Jupyter](../../images/jupyter_plot.png)
>
{: .hands_on}


## Export Data


If you want to save a file you generated in your notebook, use the `put("file_name")` command. That is what we are going to do with our distribution plot.

> <hands-on-title>Save a Jupyter-generated image into a Galaxy History</hands-on-title>
>
> 1. Create an image file with the figure you just drew with the command
>    ```python
>fig.savefig('distplot.png')
>    ```
> 2. Export your image to your history with the command  
>    ```python
>put('distplot.png')
>    ``` 
>
{: .hands_on}


## Save the Notebook in your history

Once you are done with your analysis or anytime during the editing process, you can save the notebook to your history using the `put("first_galaxy_notebook.ipynb")`. If you create additional notebooks with different names, make sure you save them all before you quit JupyterLab.

This will create a new notebook `.ipynb` file in your history every time you click on this icon.

> <hands-on-title>Closing JupyterLab</hands-on-title>
>
> 1. In the Galaxy interface, click on  **Interactive Tools** button on the left side.
>
> 2. Tick {% icon galaxy-selector %} the box of your Jupyter Interactive Tool, and click **Stop**.
>
{: .hands_on}

If you want to reopen a Jupyter Notebook saved in your history, you can use the tool  **Interactive JupyterLab Notebook**, select "Load a previous Notebook", and select the notebook from your history. 


# Conclusion


{% icon trophy %} You have just performed your first analysis in a Jupyter notebook integrated environment in Galaxy. You generated a distribution plot that you saved in your history along with the notebook to generate it.
If you wish to follow the Software Carpentries training [Plotting and Programming in Python](https://swcarpentry.github.io/python-novice-gapminder/instructor/index.html) training now, you can open a Jupyter notebook install all needed dependencies and upload all file locations from the Gapminder dataset using the `get('file_number')` function (e.g. `gapminder_all_file = get(12)`). You can use this file location now throughout the tutorial once you need to specify the file path. Meaning, if the tutorial asks you to load a dataset like this `data_oceania = pd.read_csv('data/gapminder_gdp_oceania.csv')` you can replace the path `'data/gapminder_gdp_oceania.csv'` with our file_import variable: `data_oceania = pd.read_csv(gapminder_gdp_oceania_import)`.
You can start directly from [**The-jupyterlab-interface**](https://swcarpentry.github.io/python-novice-gapminder/instructor/01-run-quit.html#the-jupyterlab-interface) section of the tutorial.


topic: galaxy-interface
tutorial: history
title: Understanding Galaxy history system
difficulty: introductory
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 302e08fdcdc9ecf543038fcd7d6e5d9d
last_modified: 2026-05-27T09:01:48.661769
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/history/tutorial.html

content:
{% snippet faqs/gtn/galaxy_tested_with.md version="24.1" %}

When data is uploaded from your computer or analysis is done on existing data using Galaxy, each output from those steps
generates a dataset. These datasets (and the output datasets from later analysis on them) are stored by Galaxy in
**Histories**.

# The history panel

All users have one 'current' history, which can be thought of as **a workspace** or **a current working directory** in
bioinformatics terms. Your current history is displayed on the right-hand side of the main 'Analyze Data' Galaxy page in
what is called the history panel.

![Screenshot of the Galaxy UI with the toolbox, center panel, and history visible](../../images/history.png "Galaxy History is simply the right pane of the interface. It lists all datasets uploaded or produced during an analysis. Galaxy was designed around history with the idea of <i>reproducibility</i> in mind!")

The history panel displays output datasets in the order in which they were created, with the oldest/first shown at the
bottom. As new analyses are done and new output datasets are generated, the newest datasets are added to the top of the
history panel. In this way, the history panel displays the history of your **analysis over time**.

**Users that have registered an account and logged in can have many histories** and the history panel allows switching
between them and creating new ones. This can be useful to organize different analyses.

**Anonymous users** (if your Galaxy instance allows them) are users that have not registered an account. Anonymous users are
only allowed one history. Users are encouraged to register and log in with the benefit that they can work on many histories and switch between them.

> <warning-title>Anonymous Users: Beware</warning-title>
> The histories of anonymous users are only associated through your browser's session. **If you close the browser or
> clear you sessions - that history will be lost!** We can not recover it for you if it is.
{: .warning}

## Global history controls

{% snippet faqs/galaxy/histories_top_control_buttons.md box_type="none"%}

### {% icon new-history %} "**Create new history**"

{% snippet faqs/galaxy/histories_create_new.md box_type="none"%}

### {% icon switch-histories %} "**Switch to history**"

{% snippet faqs/galaxy/histories_switch.md box_type="none"%}

### {% icon galaxy-history-options %} "**History options**" 

{% snippet faqs/galaxy/histories_options.md box_type="none"%}

# History manipulation

## Renaming a history

All histories begin with the name 'Unnamed history'. Non-anonymous users can rename the history as they see fit:

{% snippet faqs/galaxy/histories_rename.md box_type="none"%}

## Tagging a history

{% snippet faqs/galaxy/histories_tagging.md box_type="none"%}

## Annotating a history

{% snippet faqs/galaxy/histories_annotation.md box_type="none"%}

## Undeleting ... deleted histories

{% snippet faqs/galaxy/histories_undelete.md box_type="none"%}




## History size, storage selection, and views

The lower part of the history header contains a number of buttons:

![Buttons for figuring out history size, selecting storage location, and controlling views](../../../../shared/images/history_size_storage_views.png "Using these buttons, one can obtain information about history's disk footprint, change storage options, and control dataset views.")

> <warning-title>Some buttons are instance-specific</warning-title>
> The list of buttons shown above may vary depending on which Galaxy instance you are using. For example, at the time of writing, the **Preferred storage** {% icon galaxy-history-storage-choice %} button is not available on all Galaxy instances.
{: .warning}

- {% icon galaxy-history-size %} **History size** - shows history storage overview in the central pane of the interface.
- {% icon galaxy-history-storage-choice %} **Preferred storage location** - allows users to specify where history datasets will be stored. It is an optional feature when you have multiple storage locations available. The button might therefore not be visible on all Galaxy instances.
- {% icon galaxy-show-active %} **Show active** - shows active (non-deleted and non-hidden) datasets in the history.
- {% icon galaxy-delete %} **Include deleted** - includes deleted datasets into the history view. If you delete a dataset, it does not disappear unless you explicitly purge it.
- {% icon galaxy-show-hidden %} **Include hidden** - include hidden datasets into the history view. Any dataset in history can be hidden. For example, workflow executions frequently hide intermediate datasets so that they do not complicate the history view. It is a way to hide non-important datasets from view. 

# History datasets

So far, we have only discussed functions and controls affecting the *entire* history. Yet history is a collection of datasets. Now it is time to discuss the interface elements of individual datasets.  

## Datasets can be individual or bundled into collections

A history dataset can exist by itself, as an independent entity, or as a part of a **collection**. Collections make it possible to analyze datasets with hundreds of thousands of samples.

{% snippet faqs/galaxy/histories_datasets_vs_collections.md box_type="none"%}

### Creating a dataset collection


{% snippet faqs/galaxy/collections_build_list.md box_type="none"%}

### Creating a paired collection

{% snippet faqs/galaxy/collections_build_list_paired.md box_type="none"%}




## United Colors of Galaxy: Dataset states

{% snippet faqs/galaxy/histories_dataset_colors.md box_type="none"%}

## Dataset snippet in detail

{% snippet faqs/galaxy/histories_dataset_item.md box_type="none"%}

## Managing Datasets Individually

### Hiding and unhiding datasets

Datasets in a Galaxy history can be hidden. This is useful for reducing the complexity of a history. For example, some intermediate datasets generated during an analysis or workflow execution are not important, and there is no need to see them. 

#### Hiding datasets

{% snippet faqs/galaxy/datasets_hidden.md box_type="none"%}

#### Unhiding datasets

{% snippet faqs/galaxy/datasets_unhidden.md box_type="none"%}

### Deleting and undeleting datasets

You can **delete** any dataset in your history. Unless you explicitly tell Galaxy to delete a dataset permanently (see below), this does not immediately remove the dataset from Galaxy **it is reversible**. When you delete a dataset from the history, it will be removed from the pane (just like hidden datasets).

#### Deleting datasets

{% snippet faqs/galaxy/datasets_deleting.md box_type="none"%}

#### Undeleting datasets

{% snippet faqs/galaxy/datasets_undelete.md box_type="none"%}

> <comment-title>Admins may purge your deleted datasets</comment-title>
> Depending on the policy of your Galaxy server, administrators may run scripts that search for and purge the
> datasets you've marked as deleted. Often, deleted datasets and histories are purged based on the age of the deletion
> (e.g. datasets that have been marked as deleted for 90 days or more). Check with the administrators of your Galaxy instance to
> find out the policy used.
{: .comment}

## Tagging datasets

{% snippet faqs/galaxy/datasets_add_tag.md box_type="none"%}

## Managing Multiple Datasets Easily

{% snippet faqs/galaxy/datasets_multiple.md box_type="none"%}

<!-- Including search to deduplicate -->
{% include topics/galaxy-interface/tutorials/search/search.md %}


topic: galaxy-interface
tutorial: onedata-byos
title: Onedata user-owned storage
difficulty: introductory
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: ["storage"]
content_hash: 3b7d104b40f65812835b730a728f9709
last_modified: 2026-05-27T09:01:48.662186
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/onedata-byos/tutorial.html

content:
> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Prerequisites

1. This tutorial assumes that you have basic knowledge about Onedata and access 
   to a Onedata ecosystem. If needed, follow 
   [this tutorial]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html %})
   first!
2. To use Onedata as user-owned storage, you need the **domain** of the
   **Onezone service**, a **Space** name, and a suitable **access token**. Here
   is the relevant 
   [guide on how to get them]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#accessing-onedata-services %}). 
3. The Galaxy server must be properly configured by the admins for the Onedata
   Storage Location (BYOS) templates to be available. Here is the 
   [corresponding tutorial]({% link topics/admin/tutorials/onedata-configuration/tutorial.html %}).


# Introduction

Your account in Onedata can be used to store your Galaxy datasets, effectively
extending your quota. Onedata then serves as a so-called **Storage Location**,
acting as an **Object Store** for the Galaxy server. Below hands-on tutorial
will help you configure and use a Onedata Storage Location using the so-called
Bring Your Own Storage (BYOS) approach.


> <tip-title>Disambiguation</tip-title>
> While Onedata can be used for both, a **Storage Location** is not the same as
> a **Remote File Source**. In this tutorial, you will be setting up a
> Onedata-based **Storage Location**, which allows storing your Galaxy datasets
> directly in a Onedata Space in a transparent way. If you are looking to use a
> Remote File Source, refer to 
> [this tutorial]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %}).
{: .tip}


# Configuration

Follow these steps:

1. Log in to your Galaxy account.
2. Go to the **Manage Your Storage Locations** section of the **Preferences** menu.
3. Click **Create** in the top right corner to create a new Storage Location.
   ![Create a Remote File Source](../../images/onedata-byos/configure-create-sl.png)
4. Choose the **Onedata Storage** template. If there is no such template, the 
   Galaxy server is not configured to support it. Consider contacting its admins.
5. Fill in the information, following the hints visible on the form.
   ![Configure Onedata Storage Location](../../images/onedata-byos/configure-onedata-sl.png)
6. Click **Create** to finalize.

You have now configured a new Storage Location, but you still need to tell
Galaxy your preferences so that it is effectively used among different available
options:

1. Go to the **Preferred Storage Location** section of the **Preferences** menu.
2. Choose the newly added Storage Location:
   ![Prefer Onedata Storage Location](../../images/onedata-byos/preferred-storage-location.png)


# See it in action

Upload some new data to Galaxy, or run a workflow to produce results. Then
navigate to your Onedata account (e.g. [https://datahub.egi.eu](https://datahub.egi.eu))
and open the Space (and the path) that you have put down in the config. You
should see the Galaxy data:

![Data in Onedata](../../images/onedata-byos/data-in-onedata.png)


# Troubleshooting

If you experience any problems, take a look at the 
[troubleshooting]({% link topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html#troubleshooting %}) 
guide.


topic: galaxy-interface
tutorial: workflow-fairification
title: Annotate, prepare tests and publish Galaxy workflows in workflow registries
difficulty: intermediate
time_estimation: 2H
requirements_json: ["type: internal"]
tags_json: ["workflows", "FAIR"]
content_hash: 7d4acbe1da68a7caeed51939d4836af0
last_modified: 2026-05-27T09:01:48.666556
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/workflow-fairification/tutorial.html

content:
Research data is accumulating at an unprecedented rate, presenting significant challenges for achieving fully reproducible science. As a result, implementing high-quality management of scientific data has become a global priority. One key aspect of this effort is the use of computational workflows, which describe the complex, multi-step methods used for data analysis. In Galaxy, workflows are a powerful feature that allows researchers to link multiple steps of complex analyses seamlessly. To maximize their impact, these workflows should adhere to best practices that make them **FAIR: Findable, Accessible, Interoperable, and Reusable**.

The FAIR principles —-Findable, Accessible, Interoperable, and Reusable—- provide practical guidelines for enhancing the value of research data:

- **Findable**: Easy to locate through rich metadata and unique identifiers.
- **Accessible**: Stored in a way that allows them to be retrieved by those who need them.
- **Interoperable**: Usable across different systems and platforms without extensive adaptation.
- **Reusable**: Well-documented and clearly licensed to enable reuse in different contexts.
> <comment-title></comment-title>
> You can learn more about FAIR Data, Workflows, and Research in our [dedicated topic with numerous tutorials]({% link topics/fair/index.md %})
{: .comment}
Applying these principles to workflows is equally important for good data management:

- **Enhanced Discoverability**: Well-annotated and documented workflows are easier to find, making them more likely to be used and cited.
- **Improved Reproducibility**: Standardized and tested workflows ensure that analyses can be reproduced, validating research findings.
- **Community Collaboration**: Sharing workflows through centralized registries fosters collaboration and innovation within the bioinformatics community.
- **Sustainability**: Regular updates and versioning ensure that workflows remain compatible with the latest tools, extending their lifespan and utility.

While making a workflow FAIR might seem complicated ({% cite wilkinson2025applying %}, {% cite goble2020fair %}), publications like "Ten quick tips for building FAIR workflows" ({% cite de2023ten %}) provide practical guidelines to simplify the process:

![This image provides ten quick tips for building FAIR workflows, focusing on findability (registering the workflow and describing it with rich metadata), accessibility (making source code available in a public repository and providing example input data and results), interoperability (adhering to file format standards and making the workflow portable), and reusability (providing a reproducible computational environment, adding a configuration file with defaults, modularizing the workflow, and offering clear documentation).](./images/journal.pcbi.1011369.g001.PNG "Ten quick tips for building FAIR workflows. Source: {% cite de2023ten %}")

Using Galaxy as a workflow management system, many of these tips (all related to **Interoperability** and **Reusability**) are already fulfilled:
- Tip 5 (**Interoperability**): *"The tools integrated in a workflow should adhere to file format standards."*
- Tip 6 (**Interoperability**): *"Make the workflow portable."*
- Tip 7 (**Reusability**): *"Provide a reproducible computational environment to run the workflow."*
- Tip 8 (**Reusability**): *"Add a configuration file with defaults."*
- Tip 9 (**Reusability**): *"Modularize the workflow."*

In this tutorial, we will demonstrate how to fulfill the remaining tips:
- Tip 1 (**Findability**): *"Register the workflow."*
- Tip 2 (**Findability**): *"Describe the workflow with rich metadata."*
- Tip 3 (**Accessibility**): *"Make source code available in a public code repository."*
- Tip 4 (**Accessibility**): *"Provide example input data and results along with the workflow."*
- Tip 10 (**Reusability**): *"Provide clear and concise workflow documentation."*

To illustrate the process, we will use a simple workflow with 2 steps as an example.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Prepare the workflow

Here, we will use a workflow running Falco (FastQC alternative, {% cite de2021falco %}) and MultiQC ({% cite ewels2016multiqc %}) but you can use your own workflow.

> <hands-on-title>Create the workflow into Galaxy</hands-on-title>
>
> 1. Go to the workflow page
> 2. Create a new workflow
>
>    {% snippet faqs/galaxy/workflows_create_new.md %}
>
> 3. Add a single data input
>
> 4. Add {% tool [Falco](toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0) %} and {% tool [MultiQC](toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3) %}
>
> 5. Connect the input to **Falco**
> 6. Define **MultiQC** parameter
>    - **Which tool was used generate logs?**: `FastQC`
> 
> 7. Connect `text_file` **Falco** to **MultiQC** input
{: .hands_on}

# Annotate the Galaxy workflow with essential metadata

The first step to FAIRify a Galaxy workflow is to fulfill Tip 2 (**Findability**): *"Describe the workflow with rich metadata"*. Describing the workflow with rich metadata helps both humans and machines understand its purpose, facilitating discovery by search engines. Galaxy allows to associate workflows with metadata directly in its interface.

> <hands-on-title>Open workflow attributes</hands-on-title>
>
> 1. Click on **Attributes** on the left side of the workflow editor (or right side for older Galaxy version for clicking on {% icon galaxy-pencil %} **Edit Attributes**)
{: .hands_on}

Some workflow metadata should appear.

> <question-title></question-title>
>
> 1. What is the name of the workflow?
>
> > <solution-title></solution-title>
> > 1. "Unnamed Workflow"
> {: .solution}
{: .question}

Metadata should include details about the workflow's components and clearly outline its purpose, scope, and limitations. This ensures the workflow is easily findable and understandable. 

> <question-title></question-title>
>
> Which metadata is supported in Galaxy workflow interface?
>
> > <solution-title></solution-title>
> > Galaxy workflow interface supports:
> > - Name
> > - Version
> > - Annotation: These notes will be visible when this workflow is viewed.
> > - License
> > - Creator: Either a person or an organization. 
> > - Tags: Apply tags to make it easy to search for and find items with the same tag. 
> {: .solution}
{: .question}

Galaxy workflow interface supports some metadata. Are they enough to fulfill Tip 2 and also Tip 1 *"Register the workflow."*? WorkflowHub, a workflow registry we will explain more later, supports the [following metadata](https://about.workflowhub.eu/docs/metadata-list/):

Name | Description | Mandatory
--- | --- |
**Title** | This field is mandatory and is with some workflow types pre-filled with the title of the workflow. | Yes
**Description** | If a CWL (abstract) file is given, the description will be parsed automatically out of the doc attribute. In any other case this field can be used to write some documentation that will be shown on the workflow page. | No
**Source** | If the workflow came from an external repository (i.e. GitHub), you can include its original URL here. | No
**Maturity** | This field can be used to specify in which maturity state the workflow is. The two available options are: `work-in-progress`, `stable` | No
**Teams** | Every workflow registration is linked to one or more teams. | Yes
**Licence** | The standard licence is Apache Software Licence 2.0. If you did not make the workflow yourself, be sure that the licence corresponds to the licence where you took the workflow from (for example github licences). | No
**Sharing** | Specify who can view the summary, get access to the content, and edit the Workflow. This is possibly already filled in according to the selected project. | No
**Tags** | Choose an appropriate tag for your workflow. Please check if your tag is already available and use the existing one if so. If you make a new tag, keep it simple without capitals or spaces. For example all new covid-19 workflows need to be tagged with covid-19. | No
**Creators** | This is an important section where all the people that were involved in making / publishing this workflow are listed. | No

> <question-title></question-title>
>
> 1. Which WorkflowHub metadata is supported in Galaxy workflow interface?
> 2. Which values should we put for our workflow?
>
> > <solution-title></solution-title>
> >
> > Workflow metadata | Galaxy metadata | Mapping workflow value
> > --- | --- | ---
> > Title | Name | Quality control and mapping
> > Description | Annotation | Workflow runs quality control and mapping of paired-end short-reads data.
> > Source | *Not supported in Galaxy workflow interface* | 
> > Teams | *Not supported in Galaxy workflow interface* | 
> > License | License | MIT License
> > Sharing | *Supported in Galaxy workflow interface but not in the same way as WorkflowHub* | 
> > Tags | Tags | `sequence-analysis`, `mapping`
> > Creators | Creator | 
> >
> {: .solution}
{: .question}

Let's annotate the workflow

> <hands-on-title>Add metadata to the workflow</hands-on-title>
>
> 1. Add a proper title in **Name**
> 2. Add a workflow description in **Annotation**
> 3. Select a license in **License**
>
>    > <comment-title></comment-title>
>    > The most appropriate **license** are **MIT** or **GNU Affero** (if you want a [copyleft](https://en.wikipedia.org/wiki/Copyleft)).
>    {: .comment}
>
> 4. Add creators in **Creator** with their name and a unique identifier (typically an orcid.org ID)
>
>    > <comment-title></comment-title>
>    > Do not forget to click on **Save**
>    {: .comment}
>
> 5. Add tags in **Tags**
> 6. Save the workflow
{: .hands_on}

Tip 2 is now fulfilled. We can now move toward the other recommendations:

- Tip 1 (**Findability**): *"Register the workflow."*
- Tip 3 (**Accessibility**): *"Make source code available in a public code repository."*
- Tip 4 (**Accessibility**): *"Provide example input data and results along with the workflow."*
- Tip 10 (**Reusability**): *"Provide clear and concise workflow documentation."*

   The workflow annotation provides a short and concise description of the workflow. So Tip 10 is partially fullfilled. In addition to the workflow annotation, each step could annotated and [commented](https://galaxyproject.org/news/2024-04-26-workflows-workflows-workflows/#-workflow-comments) so users could have an idea about the purpose of the steps

   > <details-title>Add step descriptions to fulfill Tip 10</details-title>
   > 
   > Let's add label and annotation to each step and comments to the workflow.
   >
   > > <hands-on-title>Add tool label and annotation and comments</hands-on-title>
   > >
   > > 1. Add tool label and step annotation to **Falco**
   > >    1. Click on **Falco**
   > >    2. Fill in **Label** with `Quality control` in the right panel
   > >    3. Fill in **Step Annotation** with `This step uses Falco (FastQC alternative) to generate statistics of raw reads quality including basic statistics, per base sequence quality, per sequence quality scores, adapter content, etc.`
   > >    4. Save the workflow
   > > 2. Add tool label and step annotation to **MultiQC**
   > > 3. Add comments and annotations to workflow to create an high resolution image of the Galaxy Workflow by following the [dedicated tutorial]({% link topics/galaxy-interface/tutorials/workflow-posters/tutorial.md %})
   > {: .hands_on}
   >
   {: .details}

Before we care about Tip 3 and 4, lets use the build-in Galaxy Wizard to check if our workflow adheres to best practices of Galaxy workflows.

# Adhere to best practices for Galaxy workflows

Galaxy has a build-in Wizard to check against [community developed best practice workflows](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html). It will recommend to use a License, add Authors etc but also ensures that workflows are easy to test, that they are usable as subworkflows and invocation reports, and can be consumed easily via an API. The reusablility will be greatly enhanced if you stick to those recommendations.

> <hands-on-title>Check alignment with best practices</hands-on-title>
>
> 1. Click on {% icon galaxy-wf-best-practices %} **Best Practices** on the right
>
>    {% snippet faqs/galaxy/workflows_best_practices.md %}
>
{: .hands_on}

A side panel will open showing a review of the best practices with
- Green checks indicating that certain elements are already correctly defined (e.g., creator information, license).
- Orange warnings highlighting potential issues such as missing annotations, unconnected inputs, or other workflow problems.

By following the steps with orange warnings and checking the side panel, we will ensure our **workflow adheres to best practices** and is **ready for public use**. 

> <question-title></question-title>
>
> How many orange warnings do we have?
>
> > <solution-title></solution-title>
> >
> > 2
> >
> {: .solution}
{: .question}

The first warning is `Some workflow inputs are missing labels and/or annotations`. To follow best practices, all inputs should be explicit (with labelled input nodes) and tool steps should not have disconnected data inputs (even though the GUI can handle this) or consume workflow parameters. Older style "runtime parameters" should only be used for post job actions and newer type workflow parameter inputs should be used to manipulate tool logic.

> <hands-on-title>Add label to input</hands-on-title>
>
> 1. Click on **Input dataset: Missing a label and annotation**
> 2. Fill in **Label** in the `1: Input dataset` side panel that opened on the right
> 3. Save the workflow
> 4. Check the best practices
{: .hands_on}

> <question-title></question-title>
>
> Is it okay for the input?
>
> > <solution-title></solution-title>
> >
> > The input is missing an annotation 
> >
> {: .solution}
{: .question}

Let's add annotation to input.

> <hands-on-title>Add annotation to input</hands-on-title>
>
> 1. Click on **Input dataset: Missing a label and annotation**
> 2. Fill in **Annotation** in the `1: Input dataset` side panel that opened on the right
> 3. Save the workflow
> 4. Check the best practices
{: .hands_on}

We now have a green check for the input. The second orange warning mentions `This workflow has no labeled outputs, please select and label at least one output.` 

As for inputs, workflows should define explicit, labeled outputs. While Galaxy does not require this, declaring and labeling outputs offers significant advantages. A workflow with clearly defined outputs provides an explicit interface, making it easier to generate reports, test the workflow, and document them.

> <hands-on-title>Add labels to outputs</hands-on-title>
>
> 1. Rename labels for all outputs to add tool name (e.g. Falco `text_file` becomes `falco_text_file`)
> 2. Check boxes on the left of HTML outputs of both Falco and MultiQC in the middle pannel
> 3. Save the workflow
> 4. Check the best practices
{: .hands_on}

> <question-title></question-title>
>
> Is the workflow following all best practices?
>
> > <solution-title></solution-title>
> >
> > Yes!
> >
> {: .solution}
{: .question}

After confirming that all the best practices are applied, **make the workflow public**. This allows other users to access, reuse, and share the workflow in the Galaxy community.

> <hands-on-title> Make the workflow public</hands-on-title>
> 1. Make the workflow public
>
>    {% snippet faqs/galaxy/workflows_publish.md %}
{: .hands_on}

# Prepare example input data and results

To fulfill Tip 4 (*"Provide example input data and results along with the workflow."*), we will **create test data** that can be used **to test the workflow**. The data should be small and simple, so it can interact with the workflow and generate results without excessive processing time. 

> <hands-on-title>Create an example Galaxy History</hands-on-title>
>
> 1. **Create a new Galaxy history** for this analysis.
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. **Rename the history**.
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% icon galaxy-upload %} **Upload** test data to the newly created history from the following link:
>
>    ```
>    https://zenodo.org/record/3977236/files/female_oral2.fastq-4143.gz
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 4. {% icon galaxy-eye %} **Preview the datasets** to verify that the uploaded data looks correct and is ready for analysis.
>
> 5. Run the workflow using the uploaded test data.
>
>    {% snippet faqs/galaxy/workflows_run.md %}
>
> 4. Make the **history public**
>
>    {% snippet faqs/galaxy/histories_sharing.md %}
>
{: .hands_on}

With this example history we fulfilled Tip 4. We will now fulfill Tip 1 (*"Register the workflow."*) and Tip 3 (*"Make source code available in a public code repository."*) at the same time

# Register the workflow by making the workflow available in the public IWC GitHub repository

To make your workflow FAIR, it's essential to start by making it findable. Let's now register the workflow in a public registry that enables systematic scientific annotations and supports multiple workflow languages. We recommend using specialized workflow registries such as [WorkflowHub](https://workflowhub.eu/) ({% cite gustafsson2024workflowhub %}) or [Dockstore](https://dockstore.org/) ({% cite yuen2021dockstore %}), which cater to workflows written in different languages and provide unique features like digital object identifiers (DOIs) for easy citation and version tracking.

A Galaxy workflow can be registered by anyone on the WorkflowHub and Dockstore by following their documentation. In this section, we will see how to register a workflow on both WorkflowHub and Dockstore (Tip 1 - *"Register the workflow."*) by making it available in a public GitHub repository (Tip 3 - *"Make source code available in a public code repository."*) maintained by the Intergalactic Workflow Commission (IWC), a Galaxy community effort.

The IWC maintains high-quality Galaxy Workflows via a [Galaxy Workflows Library](https://iwc.galaxyproject.org/). All workflows are reviewed and tested before publication and with every new Galaxy release. Deposited workflows follow best practices and are versioned using GitHub releases. Workflows also contain important metadata. Additionally the IWC collects further best practices, tips and tricks, FAQs and assists the community in designing high-quality Galaxy workflows.

IWC offer [guidelines for adding workflow](https://github.com/galaxyproject/iwc/blob/main/workflows/README.md#adding-workflows) that will go through:

1. Check for workflow eligibility
2. Ensure workflows follow best-practices (already done)
3. Generate tests

## Check for workflow eligibility

IWC collects production workflows targeted at users that want to analyze their own data. As such, the workflow should be sufficiently generic that users can provide their own data.

IWC encourage, but do not require, links to related Galaxy Training Network Tutorials. Importantly, each workflow should be described in a way that a user can run the workflow on their own data without modifying the workflow. If we want to deposit a workflow that accompanies a tutorial we have to make sure that the workflow does not refer to datasets that only make sense in the context of the tutorial.

By fulfilling the first Tips as we did before, we guarantee for workflow eligibility.

## Generate tests

This is usually the most difficult part and we encourage all new contributors to IWC to propose their workflows even if they did not managed to generate tests. However, the publication of these workflow will be speed up if tests are already present. 

### Find input datasets

To test the workflow, we need input datasets. By fulfilling Tip 4 (*"Provide example input data and results along with the workflow."*), we generated a toy dataset. We now need to publish it to [**Zenodo**](https://zenodo.org/) to have a permanent URL, also allowing others to **easily retrieve and reuse the data** when running or validating the workflow.

> <hands-on-title>Upload test data to Zenodo</hands-on-title>  
>
> 1. **Create an account** on [Zenodo](https://zenodo.org/) or **log in** using your **GitHub** account.  
> 2. Click on the **"+"** button at the top of the page and select **"New upload"**.  
> 3. **Drag and drop** the test dataset files into the upload area.  
> 4. **Fill in the metadata**:  
>    - **Resource type**: `Dataset`.  
>    - **Title**: `Dataset for "..." workflow ` 
>    - **Creator**: Add the relevant author(s).  
>    - **Description**: `This dataset is associated with the Galaxy workflow "..."`  
>    - **License**: Choose `GNU General Public License v3.0 or later`
> 5. Click **"Publish"** to make the dataset publicly available.  
>
{: .hands_on}

Uploading test data to Zenodo ensures that it has a **permanent DOI**, making it easy to reference in workflow documentation, publications, and testing pipelines.  

### Generate test from a workflow invocation

To generate tests, we can either write test cases by hand, or use a workflow invocation to generate a test case. We will do the second case. Let's first prepare the folder to store the tests in the IWC git repository.

> <hands-on-title>Prepare the folder for the tests</hands-on-title>  
>
> 1. **Fork** the [IWC GitHub repository](https://github.com/galaxyproject/iwc)to your GitHub account.  
>
> 2. **Clone your fork locally**:  
>
>    ```bash
>    git clone https://github.com/yourusername/iwc.git
>    cd iwc
>    ```
>
> 3. **Create a new branch** for adding the workflow:  
>
>    ```bash
>    git checkout -b add-new-workflow
>    ```
>
> 4. **Create a new directory** under one of the directories that represent categories
>
>    > <comment-title></comment-title>
>    > If no category is suitable, we can create a new category directory. We should name the directory that contains our workflow(s) appropriately, as it will become the name of the repository deployed to [iwc-workflows GitHub organization](https://github.com/iwc-workflows) and only use lower-case and `-` in names of categories and repositories.
>    {: .comment}
>
> 5. Move the newly created directory
>
{: .hands-on}

We have earlier executed the workflow on our example dataset. We can use this **workflow invocation** to generate the tests using Planemo, a software development kit for tools and workflows ({% cite bray2023planemo %})

> <hands-on-title>Extract the tests from the example history workflow invocation</hands-on-title>  
>
> 1. **Install Planemo** (if not already installed) by following the [Planemo installation guide](https://planemo.readthedocs.io/en/stable/installation.html).  
>
> 3. Get your **Galaxy API key** from the Galaxy server.
>
>    {% snippet faqs/galaxy/preferences_admin_api_key.md %} 
>
> 4. Get the workflow invocation.
>
>    {% snippet faqs/galaxy/workflows_get_invocation_id.md %} 
>
> 5. Extract the workflow with test data using:  
>
>    ```bash
>    planemo workflow_test_init \
>       --from_invocation <workflow_invocation_id> \
>       --galaxy_url <galaxy_server_url> \
>       --galaxy_user_key <your_api_key>
>    ```
>
{: .hands-on}

This will place in the current working directory:
- a `<workflow_name>.ga` with the workflow
- a `<workflow_name-test>.yml` to describe the tests
- a `test-data` folder with input file and selected outputs

> <question-title></question-title>
>
> In the `<workflow_name-test>.yml`,
> 1. How is the input file provided?
> 2. How are the workflow outputs checked for validity?
> 3. How big are the files in the `test-data` file?
>
> > <solution-title></solution-title>
> >
> > 1. The input file is given by:
> >
> >    ```
> >    job:
> >      FastQ:
> >        class: File
> >        path: test-data/FastQ.fastqsanger.gz
> >        filetype: fastqsanger.gz
> >    ```
> >
> > 2. The workflow outputs are checked for validity by comparing the generated outputs to the files stored in the `test-data` folder.
> > 
> >    ```
> >    outputs:
> >      multiqc_html:
> >        path: test-data/multiqc_html.html
> >      falco_html:
> >        path: test-data/falco_html.html
> >    ```
> >
> > 3. The files are quite large:
> >    - `FastQ.fastqsanger.gz`: 132 KB
> >    - `falco_html.html`: 169 KB
> >    - `multiqc_html.html`: 5 MB
> {: .solution}
{: .question}

The HTML files in the `test-data` folder are quite large, above the 1 MB limit of IWC. To limit the size in the IWC GitHub repository, we will remove test files in `test-data` folder. To do that, we will:
1. use the file stored on Zenodo for the input.
2. edit the test comparisons to use assertions testing the output content, rather than comparing the entire output file with test data.

   > <details-title>Assertion documentation</details-title>
   > The description of assertion can be find in the [Galaxy XML documentation](https://docs.galaxyproject.org/en/latest/dev/schema.html#tool-tests-test-output-assert-contents).
   >
   > Do not hesitate to look at different test files in the [IWC GitHub](https://github.com/galaxyproject/iwc/tree/main/workflows) for examples.
   {: .details}
   
> <hands-on-title>Edit the test description</hands-on-title>
>
> 1. Remove the `test-data` folder
> 2. Open the `<workflow_name-test>.yml` file
> 
> 3. Replace the `path` of input by the **Zenodo URL**:
>
>      ```yaml
>      https://zenodo.org/record/3977236/files/female_oral2.fastq-4143.gz
>      ```
>
> 4. Ensure all test **output names** are included.
>
> 5. For Falco step 
>    
>    1. Open the Falco HTML output on Galaxy
>    2. Search for a text specific to this data (e.g. `female_oral2_fastq-4143_gz.gz` here)
>    3. Replace in the `<workflow_name-test>.yml` file the `path` line of `falco_html` output by:
> 
>       ```
>       asserts:
>          has_text:
>            text: "female_oral2_fastq-4143_gz.gz"
>       ```
>
> 6. Do the same for MultiQC
> 7. Save the file
>
{: .hands-on}

> <comment-title></comment-title>
> If the workflow is using build-in indexes, we should use the available indexes on [CernVM-FS (CVMFS)](https://datacache.galaxyproject.org/), a distributed filesystem perfectly designed for sharing readonly data across the globe. IWC continuous integration uses it for the tests.
{: .comment}

### Lint the workflow

To make sure the workflow and its test are syntactically correct, we can now run `planemo workflow_lint` :

> <hands-on-title>Lint the workflow</hands-on-title>  
> 1. **Run the tests** on the extracted workflow using
>
>     ```bash
>     planemo workflow_lint <workflow_file.ga>
>     ```
{: .hands-on}

### Test the workflow against an instance which have all tools installed.

Before submitting the workflow to IWC, it might be interesting to run the tests against a Galaxy instance using Planemo so we can easily see what is failing and what are the differences between our expectations and the output we get:

> <hands-on-title>Run the tests</hands-on-title>  
> 1. **Run the tests** on the extracted workflow using
>
>     ```bash
>     planemo test \
>        --galaxy_url <galaxy_server_url> \
>        --galaxy_user_key <your_api_key> \
>        <workflow_file.ga>
>     ```
{: .hands-on}

> <details-title>Using Planemo to check that the test is valid against the same workflow invocation</details-title>
>
> If the tests are not passing because of an error into the test file, we can modify the test file and use Planemo to check that the test is valid against the same invocation.
>
> ```bash
> planemo workflow_test_on_invocation \
>    --galaxy_url <galaxy_server_url> \
>    --galaxy_user_key <your_api_key> \
>    <workflow-tests.yml> \
>    <workflow_invocation_id>
> ```
{: .details}


### Add required metadata

Once the workflow tests has been created and tested, we are almost ready to submit it to IWC. Few last steps are missing to add required metadata


The first step is to generate a `.dockstore.yml` file that contains metadata needed for Dockstore.

> <hands-on-title>Generate a `.dockstore.yml` file</hands-on-title>  
>
> 1. Run:  
>    
>    ```bash
>    planemo dockstore_init
>    ```
> 
> 2. Open the generated `docker.yml` and verify that author names and details are correct.  
> 
{: .hands-on}

We now need 
1. a `README.md` file that briefly describes the workflow
2. a `CHANGELOG.md` file that lists changes, additions and fixed. 

   For that, we need to follow the formatting and principles proposed on [keepachangelog.com](https://keepachangelog.com/en/1.0.0/).

> <hands-on-title>Create README and CHANGELOG files</hands-on-title>  
>
> 1. Create a `README.md` file inside the workflow folder ([example README](https://github.com/galaxyproject/iwc/blob/main/workflows/microbiome/pathogen-identification/nanopore-pre-processing/README.md)).  
>
> 2. Create a `CHANGELOG.md` file inside the workflow folder given the following template:
>
>    ```md
>    # Changelog
>    
>    ## [0.1] yyyy-mm-dd
>    
>    First release.
>    ```
> 
{: .hands-on}

Finally, there is currently no user interface within Galaxy to define release versions, so we have to manually set a `release: "0.1"` key value pair in the `.ga` file. 

> <hands-on-title>Edit the release version in the workflow</hands-on-title>  
>
> 1. Edit the `workflow.ga` file to include the **release number**
>
>    ```bash
>    ],
>    "format-version": "0.1",
>    "license": "GPL-3.0-or-later",
>    "release": "0.1",
>    "name": "NameOfTheWorkflow",
>    "steps": {
>    "0": {
>    ```
>
{: .hands_on} 

## Submit the workflow to IWC

> <hands-on-title>Submit the workflow to IWC</hands-on-title>  
>
> 1. Commit the changes to the createw branch
> 2. Push the branch to your fork
> 3. **Open a Pull Request (PR)** on the IWC GitHub repository to submit your workflow.  
>
{: .hands_on}

Once the PR is reviewed and merged, the workflow will be available in **Galaxy's IWC GitHub repository** (Tip 3 - *"Make source code available in a public code repository."*) and indexed in both [**WorkflowHub**](https://workflowhub.eu/workflows/) and [**Dockstore**](https://dockstore.org/workflows/github.com/iwc-workflows/) registries (Tip 1 - *"Register the workflow."*), making it **discoverable and reusable** by the community.

> <comment-title></comment-title>
> The [**Best practices for workflows in GitHub repositories**]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}) GTN provides additional recommendations for structuring and maintaining workflows in Github.
{: .comment}

# Conclusion

By following this tutorial, you have ensured that your Galaxy workflow is following the **"Ten quick tips for building FAIR workflows"** ({% cite de2023ten %})  and adheres to **best practices**. From creating and refining the workflow to creating test data and submitting it to the **IWC**, each step contributes to making workflows more **reliable, shareable, and reusable**.  

For further workflow optimization and maintenance, the **Galaxy community** provides a [comprehensive guide on best practices](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html). This guide extends beyond the **best practices panel in Galaxy** and includes additional recommendations, such as:  

- **Adding automated tests** to validate workflow functionality.  
- **Publishing workflows** on platforms like GitHub, GitLab, or other public version-controlled repositories.  
- **Registering workflows** in well-known registries like [WorkflowHub](https://workflowhub.eu/workflows/) and [Dockstore](https://dockstore.org/workflows/github.com/iwc-workflows/), ensuring wider accessibility and discoverability.  

By continuously following these practices, you contribute to a **stronger, more open, and collaborative** bioinformatics community, where workflows are easily shared, improved, and adapted to new challenges.

tools_json:
["Falco", "toolshed.g2.bx.psu.edu/repos/iuc/falco/falco/1.2.4+galaxy0", "MultiQC", "toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.27+galaxy3"]


topic: galaxy-interface
tutorial: group-tags
title: Group tags for complex experimental designs
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 2425262487772b48ec10c72b455b46fd
last_modified: 2026-05-15T22:28:13.334880
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/group-tags/tutorial.html

content:
Advanced uses of Galaxy often require the use of dataset collections,
which can contain between one and tens of thousands of datasets.
Grouping datasets in this way has numerous advantages:
  - It is easy to represent a single collection in the History
  - Dataset names ("Element Identifiers") are immutable and preserved
  - Collections can be split and nested in arbitrary ways

While collections can be split in any way, doing so for multi-factor analysis
quickly becomes cumbersome and messy. An alternative is to label collection
elements with special group tags, i.e. tags prefixed by the string `group:`.
Note that group tags currently do not propagate, i.e. they are not inherited
to datasets resulting from analyses. These tags can be displayed in the Tool
form, allowing users to select subsets
of collections.

This tutorial outlines how to set and use group tags with the DESeq2 tool.
For a more detailed description and background for differential expression
testing see the [Reference-based RNA-Seq data analysis]({{ site.baseurl }}/topics/transcriptomics/tutorials/ref-based/tutorial.html).


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Setting group tags using the apply rules tool

There are several ways to set group tags:
  - Using the Rule Based Uploader
  - Using the "Tag elements from file" tool
  - Using the "Apply Rules" tool
  - Manually adding dataset tags with the prefix `group:`

We will use the first two methods in this tutorial.
The second and third methods work at any step during the analysis.
Note that the function of the "Apply Rules" tool is (nearly) identical
to the Rule Based Uploader. 

## Set group tags during upload

> <hands-on-title>Set group tags during upload</hands-on-title>
>
> 1. Create a new history for this tutorial
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Open the Galaxy Upload Manager ({% icon galaxy-upload %} on the top-right of the tool panel)
> 3. Click on **Rule-based** on the top
>
>    ![Rule-based upload](../../images/group-tags/rule_upload.png)
>
>    As you can see in this dialog, data can be selected from a history dataset or pasted in directly
>
> 4. Set **Upload data as:** to `Collection(s)`
> 5. Paste the following links into the text box
>
>    ```
>    https://zenodo.org/record/1185122/files/GSM461176_untreat_single.counts
>    https://zenodo.org/record/1185122/files/GSM461177_untreat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461178_untreat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461179_treat_single.counts
>    https://zenodo.org/record/1185122/files/GSM461180_treat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461181_treat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461182_untreat_single.counts
>    ```
>
> 6. Click **Build**
> 7. We will add a regex that creates 3 new columns with accession, treatment and library type:
>    - Click on the **Column** button and then **Using a Regular Expression**
>    - Select **Create columns matching expression groups**
>    - Paste `.*(GSM.*)_(.*)_(.*).counts` in *"Regular Expression"*
>    - Set *"Number of Groups"* to 3
>    - Click on **Apply**
>
>       We should have now a table with 4 columns: link, sample name, treatment, sequencing type
>
>    - Click on **Rules** and then **Add / Modify Column Definitions**
>    - Click on **Add Definition** and select:
>      - *"URL"*: Column A (Note that this option is absent when using the "Apply rules tool")
>      - *"List Identifiers"*: Column B
>      - *"Group Tags"*: Columns C and D (Select Column C first and then add D by clicking on *"... Add another column"*)
>    - Click **Apply**
>    - Enter a name for the new collection
>    - Click **Upload**
>
> 8. Expand the generated collection and the files in it and check their names and tags
>
>    ![Group tags in Galaxy UI are prefixed with group:](../../images/group-tags/group-tags.png)
{: .hands_on}

## Set group tags using the "Tag elements from file" tool

We now want to add group tags using the "Tag elements from file" tool.

> <hands-on-title>Upload and create a collection</hands-on-title>
>
> 1. Create a new history for this tutorial
> 2. Import the following files
>
>    ```
>    https://zenodo.org/record/1185122/files/GSM461176_untreat_single.counts
>    https://zenodo.org/record/1185122/files/GSM461177_untreat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461178_untreat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461179_treat_single.counts
>    https://zenodo.org/record/1185122/files/GSM461180_treat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461181_treat_paired.counts
>    https://zenodo.org/record/1185122/files/GSM461182_untreat_single.counts
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 3. Create a **Dataset List (Collection)** with these 7 files
>
>    {% snippet faqs/galaxy/collections_build_list.md datasets_description="The 7 datasets you've just imported" n="7" %}
>
{: .hands_on}

We have now a collection with our files. We can now either upload a tabular file containing the element identifiers
and the tags we want to apply, or we can extract the element identifiers and extract the tags using a Regular Expression.
We will do the latter.

> <hands-on-title>Set group tags using the "Tag elements from file" tool</hands-on-title>
> 1. **Extract element identifiers** {% icon tool %}
>      - {% icon param-collection %} *"Dataset collection"*: created collection
> 2. **Replace Text in entire line** {% icon tool %}
>      - {% icon param-file %} *"File to process"*: output of **Extract element identifiers** {% icon tool %}
>      - In *"Replacement"*:
>         - In *"1: Replacement"*
>            - *"Find pattern"*: `(.*)_(.*)_(.*).counts`
>            - *"Replace with"*: `\1_\2_\3.counts\tgroup:\2\tgroup:\3`
>
>     This step adds an additional columns that can be used with the ``Tag elements from file`` tool
>
> 3. Change the datatype to `tabular`
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
> 4. **Tag elements from file** {% icon tool %}
>      - {% icon param-collection %} *"Input Collection"*: created collection
>      - {% icon param-collection %} *"Tag collection elements according to this file"*: output of **Replace Text** {% icon tool %}
{: .hands_on}

You should now have a properly tagged collection of tabular files that can be used in DESeq2.

# Using group tags in tool, e.g. DESeq2

DESeq2 has two modes for specifying factors. One can either
select datasets corresponding to factors, or use group tags
to specify factors. We will use the group tags present in
our collection to specify factors.

The tool interface will prompt you with the group tags that are available for your inputs:

![Group tags in the tool UI](../../images/group-tags/tool-ui.png)

> <hands-on-title>Running <b>DESeq2</b> with group tags</hands-on-title>
>
> 1. **DESeq2** {% icon tool %} with the following parameters:
>    - *"how"*: `Select group tags corresponding to levels`
>       - {% icon param-collection %} *"Count file(s) collection"*: Generated collection
>       - In *"Factor"*:
>          - In "1: Factor"
>              - *"Specify a factor name"*: `Treatment`
>              - In *"Factor level"*:
>                  - In *"1: Factor level"*:
>                      - *"Specify a factor level"*: `treat`
>                      - *"Select groups that correspond to this factor level"*: `Tags: treat`
>                  - In *"2: Factor level"*:
>                      - *"Specify a factor level"*: `untreat`
>                      - *"Select groups that correspond to this factor level"*: `Tags: untreat`
>          - Click on {% icon param-repeat %} *"Insert Factor"* (not on "Insert Factor level")
>          - In "2: Factor"
>              - "Specify a factor name" to `Sequencing`
>              - In *"Factor level"*:
>                  - In *"1: Factor level"*:
>                      - *"Specify a factor level"*: `paired`
>                      - *"Select groups that correspond to this factor level"*: `Tags: paired`
>                  - In *"2: Factor level"*:
>                      - *"Specify a factor level"*: `single`
>                      - *"Select groups that correspond to this factor level"*: `Tags: single`
>    - *"Files have header?"*: `No`
>    - *"Output normalized counts table"*: `Yes`
{: .hands_on}

# Conclusion


We can select a subset of Collections using the special group tag.


topic: galaxy-interface
tutorial: history-to-workflow
title: Extracting Workflows from Histories
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["workflows"]
content_hash: bf0f2864e412f1f2893cb78db4f97bea
last_modified: 2026-05-15T22:28:13.335784
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/history-to-workflow/tutorial.html

content:
# Create a reusable workflow from a history


This practical shows how to create a reusable analysis pipeline, called a *workflow* in Galaxy, from an analysis that you have already run in Galaxy, called a *history*.


> ###  {% icon comment %} Audience
> This tutorial is intended for those who are new to Galaxy.
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Pretreatments

> <comment-title>Requirements</comment-title>
>
> To run this practical you will need
>
> 1. Previous experience running at least one analysis in Galaxy.
> 1. An internet-connected computer.  Galaxy can run on your laptop without an internet connection, but this practical requires access to resources on the web.
> 1. A web browser. [Firefox](https://www.mozilla.org/firefox) and [Google Chrome](https://www.google.com/chrome/) work well, as does [Safari](https://www.apple.com/safari/).  Internet Explorer is known to have issues with Galaxy so avoid using that.
> 1. Access to a Galaxy instance, and an account on that instance. Galaxy is available in many ways. If you are doing this practical as part of a workshop, the instructor will tell you which instance to use. If you are doing this on your own, you can use [usegalaxy.org](https://usegalaxy.org).
> 1. A Galaxy *history* from an analysis that you have already run.  If you don't have one handy, you can use [this history](https://usegalaxy.org/u/tnabtaf/h/overlapping-genes-on-opposite-strands-1). on usegalaxy.org.  Click **Import History** (top right) to bring that history into your workspace on usegalaxy.org. Any history will work with this tutorial, but that's the one used in the examples.
{: .comment}

This tutorial is a good second step after running your first analysis on Galaxy.

## Motivation

Galaxy excels at helping researchers figure out how to do analyses.  It's easy to try different tools and settings, and if your first result isn't quite what you want, you can just keep trying until you get what you want.  What you may end up with is a slightly (or very) messy analysis *history* under your account.  If this is the only time you'll run that particular analysis then it's fine to just hang on to that history.   It might be handy to look at the next time you want to do something in Galaxy.

But what if you want to run that analysis again, maybe on updated datasets, or maybe even on entirely different datasets?  You could manually rerun all the steps in your first analysis, using the new datasets instead of the old.  But that is tedious, and error-prone. What you want is the ability to easily run exactly the same tools, with the same settings, in the same order as the first time you ran the analysis.

Galaxy *workflows* enable this, and this tutorial shows how you can create one from the analysis you have already done, and then run the analysis exactly as you did before, but on the new datasets.

> <tip-title>Confused about <i>Histories</i> and <i>Workflows?</i></tip-title>
>
> Well, you are not alone.  This analogy may help.
>
> * A *history* is a record of analysis that you have done, or of the analysis you are currently working on.  **Think of a history as a baked cake.**  It's something that been done.  It has input datasets, possibly intermediate datasets, and output datasets.  Think of the input datasets as the basic ingredients used to make the cake, like eggs and flour, and the output datasets are the cake.
>
> * A *workflow* is a **recipe** for running an analysis.  It doesn't contain any of the ingredients or the cake.  It does specify what you'll need to run the recipe: input datasets (the eggs and flour), and all of the steps you'll need to do to create the cake (beat the eggs, mix the flour, sugar and baking powder, ...).
>
> In this tutorial we will take a fully baked cake (a history), *ask it to tell us the recipe that was used to create it,* and then create a recipe (a workflow) based on what the cake tells us.
>
> Happy baking.
{: .tip}

## Set your current history

By this time, you may have multiple histories under your Galaxy account.  You'll need to make the history that you want create a recipe/workflow for be your *current* history.  Your current history is the one shown in the History panel on the right.

> <hands-on-title>Managing your histories</hands-on-title>
>
>  1. Make sure you are *logged in*.
>    - If you haven't yet created an account, now is an ideal time to that.  Your current history will be saved as your first history under your new account.
> 2. *Click* on the **History options** at the top-right of the history panel and select **Show Histories Side-by-Side** to switch to *multiple history view*.
>    - This lists your histories, from left to right in reverse chronological order, based on the last time each history was your current history.
> 3. *Switch to* the history you want to extract a workflow from.
>    - If the history that you want to create a repeatable workflow for is *not* your *current* history, then find the history you want and then *click* the **Switch to** button at the bottom of the history.
>    - This makes that history the current history.
{: .hands_on}

> <tip-title>Always name your histories</tip-title>
>
>If you just saw multiple `Unnamed history` entries in your histories view, then now you know why.
>
> **Bonus tip:** You can name your histories in your histories view.
{: .tip}

## Extract the recipe from your history

Now that we have the history we want, let's use Galaxy to create a reusable workflow from it.  To do this we'll use the history's **gear / history options** pull-down list.


> <hands-on-title>Extract workflow</hands-on-title>
>
> 1. *Click* on the {% icon galaxy-history-options %} **History options** at the top-right of the history panel.
>    - This opens a pull-down menu showing lots of actions that you can perform on the current history.  Right now, we are interested in only one of them:
> 2. *Click* on **Extract Workflow**.  It's about half-way down the menu.
>    - This launches a form to create a workflow.
> 3. *Give* your workflow a meaningful name.
>     - Think about giving it a more general name than what your history has. The example history is about looking for overlapping genes on opposite strands in human on chromosome 22.  That's pretty specific.  We want to create a workflow that can be used to detect any type of overlapping features, not just genes, and in any species and on any chromosome (or on all chromosomes).  The name should reflect this goal.
>     - For this example we'll name it `Overlapping features on opposite strands`
> 4. *Remove* unwanted steps
>     - If there were any missteps or dead-ends in your history, uncheck those steps here.
>     - Don't worry if you miss one - we can drop it in a subsequent step.
> 5. Once you have named your workflow, click the **Create Workflow** button.
>
>    ![Create workflow form](../../images/workflow_strand_create.png)
>
>    - This replaces the workflow creation form with a box that says:
>       - Workflow *`whatever name you entered`* created from current history. You can edit or run the workflow.
> 6. *Click* on the **edit** link in the box.
>
{: .hands_on}

This launches the workflow editor and shows a graphical representation of the inputs and steps taken during the analysis.  Each box is either an input dataset, or a tool that was run.  Boxes are connected by lines that show the flow of data from input datasets, through intermediate tools, to the tools that produce the final output(s).  Each tool box show a list of inputs (at the top) and a list of outputs (at the bottom).  The flow of data is generally from left to right.

![Workflow Editor](../../images/workflow_editor_landing_strand.png)

While we could run this workflow right now, here are a few cleanup steps we should do first.

### Rename input datasets

The *extract workflow* step assigns the name of input datasets in your history to the corresponding inputs in your workflow.  This is very helpful in the short term because it makes it clear which of the boxes are which datasets.  But, for a workflow, we want a more general (but still helpful) name for input datasets.

> <hands-on-title>Rename inputs</hands-on-title>
>
> 1. *Click* on the first input dataset.
>    - This changes the right panel to show information about the selected dataset.  In this example, this dataset is a set of genomic features (like genes or exons or repeats) that exist in an organism.
> 2. *Set* the **Label** field to something more general, yet still informative.
>    - In the example workflow, we'll name this `Genomic features` about as general as you can get.
> 3. Add a *description* for your input datasets
>     - This helps make your general input dataset name more informative.
>     - *add* a description of the dataset in the **Annotation** box.
>     - For the example, we'll use:
>        - `Workflow checks this set of genomic features to find any that overlap with each other on opposite strands.  **It must be at least a 6 column BED file,** as the workflow makes decisions based on column 6 (the strand).`
>      ![Renamed and annotated input dataset](../../images/workflow_strand_renamed_plus_annot.png)
{: .hands_on}

There are several other things we could do in the workflow editor, but let's focus on just two of them.

First, your history might contain several false starts or tool runs where the parameters weren't quite right. Lets get rid of those now.

> <hands-on-title>Drop unused input datasets and tool boxes</hands-on-title>
>
> 1. *Delete* unwanted steps
>    - If your history contained any false starts or tool runs that didn't contribute to your final result, then *delete* them from the workflow by clicking the black **x** in the corner of those datasets/tools' boxes.
>    - This will remove them from the workflow.
{: .hands_on}

> <tip-title>Removing unnecessary steps before creating workflow</tip-title>
> You can also delete these steps when you are first extracting the workflow from your history, by unchecking them.  However, it is often easier to identify the steps to drop in the workflow view, where connections and dead-ends are more obvious.
{: .tip}

Second, you might want to give your output datasets meaningful names too.

> <hands-on-title>Name output datasets</hands-on-title>
>
> 1. *Click* on a step that produces an output file.
>    - This brings up information about that step in the right panel.
> 2. *Scroll* down in the right panel until you see a **Configure Output** link.  *Click* on it.
> 3. *Enter* a meaningful name in the **Rename dataset** field.
>    - In the example, we'll enter `Features that overlap another feature on opposite strand.`
> 4. Repeat this for all your output datasets.
{: .hands_on}

We could add annotation to each step in the process as well, and if this workflow is going to be published in a paper, or shared with others, then that is definitely worth doing.

Now that our edits are done, let's save our work and run the workflow.

> <hands-on-title>Save the workflow edits</hands-on-title>
>
>  1. *Click* on the {% icon galaxy-save %} at the top right to save workflow.
{: .hands_on}


### Test the workflow

Now that we have finished creating our workflow, it is time to test it.

> <hands-on-title>Run the workflow</hands-on-title>
>
> 1. *Click* on the {% icon workflow-run %} to run workflow.
> 2. *Examine* the workflow run form.
>
>
>    This form lists all the inputs and steps in the workflow also asks if you want to run this workflow in a new history, of just add it to your existing history.
>    ![Run workflow form](../../images/workflow_run_strand_landing.png)
>
>    In this form you can also change the run-time parameters of any of the steps.  We aren't going to do that, but it can be useful when are experimenting with different parameters in different steps.
>    Let's test the workflow by running it on the same input datasets we used in the current history.  If we created the workflow correctly, we should get the same results as in the history.
>
> 3. *Select* **Yes** under **Send results to a new history**.
>    - Sometimes you'll be building analyses out of component workflows and you will want to run them all in the same history.  Here we want a new history because the results will be cleaner and easier to understand.
> 4. Give the new history a meaningful name.
> 5. *Set* your first input dataset using the pull-down menu.
>   - *Repeat* until all input datasets are set.
> 7. *Click* the **Run workflow** button.
{: .hands_on}

And Galaxy launches the workflow, running in a new history.  It says (in a nice big green box) something like:

You can check the status of queued jobs and view the resulting data *by refreshing the History pane.* When the job has been run the status will change from 'running' to 'finished' if completed successfully or 'error' if problems were encountered.

But, *that's a lie*.  In our case we sent the results to a new history, and the history panel is still showing us the old history.  To get to the new history, open the multiple history view (*click* the {% icon galaxy-history-options %} **History Options** at top of the history panel) and then make the newly created history be the current history.

Depending on your workflow and datasets, your analysis may already be done by the time you make it the current history and go back to the standard view.

At this point it's worth doing a little cleanup.  In particular think about updating the output dataset names to be more specific to the data we actually used.


## Final thoughts

* Workflows can also be created from scratch
* If you ever work on large studies or even if you have just a few samples, workflows are critical to
  1. Running the same analysis on each dataset.
  1. Not going insane.


topic: galaxy-interface
tutorial: jupyterlab
title: JupyterLab in Galaxy
difficulty: intermediate
time_estimation: 1H
tools_json: ["JupyterLab", "interactive_tool_jupyter_notebook"]
requirements_json: []
tags_json: ["interactive-tools"]
content_hash: 7f00937b069601c89b451d7db26fc4a0
last_modified: 2026-05-27T09:01:48.661886
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/jupyterlab/tutorial.html

content:
[JupyterLab](https://jupyterlab.readthedocs.io/en/stable) is an [Integrated Development Environment (IDE)](https://en.wikipedia.org/wiki/Integrated_development_environment).
Like most IDEs, it provides a graphical interface for R/Python, making it more user-friendly, and providing dozens of useful features.
We will introduce additional benefits of using JupyterLab as you cover the lessons.

{% include topics/galaxy-interface/tutorials/jupyterlab/tutorial_origin.md %}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# JupyterLab

Opening up your {% tool [JupyterLab](interactive_tool_jupyter_notebook) %}:

{% snippet faqs/galaxy/interactive_tools_jupyter_launch.md %}

You should now be looking at a page with the JupyterLab interface:

![Jupyterlab default session](../../images/jupyterlab_default_session.png)

As shown on the figure above, JupyterLab interface is made of 3 main areas:
- The menu bar at the top
- The left side bar with in particular the **File Browser**
- The main work area in the central panel

## Start your first notebook

Now that we are ready to start exploring JupyterLab, we will want to keep a record of the commands we are using. To do this we can start a Notebook.

> <hands-on-title>Start a notebook</hands-on-title>
>
> 1. On the left side bar, in the **File Browser**: 
>    - Double-click on the file called **ipython_galaxy_notebook.ipynb**. 
>    - Then select the default **This will open the **Python [conda env:base]** Kernel
>    - This will open a default notebook in the main work area.
> 2. If **ipython_galaxy_notebook.ipynb** does not exist (for instance on *Try JupyterLab*), click on **+** (top left) to start **The launcher**. Then click on the **Python** icon in the **Notebook** section to create a new blank notebook.
{: .hands_on}

A new notebook appears in the centre panel. Before we go any further, you should learn to save your script.

> <hands-on-title>Save a Python notebook</hands-on-title>
>
> 1. There are several options to save your file:
>    - Click the **File** menu and select **Save Notebook As...**.
>    - Click the {% icon galaxy-save %} icon (**Save the notebook contents and create checkpoint**) in the bar above the file tab line in the script editor.
>    - Click the **File** menu and select **Save Notebook**.
>    - Type <kbd>CTRL</kbd>+<kbd>S</kbd> (<kbd>CMD</kbd>+<kbd>S</kbd> on OSX).
>
> 2. There are several options on how to (re-)name your file:
>    - After clicking on the option **Save Notebook As**, a window opens, where the file can be (re-)named.
>    - Right click on the current file name (`ipython_galaxy_notebook.ipynb` or `Untitled.ipynb`) in the bar above the first line in the script editor and select **Rename Notebook** in the opened selection window.
>    - Alternatively, right click on the file name you would like to change on the left side in the folder menu. A window will open where you can select **Rename**.
{: .hands_on}

The new script `ipython_galaxy_notebook.ipynb` should appear in the **File Browser*** in the left panel. By convention, Jupyter notebooks end with the file extension `.ipynb` independently of the programming language (R, Python, Octave, Julia, etc.).

![jupyterlab default notebook](../../images/jupyterlab_session_default_notebook.png)

> <comment-title>Note: supported programming languages</comment-title>
>
> Depending on your JupyterLab instance, the list of supported programming languages may vary.
> On [Live.useGalaxy.eu](https://live.usegalaxy.eu), the following programming languages are currently supported:
> - Python 3
> - Julia
> - R
> - Octave
> - Ansible
> - Bash
> - SciJava
{: .comment}

By default, a **Python notebook** is started. Don't worry if you are not familiar with *Python* programming language, it is not necessary for this tutorial. The same functionalities applies for any available programming languages.

> <comment-title>Note: switching to another programming language</comment-title>
>
> Once you have created a Notebook, you can switch to another available programming language (**Switch Kernel**).
> 1. On the top right of your running Notebook, click on **Python 3**. A new window pops up:
> ![jupyterlab switch kernel](../../images/jupyterlab_switch_kernel.png)
> 2. In this new window, click on **Python 3** to select an available programming language (R, octave, Julia, etc.).
> 3. Click on **Select** to activate your selection. The pop-up window closes and you are ready to use your notebook with the selected programming language.
>    Alternatively, you can also:
> 1. Click on **+** (top left menu) to start the **Launcher**. The list of available programming language is given in the **Notebook** section.
> 2. Click on icon of your choice in the **Notebook** section.
> 3. A new notebook is created with the programming language of your choice.
{: .comment}

In the next step, we will create some visualizations. To visualize data, programming languages offer specific libraries. Before we use these libraries, we first need to ensure that these dependencies are installed.

> <hands-on-title>Install dependencies using Conda</hands-on-title>
>
> 1. To open a console to install the dependencies, open the launcher window by clicking on the plus sign in the top-left corner. Within this window, click on `Python [conda env:python-kernel-3.12]` iin the Console section.
> 2. After the console opens, you can enter the following lines into the cell at the bottom:
>    ```python
>    conda install -c conda-forge numpy matplotlib
>    ```
>
> 3. <kbd>Shift+Return</kbd> to run the cell or click on the run cell button.
>
{: .hands_on}

As the dependencies are installed within one Kernel the notebook needs to run on the same Kernel.

> <hands-on-title>Select Dernel</hands-on-title>
>
> 1. First restart the Kernel, by clicking on **Kernel** in the upper menu and select **Restart Kernel...**
> 2. Next switch back to the your ipython notebook (e.g. **ipython_galaxy_notebook.ipynb**). You can find it next to the console tab on top of the right window.
>
> 3. Below the taps you will find **Python env** button (e.g. `Python[conda env:base]`). If you click on this one you can select a Kernel. Please select the Kernel, which was update in the previous step (`Python [conda env:python-kernel-3.12]`).  
>
{: .hands_on}

Now that the dependencies are installed and the Kernel is restarted, we can import the libraries. 

> <hands-on-title>Import Python libraries</hands-on-title>
>
> 1. Click on a cell of your notebook to edit it (verify that it is defined as a "Code" cell)
> 2. Enter the following lines:
>    ```python
>    import numpy as np
>    import matplotlib.pyplot as plt
>    ```
>
> 3. <kbd>Shift+Return</kbd> to run the cell or click on the run cell button.
>
{: .hands_on}

## Graph Display in JupyterLab with Python

In this tutorial we are going to simply plot a simple graph using generated data.

> <hands-on-title>Draw a simple plot</hands-on-title>
>
> 1. Generate a simple dataset
>   
>    ```python
>    x = np.linspace(0, 2, 100)
>    y = x**2
>    ```
>
> 2. Create your figure with the command:
>
>    ```python
>    fig, ax = plt.subplots( nrows=1, ncols=1 ,figsize=(15, 10) )
>    ```
>
>    - `nrows=1, ncols=1` means you will have one plot in your figure (one row and one column)
>    - `figsize` parameter determine the size of the figure
> 3. Draw the plot with the command
>
>    ```python
>    plt.plot(x, y);
>    ```
>
>    ![Simple plot in Jupyter](../../images/jupyterlab_plot.png)
{: .hands_on}

# Interaction between JupyterLab and Galaxy

## Import / export Data

You can import data from Galaxy history using the `get(12)` command, with the number of your dataset in the history (If you are working on a collection, unhide datasets to see their numbers).

If you want to save a file you generated in your notebook, use the `put("file_name")` command. That is what we are going to do with our distribution plot.

> <hands-on-title>Save an Jupyter generated image into a Galaxy History</hands-on-title>
>
> 1. Create an image file with the figure you just draw with the command `fig.savefig('simpleplot.png')`
> 2. Export your image into your history with the command  `put('simpleplot.png')`
>
{: .hands_on}


## Save the Notebook in your history

Once you are done with you analysis or anytime during the editing process, you can save the notebook into your history using the `put("ipython_galaxy_notebook.ipynb")`. If you create additional notebooks with different names, make sure you save them all before you quit JupyterLab.

This will create a new notebook `.pynb` file in your history every time you click on this icon.

> <hands-on-title>Closing JupyterLab</hands-on-title>
>
> 1. In the Galaxy interface click on  **Interactive Tools** button on the left side.
>
> 2. Tick {% icon galaxy-selector %} the box of your Jupyter Interactive Tool, and click **Stop**.
>
{: .hands_on}


topic: galaxy-interface
tutorial: onedata-getting-started
title: Getting started with Onedata distributed storage
difficulty: introductory
time_estimation: 20m
tools_json: []
requirements_json: []
tags_json: ["storage"]
content_hash: 95e9980c79b0df47c6ede61ee7a695dc
last_modified: 2026-05-27T09:01:48.662380
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/onedata-getting-started/tutorial.html

content:
> <agenda-title></agenda-title>
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction to Onedata

Onedata ([https://onedata.org](https://onedata.org)) is a data management platform that
provides easy and unified access to globally distributed storage resources, supporting a
wide range of use cases from personal data management to data-intensive scientific
computations. It is an [open-source project](https://github.com/onedata), started in 2013,
and implemented by the team from the Cyfronet Computing Center in Krakow, Poland. 

Onedata creates a virtual file system layer spanning geographically dispersed computing
centers and data providers that host heterogeneous storage resources. The virtual file
system is POSIX-compatible and based on a classic structure of directories and files. The
virtualized data can be access using multiple interfaces: Web GUI, REST API, CDMI API,
fuse-based POSIX mount, Python libraries, or S3. Regardless of the interface, the user
gets the same, unified view of all his data.

![Onedata virtual FS](../../images/onedata-getting-started/onedata-virtual-fs.png)

Onedata uses the concept of **Spaces** for data organization. A **Space** is a logical
data volume that appears as a monolithic file system from the user’s PoV. Still, it
virtualizes the physical data stored on distributed storage systems of different data
providers. Spaces facilitate collaborative data sharing between users and groups across
organizational domains — using the Onedata interfaces, users can manage and access the
data together in a unified namespace, while it is physically distributed. 

![Onedata Spaces](../../images/onedata-getting-started/onedata-spaces.png)

The [Oneclient](https://onedata.org/#/home/documentation/topic/stable/oneclient) 
software, based on the [fuse](https://github.com/libfuse/libfuse) library, 
allows mounting the Onedata file system locally, either on a personal computer,
or on a worker node in a computing cluster for 
[HPC purposes](https://onedata.org/#/home/documentation/topic/stable/oneclient-direct-io). 

![Oneclient mount](../../images/onedata-getting-started/onedata-oneclient.png)

Similarly to the Galaxy project, The Onedata software can be used to build different
ecosystems. Each Onedata ecosystem constitutes an independent data management platform, 
made up of multiple data centers. One of the flagship examples is 
[EGI DataHub](https://datahub.egi.eu), a Europe-wide ecosystem bringing together 17
data sites (as of 03-2025) and catering for many scientific projects around Europe.

For more information about Onedata, see the 
[Documentation](https://onedata.org/#/home/documentation) and 
[API reference](https://onedata.org/#/home/api).


# Onedata & Galaxy integration

Thanks to the efforts undertaken in the
[EuroScienceGateway](https://galaxyproject.org/projects/esg/) project, Galaxy
now offers integration with Onedata. It can be used as 
a **remote source for data import/export** (a.k.a. Files Source Plugin) and as 
a **storage location for Galaxy datasets** (an Object Store). 
The integration includes **BYOS** (Bring Your Own Storage) and 
**BYOD** (Bring Your Own Data) templates.

Minimal requirements:
- **Galaxy**: version **24.2**
- **Onedata**: version **21.02.5** (but it's recommended to use at least **21.02.8**)

It is possible to connect multiple Onedata accounts in different Onedata
ecosystems to the same Galaxy account.

The below image shows the Onedata file browser UI and the Galaxy **Upload** tool
making it possible to import data stored in Onedata Spaces.
![Onedata & Galaxy interfaces](../../images/onedata-getting-started/galaxy-onedata-side-by-side.png)


## What's the point?

There are many ways you (or your organization) can leverage the Onedata's
distributed data management capabilities in Galaxy:

* Onedata is all about unifying access to geographically distributed datasets.
  It may be your solution to integrate data from multiple data centers and
  bring them to Galaxy for processing.

* One of the crucial features of Onedata is seamless collaboration on data
  across different organizations. It supports groups that bring together users
  of different affiliations to work on datasets that are integrated from pieces
  coming from different data sources.

* Onedata offers powerful data sharing & publishing mechanisms and tools for
  long-term data archiving and replication, which you may find useful to improve
  your Galaxy data management.

* While Galaxy supports a plethora of different data sources and storage
  options, Onedata offers some additional storage drivers, like Ceph or XRootD.
  What can't currently be done directly via Galaxy, can be integrated into
  Galaxy through Onedata.

* If your data management and processing pipelines include other software
  besides Galaxy, the Onedata unified storage layer can help to better organize
  your environment, providing a common storage abstraction for different
  applications and pieces of middleware.

* Galaxy Pulsar and data distribution — the ongoing efforts in Galaxy to build a
  distributed network of Pulsar nodes is aligning well with the Onedata
  distributed storage model. By understanding the locality of the data, Galaxy
  will be able to schedule jobs in a smart way, assigning them to Pulsar nodes
  that are as close to the data as possible. This is where the Onedata Virtual
  Filesystem comes to play, allowing the management of data replication and
  distribution in a geographically aware manner. The synergy of distributed
  compute power and storage resources can greatly boost the scientific process,
  especially in projects that bring together people and datasets from different
  cooperating organizations.


# Accessing Onedata services

This guide will give you a kick-start, regardless if your organization is
already part of a Onedata ecosystem, you have access to Onedata services, or you
are entirely new to this software. 

Follow the Onedata
[user quickstart](https://onedata.org/#/home/documentation/topic/stable/user-quickstart)
tutorial to access the Onedata services. Newcomers can use the sandbox environment
to get familiar with the system.


## Access tokens

All Onedata connectors for Galaxy require an access token suitable for
data access in a Oneprovider service.

Follow the Onedata documentation to acquire an access token:
* GUI — follow the
  [guide](https://onedata.org/#/home/documentation/topic/stable/tokens-gui-guide),
  choosing a **Oneprovider REST/CDMI access** token template.
* REST API — to acquire an access token programatically, you can interact with 
  the [Onedata REST API](https://onedata.org/#/home/documentation/topic/stable/tokens-rest-api).

In most cases, the token **should not** be limited to read-only data access,
unless you wish to create a non-writable or publicly accessible Remote File
Source. In such a case, add a **Read-only** caveat to the token, as shown in the
[detailed guide](https://onedata.org/#/home/documentation/topic/stable/tokens-gui-guide).

Tokens can also limit the accessible spaces and providers by means of other 
[caveats](https://onedata.org/#/home/documentation/topic/stable/tokens-caveats).


## Choosing a Onedata Space

For the purposes of Remote File Sources, you do not have to designate a specific
**Space** — all of them will be available in the data import/export tools
(unless you have provided a restricted [access token](#access-tokens)). If 
you have followed the 
[user quickstart](https://onedata.org/#/home/documentation/topic/stable/user-quickstart) 
tutorial, you will be able to import from and export to the **Sandbox** Space.

In case of Storage Locations, you have to choose a specific Onedata **Space**
where the Galaxy data will be stored:

* A Space provided by your organization — if your institution is a part
  of a Onedata ecosystem, you may be assigned a specific Space for your
  Galaxy data.
* Your private Space — if you have access to a Onedata space for your
  own purposes, you can connect it as a Storage Location to Galaxy.
* The **Sandbox** Space — after following the Onedata
  [user quickstart](https://onedata.org/#/home/documentation/topic/stable/user-quickstart) 
  tutorial, you should have a personal, ACL-secured folder in the **Sandbox** Space.
  You may use it to as your Onedata Storage Location (possibly creating some nested
  directory structure inside, if needed).
* (Developers) a Space in a dev instance — if you are familiar with docker, you
  can 
  [set up a local Onedata deployment]({% link topics/dev/tutorials/onedata-dev-instance/tutorial.html %})
  and use the **demo-space** as a Storage Location. Remember to disable TLS 
  certificate validation in the configs!


# Troubleshooting

In case you are getting errors when interacting with a Onedata 
remote source or storage location, go through the below checklist:

1. Avoid whitespace characters (spaces, tabs) in the Space names and
   file paths, as they are known to cause problems.

2. Make sure the Onezone domain is correct — open it in your Web browser to double-check.

3. Log in to the Onezone service and check if there is any service outage or 
   disrupted data providers — if so, you must wait for these problems to be solved. If not,
   you should assume the problem lies in the configuration of the Galaxy server or your
   user preferences.

4. Make sure the token you provided is suitable for REST API access in a
   Oneprovider service. Some setups require the token to be write-enabled.
   Consult the [access tokens](#access-tokens) section for a guide. 

5. Make sure the token allows access to at least one Onedata Space in at least
   one provider (Remote File Source), or the the Space designated as a Storage
   Location. Caveats (contextual confinements) may be limiting the access token
   scope.

6. Make sure there is at least one data provider backing up (supporting) the
   relevant Space(s).

7. If you are familiar with command-line, you may perform basic diagnostics on the
   token like this:

   ```bash
   curl -d '{"token": "'$TOKEN'"}' -H 'Content-type: application/json' \                                                                              130 ↵
        https://$ONEZONE_DOMAIN/api/v3/onezone/tokens/infer_access_token_scope | jq
   ```

   ```json
   {
     "validUntil":null,
     "dataAccessScope":{
       "spaces":{
         "d67622d2a0999be9b9b1e6e18c14c697chbfa6":{
           "supports":{
             "42a2fb7993b8331aa107ff819101b0f1chb73a":{
               "readonly":false
             }
           },
           "name":"space-alpha"
         },
         "readonly":false,
         "providers":{
           "42a2fb7993b8331aa107ff819101b0f1chb73a":{
             "version":"21.02.8",
             "online":true,
             "name":"My provider",
             "domain":"my-provider.example.com"
           }
         }
       }
     }
   }
   ```
   
   If there is at least one entry both in `spaces` and `providers`, and at least one of
   the providers is online, the token is viable. If not, you should see point 2) or
   consider creating a token with fewer confinements.

8. If you are a Dev/Admin, check the Galaxy logs for hints.


# Related tutorials

Learn more how to make use of the Galaxy & Onedata integration:

| Audience         | Topic                                                                                                                                            |
| ---------------  | -------------------                                                                                                                              |
| User             | [Importing (uploading) data from Onedata]({% link topics/galaxy-interface/tutorials/onedata-remote-import/tutorial.html %})                                  |
| User             | [Exporting to Onedata remote]({% link topics/galaxy-interface/tutorials/onedata-remote-export/tutorial.html %})                                  |
| User             | [Onedata user-owned storage]({% link topics/galaxy-interface/tutorials/onedata-byos/tutorial.html %})                                            |
| Admin            | [Configuring the Onedata integration (remotes, Object Store, BYOS, BYOD)]({% link topics/admin/tutorials/onedata-configuration/tutorial.html %}) |
| Developer        | [Setting up a dev Onedata instance]({% link topics/dev/tutorials/onedata-dev-instance/tutorial.html %})                                          |


topic: galaxy-interface
tutorial: workflow-posters
title: Creating high resolution images of Galaxy Workflows
difficulty: introductory
time_estimation: 30M
requirements_json: ["type: external"]
tags_json: ["workflows"]
content_hash: 3c3a67fb855c7fd66629be0f126c483c
last_modified: 2026-05-27T09:01:48.671046
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/workflow-posters/tutorial.html

content:
Sometimes we want to share our workflows outside of Galaxy, like printing it on a poster.
Making screenshots of Workflows looses a lot of detail, often to an degree where information is lost.

There are third party tools which can help us in making high resolution screenshots that preserve all details
and produce pictures which are suitable for printing and sharing.

{% include _includes/cyoa-choices.html option1="Print" option2="Web" default="Print"
       text="If you want to print your Galaxy Workflow on a poster, choose Print, otherwise Web is the better option." %}

> <agenda-title></agenda-title>
>
> In this tutorial we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Making a high-resolution Screenshot of a Galaxy Workflow

<div class="Print" markdown="1">
In this section we will learn on how to make an image of a workflow which is suitable for printing on a poster.
</div>

<div class="Web" markdown="1">
In this section we will learn on how to make a screenshot of a workflow which preserves as much details as possible.
</div>

We will be using a feature built into the [Firefox](https://www.mozilla.org/firefox/) web browser to make screenshots larger than our screen.
This only works in Firefox, so make sure you have it installed and are using it to follow the rest of this tutorial.

## Hiding the Galaxy UI

<div class="Print" markdown="1">
For our Workflow poster, we will likely not want to print the rest of Galaxys user interface.
</div>

<div class="Web" markdown="1">
For our screenshot, we may want to hide the rest of Galaxys user interface, so the focus is on the Workflow itself.
</div>

We could cut it out of the final image using image-manipulation tools, but this is slow and cumbersome, especially for high-res images.

From Galaxy version 23.2 onwards there is a feature with which we can hide the Galaxy UI directly.
It is meant to be used for embedding workflows, but can be accessed directly, which is perfect for purpose.

> <hands-on-title>Hiding the Galaxy UI for a Workflow</hands-on-title>
>
> 1. **Opening the Workflow Preview**
>    - Navigate to your Workflows list
>    - Press "{% icon galaxy-eye %} View" on your desired Workflow
>    - You should now see the preview page, which should look similar to this:
>      ![Galaxys Workflow Preview page with the workflow being rendered next to an info-box](./images/workflow-preview.png)
> 2. **Add the following to the URL in your browser:**
>
>    ```
>    &embed=true&buttons=false&about=false&heading=false&minimap=false&zoom_controls=false
>    ```
> 
> 3. **You should now see your Workflow rendered as shown below:**
>
>    ![Workflow preview page with all UI elements except for the Workflow itself hidden](./images/workflow-preview-embedded.png)
>    In this view, you can still zoom in and out using your mouse-wheel, and drag the canvas around.
>
{: .hands_on}

> <details-title>What does the text in the URL do?</details-title>
>
> These are so called query-parameters. They are chained to the existing parameters with a `&` symbol. 
> Each parameter tells galaxy to render the UI slightly differently.
> If you're curios at what they all do, try removing them individually and reloading the page.
>
{: .details}

> <comment-title></comment-title>
> If you cannot see a preview page like in the first screenshot, or if the query parameters have no effect,
> you are on a Galaxy version older than 23.2
>
> You can still follow the rest of this tutorial to learn how to make high-resolution screenshots.
{: .comment}


## Exporting high-res screenshots

Now that we have a view of our Workflow without the Galaxy UI, let's make a high-res screenshot using Firefox.

<div class="Print" markdown="1">
Firstly position your Workflow, so all of it is visible in the way you would like it to be on the poster.
</div>

<div class="Web" markdown="1">
Firstly position your Workflow, so all of it is visible in the way you would like it to be on the screenshot.
</div>

> <tip-title>Can't fit your entire Workflow at once?</tip-title>
> 
> If you zoomed out as far as possible, but can't view all of your Workflow, try the following:
> * Navigate back to the full Workflow preview page (without the UI hidden)
> * Click anywhere outside of the Workflow preview (e.g. on the about section)
> * Zoom out the entire browser window a bit, by using <kbd>Ctrl</kbd> + <kbd>Mouse Wheel</kbd>, or <kbd>Ctrl</kbd> + <kbd>-</kbd>
> * Re-add the query parameters we added in the previous section
> * If you still can't fit all of it, repeat the above steps
{: .tip}

> <hands-on-title>Creating a high-resolution screenshot</hands-on-title>
> 
> 1. **Open the Firefox Javascript console.** To do this, either:
>    - Use the shortcut <kbd>Ctrl</kbd> + <kbd>Shift</kbd> + <kbd>K</kbd> (On OS X <kbd>Cmd</kbd> + <kbd>Shift</kbd> + <kbd>K</kbd>)
>    - Or navigate to the console using your mouse:
>       * Menu
>       * More Tools
>       * Web Developer Tools
>       * Console Tab
>
> 2. **Resize or move the console**
>
>    The console may be in the way of your Workflow.
>
>    You can make it take up less space by dragging down the bar at the top.
>
>    You can also reposition it to the side, by selecting the three dots in the upper right corner of the console,
>    and choosing a different position from: "Dock to Bottom", "Dock to Right", "Dock to Left", "Separate Window".
>
> 3. **Make the screenshot**
>
>    In the Javascript console type:
>    ```
>    :screenshot --dpr 6
>    ```
>    After a short moment, a new screenshot should appear in your downloads folder.
>
>    The value "6" after the `--dpr` option is just an example. Keep reading to learn how to choose the right dpr value.
>
{: .hands_on}

<div class="Print" markdown="1">
> <tip-title>Choosing the right dpr value for print</tip-title>
>
> `dpr` stands for **D**evice **P**ixel **R**atio.
>  
> This is how much Firefox will scale up your screenshot. A dpr of 2, will make your screenshot twice as wide and high as you browser window.
> 
> Always go higher than you think you need, when choosing a dpr for print.
> Here's a rule-of thumb on how to determine the correct dpr:
>
> * Think about how much bigger your poster is going to be than your browser window, in physical dimensions.
>   If unsure, round up.
> * Multiply that number by 2
>
> Example: If your poster will be roughly 4 times larger than your browser window, choose a dpr of 8.
>
{: .tip}
</div>

<div class="Web" markdown="1">
> <tip-title>Choosing the right dpr value for your screenshot</tip-title>
>
> `dpr` stands for **D**evice **P**ixel **R**atio.
>  
> This is how much Firefox will scale up your screenshot. A dpr of 2, will make your screenshot twice as wide and high as you browser window.
>
> Choosing the right dpr for a screenshot depends on how much you've zoomed out.
> If you've zoomed out to about half the starting size, go with a dpr of two.
> If you want your screenshot to be extra crisp, add 2.
>
> Example: You zoomed out to 25%. Use a dpr of 4 (`25% * 4 = 100%`), then add 2 for extra detail equals a dpr of 6.
>
> To check if you've chosen an appropriate dpr, open the screenshot in firefox,
> click it to zoom in to 100%, and check if all text is clearly readable.
>
{: .tip}
</div>

# Compressing Workflow Screenshots

<div class="Print" markdown="1">
You may have noticed the file-size of your screenshot it very large.
Some print services may have restrictions on file size, and you won't be able to upload your screenshot for printing.
</div>

<div class="Web" markdown="1">
You may have noticed the file-size of your screenshot it very large.
You will likely not be able to upload it anywhere, as it is larger than most services allow images to be.
</div>

In the following section, you will learn how to compress your screenshot to a smaller file-size.

## Palette Compression

There are many ways to compress an image.
Palette compression is particularly well suited for workflows.
It can heavily reduce file size with minimal loss, if the amount of colors in an image is fairly low.

To compress our image, we will be using a piece of software called [GraphicsMagick](http://www.graphicsmagick.org/)

This tool is known to be installed on [usegalaxy.eu](https://usegalaxy.eu), we suggest you follow these steps there:

> <hands-on-title>Uploading Your Image</hands-on-title>
>
> 1. Open [usegalaxy.eu](https://usegalaxy.eu)
> 1. At the top of the {% icon tool %} **Tools** panel (on the left), click {% icon galaxy-upload %} **Upload Data**
> 1. Select "Choose Local Files" from the bottom of the dialog
> 1. Select your saved screenshot, likely from your Downloads folder.
> 1. Click **Start**
> 1. Click **Close**
>
{: .hands_on }

With the data on your Galaxy, we can start trying to compress it!

> <hands-on-title>Compressing your screenshot</hands-on-title>
> 1. Run the {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_convert/graphicsmagick_image_convert/1.3.40+galaxy0) %} tool
>    - {% icon param-file %} *"Image to convert"*: `screenshot.png`
>    - {% icon param-select %} *"Reduce Color Palette"*: `Automatic`
>    - {% icon param-select %} *"Output Format"*: `png`
>
>    This will have GraphicsMagick use automatic palette compression.
>    This command may take a while.
>
> 1. **Open the compressed image** to check if everything is still clearly visible.
>    
>    Also check if the file size is now small enough for your purposes. (Click the dataset in your history to expand it to see the file size.)
>
> 1. If you want more control over your output file, you can provide a number of colours, instead of using the automatic reduction. Let's re-run the tool and provide a number of colours you want to keep in your output image. This number should be in powers of two (4, 8, 16, 32, 64 etc.).
>
>    Using smaller numbers will further reduce the file size at the cost of image quality.
>    Smaller values do not always yield smaller files! Experiment a bit, or go with the option from step 4 for simplicity.
>
> 1. Run the {% tool [Convert image format](toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_convert/graphicsmagick_image_convert/1.3.40+galaxy0) %} tool
>    - {% icon param-file %} *"Image to convert"*: `screenshot.png`
>    - {% icon param-select %} *"Reduce Color Palette"*: `Manual`
>    - {% icon param-select %} *"Color Palette Size"*: `4`
>    - {% icon param-select %} *"Output Format"*: `png`
>
> 1. Try different values until you get a result you like! Did you manage to create a smaller result than the automatic palette compression?
{: .hands_on }

> <tip-title>Tips for compressing images</tip-title>
> 1. JPEG is a naturally "lossy" format, you can try that instead of PNG
> 1. If the image is quite large, consider scaling it down with the "Resize %" parameter
> 1. Experiment with the colour palettes! You will likely be able to find one that is a good tradeoff between accuracy and file size for your use case.
{: .tip}


Here's an example workflow screenshot compressed from `2.4mb` to `830kb`. Click it to view it in full resolution.

![high resolution screenshot of a Workflow](./images/example-screenshot-compressed.png)

tools_json:
["Convert image format", "toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_convert/graphicsmagick_image_convert/1.3.40+galaxy0"]


topic: galaxy-interface
tutorial: collections
title: Using dataset collections
difficulty: intermediate
time_estimation: 30m
requirements_json: []
tags_json: ["collections"]
content_hash: 94be79df0cfd40c9bd41e8f30f758d14
last_modified: 2026-05-15T22:28:13.330894
zenodo_link: https://doi.org/10.5281/zenodo.5119008
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/collections/tutorial.html

content:
{% snippet faqs/gtn/galaxy_tested_with.md version="22.01" %}

Here we will show Galaxy features designed to help with the analysis of large numbers of samples. When you have just a few samples - clicking through them is easy. But once you've got hundreds - it becomes very annoying. In Galaxy we have introduced **Dataset collections** that allow you to combine numerous datasets in a single entity that can be easily manipulated.


# Getting data

First, we need to upload datasets. Cut and paste the following URLs to Galaxy upload tool (see a {% icon tip %} **Tip** on how to do this [below](#tip-upload-fastqsanger-datasets-via-links)).

```
https://zenodo.org/record/5119008/files/M117-bl_1.fq.gz
https://zenodo.org/record/5119008/files/M117-bl_2.fq.gz
https://zenodo.org/record/5119008/files/M117-ch_1.fq.gz
https://zenodo.org/record/5119008/files/M117-ch_2.fq.gz
https://zenodo.org/record/5119008/files/M117C1-bl_1.fq.gz
https://zenodo.org/record/5119008/files/M117C1-bl_2.fq.gz
https://zenodo.org/record/5119008/files/M117C1-ch_1.fq.gz
https://zenodo.org/record/5119008/files/M117C1-ch_2.fq.gz
```

> <hands-on-title>Set format to `fastqsanger.gz`</hands-on-title>
> The above datasets are in `fastqsanger.gz` format. It is necessary to explicitly set format in Galaxy. The {% icon tip %} **Tip** section below explains how to upload these data and set the correct format. There is a variety of [fastq format flavors](https://en.wikipedia.org/wiki/FASTQ_format) and it is difficult to guess them automatically.
>
> {% snippet faqs/galaxy/datasets_import_via_link.md format="fastqsanger.gz" %}
>
> {% snippet topics/assembly/tutorials/vgp_genome_assembly/faqs/dataset_upload_fastqsanger_via_urls.md %}
>
{: .hands_on}

## About these datasets

These datasets represent genomic DNA (enriched for mitochondria via a long range PCR) isolated from blood (`bl`) and cheek (buccal swab, `ch`) of mother (`M117`) and her child (`M117C1`) that was sequenced on an Illumina miSeq machine as paired-read library (250-bp reads; see our [2014](http://www.pnas.org/content/111/43/15474.abstract) manuscript for **Methods**):

- `M117-bl_1` - family 117, mother, *forward* (**F**) reads from **blood**
- `M117-bl_2` - family 117, mother, *reverse* (**R**) reads from **blood**
- `M117-ch_1` - family 117, mother, *forward* (**F**) reads from **cheek**
- `M117-ch_2` - family 117, mother, *reverse* (**R**) reads from **cheek**
- `M117C1-bl_1`- family 117, child, *forward* (**F**) reads from **blood**
- `M117C1-bl_2`- family 117, child, *reverse* (**R**) reads from **blood**
- `M117C1-ch_1`- family 117, child, *forward* (**F**) reads from **cheek**
- `M117C1-ch_2`- family 117, child, *reverse* (**R**) reads from **cheek**

# Creating a paired dataset collection

You can see that there are eight datasets forming four pairs. Obviously, we can manipulate them one-by-one (e.g., start four mapping jobs, call variants four times and so on), but this will unnecessarily tedious. Moreover, imagine if you have 100s or 1,000s of pairs: it will be impossible to process them individually.

This is exactly why we developed collections. Dataset collections allow combining multiple datasets into a single entity. Thus instead of dealing with four, a hundred, or a thousand of individual datasets you have only one item in Galaxy history to deal with.

Because our data is *paired* we need to create a hierarchical collection called **Paired Dataset Collection** or **Paired Collection**. In such collection there are two layers. The first layer corresponds to individual samples (e.g., `M117-bl`). The second layer represent `forward` and `reverse` reads corresponding to each sample:


![paired collection](../../images/collections/paired_collection.svg "The logic of Paired Collection. Here <em>N</em> datasets are bundled into a paired collection with two layers. The first layer corresponds to samples and the second to forward and reverse reads within each sample.")


To begin creating a collection we need to select datasets we would like to bundle. This is done using checkbox button of Galaxy's history menu. Fig. 2 below shows this process.



![selecting multiple datasets](../../images/collections/select_multiple_datasets.gif  "Selecting multiple datasets and creating a paired collection.")




The above process ended with appearance of Galaxy collection wizard. In this case Galaxy automatically assigned pairs using the `_1` and `_2` endings of dataset names. Let's however pretend that this did not happen. Click on **Unpair all** (highlighted in red in the figure above) link and then on **Filters** link (see animation in Fig. 3). The interface will change into its unpaired state.

Here datasets containing the first (forward) and the second (reverse) read are differentiated by having `_1` and `_2` in the filename. We can use this feature in dataset collection wizard to pair our datasets.  Type `_1` in the left **Filter text** text box and `_2` in the right. You will see that the dataset collection wizard will automatically filter lists on each side of the interface. Now you can either click **Auto pair** if pairs look good to you (proper combinations of datasets are listed in each line) or pair each forward/reverse group individually by pressing **Pair these datasets** button separating each pair.

Now it is time to name the collection: type `M117-collection` in **Name** text box and create the collection by clicking **Create collection**. A new item will appear in the history.


![using collection wizard](../../images/collections/using_collection_wizard.gif "Working with collection wizard. Text above this figure explains each step.")


Clicking on collection will expand it to show four pairs it contains (panel **B**). Clicking individual pairs will expand them further to reveal **forward** and **reverse** datasets (panel **C**). Expanding these further will enable one to see individual datasets (panel **D**).

![expanding collection](../../images/collections/expanding_collection.gif "To look what is inside a collection, just click on it.")

# Processing data organized as a collection

By now we see that a collection can be used to bundle a large number of items into a single history item. Galaxy tools take collection as input. Let's map reads contained in collection `M117-collection` against human mitochondrial genome. Before we can do this we need to upload mitochondrial genome using the following URL (see a {% icon tip %} **Tip** on how to do this [below](#tip-importing-via-links)):

```
https://zenodo.org/record/5119008/files/chrM.fa.gz
```

> <hands-on-title>Set format to `fasta.gz`</hands-on-title>
> The above dataset is in `fasta.gz` format. The {% icon tip %} **Tip** section below explains how to upload these data and set the correct format.
>
> {% snippet faqs/galaxy/datasets_import_via_link.md reset_form="True" link="https://zenodo.org/record/5119008/files/chrM.fa.gz" format="fasta.gz" %}
{: .hands_on}

## Mapping reads

**BWA-MEM** {% icon tool %} is a widely used sequence aligner for short-read sequencing datasets such as those we are analysing in this tutorial. (You can find the tool by typing `BWA MEM` in the search box at the top left corner of Galaxy interface).

> <hands-on-title>Map sequencing reads to reference genome</hands-on-title>
>
> Run {% tool [Map with BWA-MEM](toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1) %} with the following parameters:
>    - *"Will you select a reference genome from your history or use a built-in index?"*: `Use a genome from history and build index`
>        - {% icon param-file %} *"Use the following dataset as the reference sequence"*: `chrM.fa.gz` (The mitochondrial genome we just uploaded)
>    - *"Single or Paired-end reads"*: `Paired Collection`
>        - {% icon param-file %} *"Select a paired collection"*: `M117-collection` (the collection we built at the beginning of this tutorial.)
>    - *"Set read groups information?"*: `Do not set`
>    - *"Select analysis mode"*: `1.Simple Illumina mode`
>
> The interface should look like this:
>
>
> ![bwa_mem_interface](../../images/collections/bwa_mem_interface_coll_tut.png "Tool interface")
>
>
>    - Click **Run Tool** button
>
{: .hands_on}

You will see jobs being submitted and new datasets appearing in the history. Because our collection contains four paired datasets Galaxy will actually generate four separate `BWA-MEM` jobs. In the end this `BWA-MEM` run will produce a new collection containing four (4) BAM datasets. Let's look at this collection by clicking on it (panel **A** in the figure below). You can see that now this collection is no longer paired (compared to the collection we created in the beginning of this tutorial). This is because `BWA-MEM` takes forward and reverse data as input, but produces only a single BAM dataset as the output. So what we have in the result is a *list* of four dataset (BAM files; panel **B**). If you click on any of the datasets you will see that it is indeed a BAM dataset (panel **C**).

![bwa_memCollection_ABC](../../images/collections/collection_expansion.svg)

## Calling variants

After we mapped reads against the mitochondrial genome, we can now call variants. In this step a variant calling tool `lofreq` will take a collection of BAM datasets (the one produced by `BWA-MEM`), identify differences between reads and the reference, and output these differences as a collection of [VCF](https://en.wikipedia.org/wiki/Variant_Call_Format) datasets.

> <hands-on-title>Call variants  </hands-on-title>
>
> Run {% tool [Call variants](toolshed.g2.bx.psu.edu/repos/iuc/lofreq_call/lofreq_call/2.1.5+galaxy1) %} with the following parameters:
>    - {% icon param-file %} *"Input reads in BAM format"*: `Map with BWA-MEM...` (output of **BWA-MEM** {% icon tool %})
>    - *"Choose the source for the reference genome"*: `History`
>        - {% icon param-file %} *"Reference"*: `chrM.fa.gz (as fasta)` (Input dataset)
>    - *"Call variants across"*: `Whole reference`
>    - *"Types of variants to call"*: `SNVs and indels`
>
> The interface should look like this:
>
> ------
>
> ![lofreq_interface](../../images/collections/lofreq_interface.png)
>
> ------
>
>    - Click **Run Tool** button
>
{: .hands_on}

## Create table of variants using **SnpSift Extract Fields**

We will now convert VCF datasets into tab delimited format as it will be easier to work with. This will be done with `SNPSift`: a tool specifically designed for manipulation of tab-delimited data.


> <hands-on-title>Create table of variants</hands-on-title>
>
> Run {% tool [SnpSift Extract Fields](toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_extractFields/4.3+t.galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Variant input file in VCF format"*: `Call variants on collection...` (output of **Call variants with lofreq** {% icon tool %})
>    - *"Fields to extract"*: `CHROM POS REF ALT QUAL DP AF SB DP4`
>    - *"One effect per line"*: `Yes`
>
> The interface should look like this:
>
> ------
>
> ![snpsift_interface](../../images/collections/snpsift_interface.png)
>
> ------
>
>    - Click **Run Tool** button
>
{: .hands_on}


As a result of this operation we now have a collection of four tab delimited files. Yet, ultimately we want to summaize these data as one final table. The next step does just that.

## Collapse data into a single dataset

We now extracted meaningful fields from VCF datasets. But they still exist as a collection. To move towards secondary analysis we need to **collapse** this collection into a single dataset. For more information about collapsing collections see this video:

{% include _includes/youtube.html id="ypuFZ1RKMIY" title="Collapsing collections" %}

> <hands-on-title>Collapse a collection</hands-on-title>
>
> Run {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: `SnpSift Extract Fields ...` (output of **SnpSift Extract Fields** {% icon tool %})
>    - "*Keep one header line*": `Yes`
>    - "*Prepend File name*": `Yes`
>    - "*Where to add dataset name*": `Same line and each line in dataset`
>
> The interface should look like this:
>
> ------
>
> ![snpsift_interface](../../images/collections/collapse_collection.png)
>
> ------
>
>    - Click **Run Tool** button
>
{: .hands_on}

You can see that this tool takes lines from all collection elements (in our case we have two), add element name as the first column, and pastes everything together. So if we have a collection as an input:

> <code-in-title>A collection with two items</code-in-title>
> A collection element named `M117-bl.fq`
>
>```
>chrM   152 T C  3707.0 1242 0.99 0 2,2,540,697
>chrM 16519 T C 35149.0 1033 0.99 0 1,1,611,420
>```
>
>A collection element named `M117-ch.fq`:
>
>```
chrM   152 T C  4098.0 1440 0.99 0 0,1,575,863
chrM 16519 T C 36574.0 1039 0.99 2 3,0,713,321
>```
>
>A collection element named `M117C1-bl.fq`:
>
>```
>chrM   152 T C  4888.0 1235 1.00 0 0,0,548,687
>chrM 16519 T C 35220.0 1042 0.99 0 0,0,598,443
>```
>
>A collection element named `M117C1-ch.fq`:
>
>```
>chrM   152 T C  2757.0 1413 0.99 0 2,2,576,833
>chrM 16455 G A    54.0  100 0.04 0 89,7,4,0
>chrM 16519 T C 36363.0 1061 0.99 6 3,4,691,362
>```
>
{: .code-in}

We will have a single dataset as the output:

> <code-out-title>A single dataset</code-out-title>
>
>then the **Collapse Collection** {% icon tool %} will produce this:
>
>```
>M117-bl.fq   chrM   152 T C  3707.0 1242 0.99 0 2,2,540,697
>M117-bl.fq   chrM 16519 T C 35149.0 1033 0.99 0 1,1,611,420
>M117-ch.fq   chrM   152 T C  4098.0 1440 0.99 0 0,1,575,863
>M117-ch.fq   chrM 16519 T C 36574.0 1039 0.99 2 3,0,713,321
>M117C1-bl.fq chrM   152 T C  4888.0 1235 1.00 0 0,0,548,687
>M117C1-bl.fq chrM 16519 T C 35220.0 1042 0.99 0 0,0,598,443
>M117C1-ch.fq chrM   152 T C  2757.0 1413 0.99 0 2,2,576,833
>M117C1-ch.fq chrM 16455 G A    54.0  100 0.04 0 89,7,4,0
>M117C1-ch.fq chrM 16519 T C 36363.0 1061 0.99 6 3,4,691,362
>```
{: .code-out}

you can see that added a column with dataset ID taken from collection element name.

## We did not fake this:
The history described in this page is accessible directly from here:

* History [**Collections (full analysis)**](https://usegalaxy.org/u/cartman/h/collection-tutorial)

From there you can import histories to make them your own.


# Collection operations

In this brief analysis we took four paired datasets, created a collection, analyzed this collection and finally created a single report. Such "lifecycle" is shown in the figure below. Here we started with eight fastq datasets representing four paired end samples. A paired collection was reduced to a list of BAM datasets by `BWA-MEM`. Varinat calling by `lofreq` and field extraction with `SnpEff` maintained collection structure: these tools processed four individual datasets changing their formats from BAM to VCF, and from VCF to Tab-delimited. Finally, we collapsed collection by merging its content into a single dataset.

![Collection lifecycle](../../images/collections/collection_lifecycle.svg "Collection lifecycle. Arrows = individual fastq datasets; Four shades of yellow = four samples analyzed in this example. ")

The last step of our analysis, collapsing a collection, is an example of a *collection operation*. Galaxy contains an entire section of tools designed for handling of collection data. These can be classified as:

 - Tools that manipulate elements within a collection
 - Tools that change collection structure
 - Tools that combine elements of a collection

Let's look at these categories in more detail:

## Tools that manipulate elements within a collection

### Extract dataset

{% icon tool %} **Extract dataset** extracts datasets from a collection based on either position or identifier.

The tool allow extracting datasets based on position (**The first dataset** and **Select by index** options) or name (**Select belement identifier** option). This tool effectively collapses the inner-most collection into a dataset. For nested collections (e.g a list of lists of lists: outer:middle:inner, extracting the inner dataset element) a new list is created where the selected element takes the position of the inner-most collection (so outer:middle, where middle is not a collection but the inner dataset element).

### Filter empty

{% icon tool %} **Filter empty** removes empty elements from a collection.

This tool takes a dataset collection and filters out (removes) empty datasets. This is useful for continuing a multi-sample analysis when downstream tools require datasets to have content.

![Filtering empty datasets](../../images/collections/filter_empty.svg)

### Filter failed datasets

{% icon tool %} **Filter failed datasets** removes datasets in error (red) from a collection.

This tool takes a dataset collection and filters out (removes) datasets in the failed (red) state. This is useful for continuing a multi-sample analysis when one or more of the samples fails at some point.

![filter_failed](../../images/collections/filter_error.svg)

### Build list

{% icon tool %} **Build list** creates a new list collection from individual datasets or collections.

This tool combines individual datasets or collections into a new collection. The simplest scenario is building a new colection from individual datasets (case **A** in the image below). You can merge a collection with individual dataset(s). In this case (see **B** in the image below) the individual dataset(s) will be merged with each element of the input collection to create a nested collection. Finally, two or more collection can be merged together creating a nested collection (case **C** in the image below).

![Build list](../../images/collections/build_list.svg)

### Filter collection

{% icon tool %} **Filter collection** removes elements from a collection using a list supplied in a file.

This tools allow filtering elements from a data collection.  It takes an input collection and a text file with names (i.e. identifiers). The tool behaviour is controlled by **How should the elements to remove be determined?** drop-down. It has the following options:

#### Remove if identifiers are ABSENT from file

Given a collection:

```
 Collection: [Dataset A]
             [Dataset B]
             [Dataset X]
```

and a text file:

```
             A
             B
             Z
```

the tool will return two collections:


```

 (filtered):  [Dataset A]
              [Dataset B]

 (discarded): [Dataset X]
```

#### Remove if identifiers are PRESENT in file

Given a collection:

```
 Collection: [Dataset A]
             [Dataset B]
             [Dataset X]
```
and a text file:

```
             A
             B
             Z
```

the tool will return two collections:

```
 (filtered):  [Dataset X]

 (discarded): [Dataset A]
              [Dataset B]

```

### Relabel identifiers

{% icon tool %}  **Relabel identifiers** changes identifiers of datasets within a collection using identifiers from a supplied file.

New identifiers can be supplied as either a simple list or a tab-delimited file mapping old identifier to the new ones. This is controlled using **How should the new identifiers be specified?** drop-down:

#### Using lines in a simple text file

Given a collection:

```
 Collection: [Dataset A]
             [Dataset B]
             [Dataset X]
```

and a simple text file:


```
             Alpha
             Beta
             Gamma
```

the tool will return:

```
 Collection: [Dataset Alpha]
             [Dataset Beta]
             [Dataset Gamma]
```

#### Map original identifiers to new ones using a two column table

Given a collection:

```
 Collection: [Dataset A]
             [Dataset B]
             [Dataset X]
```

and a simple text file (you can see that entries do not have to be in order here):


```
             B Beta
             X Gamma
             A Alpha
```

the tool will return:

```
 Collection: [Dataset Alpha]
             [Dataset Beta]
             [Dataset Gamma]
```

### Sort collection

{% icon tool %} **Sort collection** sorts dataset collection alphabetically, numerically, or using predetermined order from a supplied file.

#### Numeric sort

The tool sort in ascending order. When *numeric* sort is chosen, the tool ignores non-numeric characters. For example, if a collection contains the following elements:

```
 Collection: [Horse123]
             [Donkey543]
             [Mule176]
```

The tool will output:

```
 Collection: [Horse123]
             [Mule176]
             [Donkey543]
```

#### Sorting from file

Alternative, one can supply a single column text file containing elements identifiers in the desired sort order. For example, suppose there a collection:

```
 Collection: [Horse123]
             [Donkey543]
             [Mule176]
```

and a file specifying sort order:

```
 Donkey543
 Horse123
 Mule176
```

the output will predictably look like this:

```
 Collection: [Donkey543]
             [Horse123]
             [Mule176]
```

### Tag collection

{% icon tool %}  **Tag collection** adds tags (including name: and group: tags) to collection elements.

The relationship between element names and tags is specified in a two column tab-delimited file. This file may contain less entries than elements in the collection. In that case only matching list identifiers will be tagged.

To create name: or group: tags prepend them with `#` (you can also use `name:`) or `group:`, respectively.

More about tags

> <tip-title>More about tags</tip-title>
> Galaxy allows tagging datasets to facilitate analyses. There are several types of tags including simple tags, name tags, and group tags. **Simple** tags allow you to attach an alternative label to a dataset, which will make it easier to find it later. **Name** tags allow you to track propagation of a dataset through the analyses: all datasets derived from the initial dataset labeled with a name tag will inherit it. Finally, **group** tags allow you to label group of datasets. This is useful. for example, for differential expression analysis where you can have two groups of datasets labeled as "treatment" and "control".
>
>To learn mote about tags go to [training site]({% link search.md %}?query=tags).
{: .tip}

## Tools that change collection structure

### Flatten collection

{% icon tool %} **Flatten collection** collapses nested collection into a simple list.

This tool takes nested collections such as a list of lists or a list of dataset pairs and produces a flat list from the inputs. It effectively "flattens" the hierarchy. The collection identifiers are merged together (using `_` as default) to create new collection identifiers in the flattened result:

![Flatten collection](../../images/collections/flatten.svg)

### Merge collections

{% icon tool %}  **Merge collections** takes two or more collections and creates a single collection from them.

By default the tool assumes that collections that are being merged have unique dataset names. If it not the case only one (the first) of the datasets with a repeated name will be included in the merged collection. For example, suppose you have two collections. Each has two datasets named "A" and "B":
```
 Collection 1: [Dataset A]
               [Dataset B]
               [Dataset X]
 Collection 2: [Dataset A]
               [Dataset B]
               [Dataset Y]
```

Merging them will produce a single collection with only two datasets:

```
 Merged Collection: [Dataset A]
                    [Dataset B]
                    [Dataset X]
                    [Dataset Y]
```

This behavior can be changed by clicking on "*Advanced Options*" link. The following options are available:

#### Keep first instance (Default behavior)

Input:

```
 Collection 1: [Dataset A]
               [Dataset B]
               [Dataset X]
 Collection 2: [Dataset A]
               [Dataset B]
               [Dataset Y]
```

Output:

```
 Merged Collection: [Dataset A]
                    [Dataset B]
                    [Dataset X]
                    [Dataset Y]
```

Here if two collection have identical dataset names, a dataset is chosen from the *first* collection.


#### Keep last instance

Input:

```
 Collection 1: [Dataset A]
               [Dataset B]
               [Dataset X]
 Collection 2: [Dataset A]
               [Dataset B]
               [Dataset Y]
```

Output:

```
 Merged Collection: [Dataset A]
                    [Dataset B]
                    [Dataset X]
                    [Dataset Y]
```

Here if two collection have identical dataset names, a dataset is chosen from the *last* collection.


#### Append suffix to conflicted element identifiers

Input:

```
 Collection 1: [Dataset A]
               [Dataset B]
               [Dataset X]
 Collection 2: [Dataset A]
               [Dataset B]
               [Dataset Y]
```

Output:

```
 Merged Collection: [Dataset A_1]
                    [Dataset B_1]
                    [Dataset A_2]
                    [Dataset B_2]
                    [Dataset X]
                    [Dataset Y]
```


#### Append suffix to conflicted element identifiers after first on encountered

Input:

```

 Collection 1: [Dataset A]
               [Dataset B]
               [Dataset X]
 Collection 2: [Dataset A]
               [Dataset B]
               [Dataset Y]
```

Output:

```
 Merged Collection: [Dataset A]
                    [Dataset B]
                    [Dataset A_2]
                    [Dataset B_2]
                    [Dataset X]
                    [Dataset Y]
```

#### Append suffix to every element identifier

Input:

```
 Collection 1: [Dataset A]
               [Dataset B]
               [Dataset X]
 Collection 2: [Dataset A]
               [Dataset B]
               [Dataset Y]
```

Output:

```
 Merged Collection: [Dataset A_1]
                    [Dataset B_2]
                    [Dataset A_2]
                    [Dataset B_2]
                    [Dataset X_1]
                    [Dataset Y_2]
```

#### Fail collection creation

This option will simply trigger an error.

### Zip collection

{% icon tool %} **Zip collection** takes two collections and creates a paired collection from them.

If you have one collection containing only forward reads and one containing only reverse, this tools will "zip" them together into a simple paired collection. For example, given two collections with `forward` and `reverse` reads they can be "zipped" into a single paired collection:

![Zip collection](../../images/collections/zip.svg)

### Unzip collection

{% icon tool %} **Unzip collection** takes a paired collection and "unzips" it into two simple dataset collections (lists of datasets).

Given a paired collection of forward and reverse reads this tool will "unzip" it into two collections containing forward and reverse reads, respectively:

![Unzip collection](../../images/collections/unzip.svg)

## Tools that combine elements of a collection

### Column join

{% icon tool %} **Column join** merges elements of a collection on a given column.

If you have a collection with three elements (image below), merging it on the first column will first produce a union on values found in the first column of each elements and then paste elements having the same value side-by-side:

![Column join](../../images/collections/join_on_column.svg)

### Collapse collection

{% icon tool %} **Collapse collection** merges elements together (head-to-tail) in the order of the collection. Its power comes from the ability to add identifiers when it performs the merge. Identifiers can be added in variety of ways specified by the **Prepend File name** option as shown in the figure below (we used option **A** in the last step of this tutorial). **A** = `Same line and each line in dataset`; **B** = `Same line and only once per dataset`; **C** =  `Line above`

![Collapse collection](../../images/collections/collapse_collection.svg)

<!-- GTN:IGNORE:002 -->

tools_json:
["Map with BWA-MEM", "toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.17.1", "Call variants", "toolshed.g2.bx.psu.edu/repos/iuc/lofreq_call/lofreq_call/2.1.5+galaxy1", "SnpSift Extract Fields", "toolshed.g2.bx.psu.edu/repos/iuc/snpsift/snpSift_extractFields/4.3+t.galaxy0", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/4.0"]


topic: galaxy-interface
tutorial: upload-data-to-ena
title: Submitting sequence data to ENA
difficulty: intermediate
time_estimation: 1h
requirements_json: []
tags_json: ["covid19"]
content_hash: 991e706f7d938a215a7d5cf3201fcb90
last_modified: 2026-05-15T22:28:13.362341
zenodo_link: https://doi.org/10.5281/zenodo.5163611
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/galaxy-interface/tutorials/upload-data-to-ena/tutorial.html

content:
Raw reads contain valuable information, such as coverage depth and quality scores, that is lost in a consensus sequence.
Submission of raw reads to public repositories allows reuse of data and reproducibility of analysis and enables discovery of minor allelic variants and intrahost variation, for example during the recent COVID-19 pandemic ({% cite Maier2021 %}).

The European Nucleotide Archive is an Open and FAIR repository of nucleotide data. As part of the International Nucleotide Sequence Database Collaboration (INSDC), ENA also indexes data from the NCBI and DDBJ {% cite Arita2020 %}. Data submitted to ENA must be accompanied by sufficient metadata. You can learn more from this [introductory slide deck]({% link topics/galaxy-interface/tutorials/upload-data-to-ena/slides.html %}) or directly from [ENA](https://ena-docs.readthedocs.io/en/latest/submit/general-guide/metadata.html).


In this tutorial we will show you how to use Galaxy's 'ENA Upload tool' to submit raw sequencing reads, consensus sequences and their associated metadata to ENA ({% cite Roncoroni2021 %}). You will learn to add your ENA Webin credentials to Galaxy, input metadata interactively or via a metadata template and submit the reads to ENA (test) server using Galaxy's 'ENA upload tool'.
Specifically, we will use one ONT sequencing file to demonstrate interactive metadata input and two sets of PE Illumina reads to demonstrate how to use the ENA metadata template. Finally, we will submit consensus sequences to ENA using 'Submit consensus sequence to ENA' tool.

Data will be submitted to ENA's test server and will not be public.

> <comment-title>Nature of the input data</comment-title>
> We will use data derived from sequencing data of bronchoalveolar lavage fluid (BALF) samples obtained from early COVID-19 patients in China as our input data.
> Human traces have been removed in [Galaxy]({% link topics/sequence-analysis/tutorials/human-reads-removal/tutorial.md %}).
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Adding ENA Webin credentials to your Galaxy user information

In order to submit data to ENA, you need to have a valid Webin account. If you don't have one already you can register for one [on ENA's Registration Page](https://www.ebi.ac.uk/ena/submit/sra/#registration). Webin credentials need to be included in your Galaxy user information before you can use the ENA Upload tool.

> <hands-on-title>Add Webin credentials to your Galaxy user information</hands-on-title>
>
> 1. If you have not already done so, log in to usegalaxy.eu
> 2. Navigate to *"User"* > *"Preferences"* on the top menu
>   - Click on <i class="fa fa-user" aria-hidden="true"></i> **Manage Information**
>     - Scroll down to *"Your ENA Webin account details"* and fill in your ***ENA Webin ID*** and ***Password***
>![ENA Webin Account details in Galaxy](../../images/upload-data-to-ena/ENA-credentials.png "ENA Webin Account details")
{: .hands_on}

# Submitting raw sequence data (reads) to the ENA
## Option 1: submitting to ENA using interactive metadata generator

In this first example, you will submit one ONT sequence file using the interactive metadata forms from the ENA Upload tool. This method is only convenient for small submissions. For bulk submissions, we recommend you use the metadata template described below in Option 2.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Upload the ONT data from Zenodo via URLs
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    The URL for our example data is this:
>
>    ```
>    https://zenodo.org/record/6912963/files/SRR10902284_ONT.fq.gz
>    ```
>
>
{: .hands_on}

Once the data is uploaded, we fill the metadata using  the ENA Upload tool. Interactive metadata forms are nested to fit [ENA's metadata model](https://ena-docs.readthedocs.io/en/latest/submit/general-guide/metadata.html). Briefly, you add Samples to a Study, Experiments to Samples and Runs to Experiments. The interactive metadata form does only two [ENA Sample Checklists](https://www.ebi.ac.uk/ena/browser/checklists), the basic minimal sample metadata and the ENA virus pathogen reporting standard checklist. Switch between the basic template and the virus pathogen one under the "Select your sample type" option. If you wish to include additional metadata from a sample checklist, please use Option 2 below.

We recommend always submitting to the test server before submitting to the public one.
After you confirm that all the data and metadata looks ok, you can go ahead and submit to the public ENA server.




> <hands-on-title>add metadata interactively and submit a single sequence to ENA</hands-on-title>
>
> 1. {% tool [ENA Upload tool](toolshed.g2.bx.psu.edu/repos/iuc/ena_upload/ena_upload/0.7.1) %}:
>    - *"Action to execute"*: `Add new (meta)data`
>    - *"Select the metadata input method"*: `Interactively (only recommended for small studies)`
>    - *"Add .fastq (.gz, .bz2) extension to the Galaxy dataset names to match the ones described in the input tables?"*: `No`
>    - *"Select your sample type"*: `Viral`
>    - Under "Submission options":
>       - *"Affiliation centre"*: your institution  
>       - *"Submit to ENA test server"*: `yes`
>       - *"Create test outputs without submitting (meta)data to ENA"*: `no`
> 2. Fill all metadata boxes and make sure that:
>    - *""*:
>    - *"Please select the type of study"*: `Whole Genome Sequencing`
>    - *"Enter the species of the sample"*: `Severe acute respiratory syndrome coronavirus 2`
>    - *"Enter the taxonomic ID corresponding to the sample species"*: `2697049`
>    - *"Host common name"*: `human`
>    - *"Host subject id"*: avoid using ID that can be use to trace samples back to patients
>    - *"Host scientific name"*: `Homo sapiens`
>    - *"Library strategy"*: `RNA-Seq`
>    - *"Select library source"*: `METAGENOMIC`
>    - *"Library selection"*: `RANDOM`
>    - *"Library layout"*: `SINGLE`
>    - *"Select the sequencing platform used"*: `Oxford Nanopore`
>    - *"Instrument model"*: `minION`
>    - *"Runs executed within this experiment"*
>      - {% icon param-files %} *"File(s) associated with this run"*: `SRR10902284_ONT.fq.gz`
>
{: .hands_on}

> <warning-title>Do not include personal identifiable data</warning-title>
> In some cases, some information is requested by ENA that may classify as personal or could be used to identify persons (e.g. 'host ID' for checklist ERC000033). Make sure that you do not publish any personal metadata that infringes privacy protection regulations in your jurisdiction.
{: .warning}

> <warning-title>Submit to the test server first</warning-title>
> Make sure *"Submit to ENA test server"*: `yes`. Otherwise your data will be submitted to the public server.
{: .warning}

Four metadata tables (Study, Sample, Experiment and Run), and a metadata ticket with submission information are generated. You can confirm a successful submission at ENA [test server](https://wwwdev.ebi.ac.uk/ena/submit/webin/) (or the [public server](https://www.ebi.ac.uk/ena/submit/sra), if you chose it).

> <tip-title>Metadata ticket</tip-title>
> Upon successful submission, a metadata ticket is generated. This contains information of the submission, including parseable metadata. Importantly, it contains Study, Sample, Run and Experiment accession numbers. The former two you will use later to link the consensus sequence to the raw data.
{: .tip}


## Option 2: submitting to ENA using a metadata template

For larger submissions, interactive metadata input can be tedious and not practical. In the second example, you will submit two sets of Illumina PE sequence files and input metadata using a template spreadsheet. This template contains all fields for ENA sample checklist *'ERC000033 - ENA virus pathogen reporting standard checklist'*. Tabular (.tsv and .xlsx) metadata templates for all sample checklist can be found in [this repository](https://github.com/ELIXIR-Belgium/ENA-metadata-templates).
For this tutorial, we provide you with a pre-filled template and encourage you to explore it.

> <hands-on-title>Upload and inspect data</hands-on-title>
>
> 1. Upload the ONT data from Zenodo via URLs:
>
>    ```
>    https://zenodo.org/record/6912963/files/SRR10903401_1.fastq.gz
>    https://zenodo.org/record/6912963/files/SRR10903401_2.fastq.gz
>    https://zenodo.org/record/6912963/files/SRR10903402_1.fastq.gz
>    https://zenodo.org/record/6912963/files/SRR10903402_2.fastq.gz
>    https://zenodo.org/record/6912963/files/metadata_template_ERC000033_mock_complete.xlsx
>    ```
> 2. Arrange the data into a paired dataset collection
>
>    {% snippet faqs/galaxy/collections_build_list_paired.md %}
>
>    For the example datasets this means:
>    - You need to tell Galaxy about the suffix for your forward and reverse reads, respectively:
>      - set the text of *unpaired forward* to: `_1.fastq.gz`
>      - set the text of *unpaired reverse* to: `_2.fastq.gz`
>      - click: `Auto-pair`
>
>      All datasets should now be moved to the *paired section* of the dialog, and the middle column there should show that only the sample accession numbers, *i.e.* `SRR10903401` and `SRR10903402`, will be used as the pair names.
>
>    > <warning-title>Paired collection names</warning-title>
>    > It is very important that the paired collection names contain no suffix (e.g. _1, _R1, etc.) or file extensions (.fastq, .fastq.gz). The submission tool will add these at runtime and leaving them in the paired collection names will cause a mismatch with the filenames in the metadata table.
>    {: .warning }
>    - Make sure *Hide original elements* is checked to obtain a cleaned-up history after building the collection.
>    - Give your collection a name
>    - Click *Create Collection*
>
> 3. Inspect the `GTN_tutorial_mock_metadata.xlsx` (filled-in template) file by clicking on the {% icon galaxy-eye %} (eye) icon 
>
>    https://github.com/enasequence/webin-cli
>
>    > <question-title></question-title>
>    >
>    > 1. How many metadata sheets are there?
>    > 2. How is the 'Sample' section in the template different from that in the interactive metadata input?
>    >
>    > > <solution-title></solution-title>
>    > > 1. There are four metadata sheets, one per metadata object (Study, Sample, Experiment, Run)
>    > > 2. The Sample section is more extensive in the template spreadsheet, because it contains all the fields from ENA ERC000033 sample checklist, as well as all 'Recommended' and 'Optional' fields in the other sections.
>    >  {: .solution }
>    {: .question}
>
{: .hands_on}


As before, the submission is done to the test server before submitting to the public one.

> <hands-on-title>use a metadata template and submit multiple sequences to ENA</hands-on-title>
>
> 1. {% tool [ENA Upload tool](toolshed.g2.bx.psu.edu/repos/iuc/ena_upload/ena_upload/0.7.1) %}:
>    - *"Action to execute"*: `Add new (meta)data`
>    - *"Select the metadata input method"*: `Excel file`
>    - *"Select the ENA sample checklist"*: `ENA virus pathogen reporting standard checklist (ERC000033)`
>    - *"Select Excel (.xlsx) file based on template"*: `metadata_template_ERC000033_mock_complete.xlsx`
>    - *"Select input data"*: `Paired dataset collection`
>    - *"List of paired-end sequencing data files"*: select the collection you made above during data upload
>    - Under "Submission options":
>       - *"Affiliation centre"*: your institution 
>       - *"Submit to ENA test server?"*: `yes`
>       - *"Create test outputs without submitting (meta)data to ENA"*: `no`
>
{: .hands_on}

> <warning-title>Do not include personal identifiable data</warning-title>
> In some cases, some information is requested by ENA that may classify as personal or could be used to identify persons (e.g. 'host ID' for checklist ERC000033). Make sure that you do not publish any personal metadata that infringes privacy protection regulations in your jurisdiction.
{: .warning}

> <warning-title>Submit to the test server first</warning-title>
> Make sure *"Submit to ENA test server?"*: `yes`. Otherwise your data will be submitted to the public server.
{: .warning}

Four metadata tables (Study, Sample, Experiment and Run), and a metadata ticket with submission information are generated. You can confirm a successful submission at ENA [test server](https://wwwdev.ebi.ac.uk/ena/submit/webin/) (or the [public server](https://www.ebi.ac.uk/ena/submit/sra), if you chose it).

> <tip-title>Metadata ticket</tip-title>
> Upon succesful submission, a metadata ticket is generated. This contains information of the submission, including parseable metadata. Importantly, it contains Study, Sample, Run and Experiment accession numbers. The former two you will use later to link the consensus sequence to the raw data.
{: .tip}

# Submitting consensus sequences to ENA

We produced consensus sequences for the Illumina data from Option 2 above following [SARS-CoV-2-PE-Illumina-WGS-variant-calling](https://workflowhub.eu/workflows/113?version=4), [SARS-CoV-2-variation-reporting](https://workflowhub.eu/workflows/109?version=5) and [COVID-19-consensus-construction](https://workflowhub.eu/workflows/138?version=4) workflows.

In this step we will submit one consensus sequence to ENA.
We will link it to the reads submitted in the first step using the accession numbers given to those submissions. Galaxy's 'ENA Upload tool' captures and stores metadata on a metadata ticket.


> <hands-on-title>submit consensus sequences to ENA</hands-on-title>
>
> 1. Upload the consensus sequence to Galaxy from Zenodo via URLs:
>    ```
>    https://zenodo.org/record/6912963/files/SRR10903401.fasta
>    ```
> 2. {% icon galaxy-eye %} Open the 'ENA submsission receipt' and find the Study and Sample accession numbers from the raw data submission.
> 3. {% tool [Submit consensus sequence to ENA](toolshed.g2.bx.psu.edu/repos/ieguinoa/ena_webin_cli/ena_consensus_submit/4.3.0) %}:
>    - *"Submit to test ENA server?"*: `yes`
>    - *"Validate files and metadata but do not submit"*: `no`
>    - Fill the assembly metadata. For our assembly:
>      - *"Assembly type"*: `Clone`
>      - *"Assembly program"*: `BWA-MEM`
>      - *"Molecule type"*: `genomic RNA`
>      - *"Coverage"*: `1000`
>      - *"Select the method to load study and sample metadata"*: `Fill in required metadata`
>      - *"Assembly name"*: give a name to your assembly.
>      - *"Study accession"*: `ERP139884` (you can find the Study accession number from your raw data submission metadata ticket)
>      - *"Sample accession"*: `ERS12519941` (you can find the Sample accession number from your raw data submission metadata ticket)
>      - *"Sequencing platform"*: `Illumina`
>    - Select the consensus sequence assembly file from your history: `SRR10903401.fasta`
>
{: .hands_on}

The output are a list of manifest (in our example one manifest) and a 'submission log'. The manifest file is the metadata required by [ENA's Webin CLI tool](https://github.com/enasequence/webin-cli). The 'submission log' can be useful to troubleshoot failed submissions.

tools_json:
["ENA Upload tool", "toolshed.g2.bx.psu.edu/repos/iuc/ena_upload/ena_upload/0.7.1", "Submit consensus sequence to ENA", "toolshed.g2.bx.psu.edu/repos/ieguinoa/ena_webin_cli/ena_consensus_submit/4.3.0"]


topic: community
tutorial: community-tool-table
title: Creation of resources listing all the tools and their metadata relevant to your community
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["Community", "SIG", "CoDex"]
content_hash: 9457dba6b01bc660048992b463ebaa40
last_modified: 2026-05-27T09:01:47.880171
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/community-tool-table/tutorial.html

content:
Galaxy offers thousands of tools. They are developed across various GitHub repositories. Furthermore, Galaxy also embraces the granular implementation of software tools as sub-modules. In practice, this means that tool suites are separated into Galaxy tools, also known as wrappers, that capture their component operations. Some key examples of suites include [Mothur](https://bio.tools/mothur) and [OpenMS](https://bio.tools/openms), which translate to tens and even hundreds of Galaxy tools.

While granularity supports the composability of tools into rich domain-specific workflows, this decentralized development and sub-module architecture makes it **difficult for Galaxy users to find and reuse tools**. It may also result in Galaxy tool developers **duplicating efforts** by simultaneously wrapping the same software. This is further complicated by a lack of tool metadata, which prevents filtering for all tools in a specific research community or domain, and makes it all but impossible to employ advanced filtering with ontology terms and operations like [EDAM ontology](https://edamontology.org/page).

The final challenge is also an opportunity: the global nature of Galaxy means that it is a big community. Solving the visibility of tools across this ecosystem and the potential benefits are far-reaching for global collaboration on tool and workflow development.

To provide the research community with a comprehensive list of available Galaxy tools, [Galaxy Codex](https://github.com/galaxyproject/galaxy_codex) was developed to collect Galaxy wrappers from a list of Git repositories and automatically extract their metadata (including Conda version, [bio.tools](https://bio.tools/) identifiers, and EDAM annotations). The workflow also queries the availability of the tools and usage statistics from the three main Galaxy servers ([UseGalaxy.org, UseGalaxy.eu and UseGalaxy.org.au](https://galaxyproject.org/use/)).

![A diagram illustrating the Galaxy Codex pipeline, showcasing the various steps involved in creating a community Galaxy tool table.](./images/galaxy_tool_metadata_extractor_pipeline.png "Workflow of the Galaxy Codex pipeline. Tool wrappers are parsed from different repositories and additional metadata is retrieved from bio.tools, BioConda, and the main public Galaxy servers. Upon filtering and manual curation of the data for specific scientific communities, the data is transformed into interactive web tables and a tool usage statistic-base word cloud that can be integrated into any website.")

The pipeline creates a [table with all tools and their metadata](https://galaxyproject.github.io/galaxy_codex/communities/). This table can be **filtered to only include tools that are relevant to a specific research community**. Here is an example of the microbial related tools:

<iframe id="edam" src="https://galaxyproject.github.io/galaxy_codex/communities/microgalaxy/resources/" frameBorder="0" width="100%" height="600px"> ![Interactive table for microgalaxy tools](./images/microgalaxy_tools.png) </iframe>

The generated community-specific table can be used as is (e.g. downloadable tsv file) and/or embedded (e.g. into a Galaxy Hub page or Galaxy subdomain). This table allows further filtering and searching for fine-grained tool selection.

The pipeline is **fully automated** and executes on a **weekly** basis. Any research community can apply the pipeline to create a table specific to their community.

The aim of this tutorial is to create such a table for a community.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add your community to the Galaxy CoDex

You first need to check if your community is in the [Galaxy CoDex](https://github.com/galaxyproject/galaxy_codex/tree/main/communities), a central resource for Galaxy communities.
If the community is already there, you can move to the next step of this tutorial.

If your community is not already included, follow these steps :
> <hands-on-title>Add your community to the Galaxy CoDex</hands-on-title>
>
> You need to create a new folder in the data/community folder within Galaxy CoDex code source.
> 1. If not already done, fork the [Galaxy Codex repository](https://github.com/galaxyproject/galaxy_codex)
> 2. Go to the `communities` folder
> 3. Click on **Add file** in the drop-down menu at the top
> 4. Select **Create a new file**
> 5. Fill in the `Name of your file` field with:  name of your community + `/metadata/categories`
>    This will create a folder for your community, a new folder for your community called metadata and a file called categories.
> 6. Click on **Commit changes**
> 7. Fill in the commit message with something like `Add X community`
> 8. Click on `Create a new branch for this commit and start a pull request`
> 9. Create the pull request by following the instructions
>
{: .hands_on}

# Pull a list of tools relevant to your community from the Galaxy ToolShed

To add tools to your community tool table, you will need to indicate a list of tags relevant to your community, and tools associated with this tag will be automatically pulled from the [Galaxy ToolShed](https://toolshed.g2.bx.psu.edu/) on a weekly basis. Only tools in the selected ToolShed categories will be added to the filtered table. As a result, it is recommended to include broad categories.
You will then be able to remove tools that are not relevant to your community or deprecated.

> <hands-on-title>Select the ToolShed categories</hands-on-title>
>
> 1. Go to the [Galaxy ToolShed](https://toolshed.g2.bx.psu.edu/)
> 2. On the main page, pick the most obvious categories that represent tools used by your community and note the names of these categories
> 3. Search on the [Galaxy ToolShed](https://toolshed.g2.bx.psu.edu/) for some of the popular tools in your community
> 4. Open the tool entries on the ToolShed, and note their categories
{: .hands_on}

# Add the list of relevant tags for your community to the categories file\.

> <hands-on-title>Add the ToolShed categories to the categories file</hands-on-title>
> 1. Open or create the file named `categories` in your community metadata folder (`communities/<your community>/metadata/categories`)
> 2. Add the name of the categories relevant to your community in the `categories` file you started above, with 1 ToolShed category per row. For example, see [a list of valid ToolShed categories](https://toolshed.g2.bx.psu.edu/).
>
>    For example:
>    ```
>    Assembly
>    Metagenomics
>    ```
{: .hands_on}

Once you have a list of the ToolShed categories that you wish to keep, you can submit this to Galaxy CoDex.

> <hands-on-title>Submit the new list of categories to Galaxy CoDex</hands-on-title>
>
> 1. Click on **Commit changes** at the top
> 2. Fill in the commit message with something like `Add tools categories for my community`
> 3. Click on `Create a new branch for this commit and start a pull request`
> 4. Create the pull request by following the instructions
>
{: .hands_on}

The Pull Request will be reviewed. Make sure to respond to any feedback.

On the Sunday following the merging of the previous pull request, two files will be generated :
- A JSON file with all tool suites and a short description will be created in `communities/<your community>/resources/tools_filtered_by_ts_categories.json`.
- A table with all tool suites and a short description will be created in `communities/<your community>/metadata/tool_status.tsv`.

# Review the generated table to curate tools

The generated table will contain all the tools associated with the ToolShed categories that you selected. However, not all of these tools might be interesting for your community.

Galaxy Codex allows for an additional optional filter for tools that can be defined by the community curator (maybe that is you!).

The additional filter must be stored in the file called `tools_status.tsv` located in `communities/<your community>/metadata`. The file must include at least 3 columns (with a header):
1. `Suite ID`
2. `To keep` indicating whether the tool should be included in the final table (TRUE/FALSE).
3. `Deprecated` indicating whether the tool is deprecated (TRUE/FALSE).

Example of the `tools_status.tsv` file:
```
Suite ID	To keep	Deprecated
abacas	TRUE	FALSE
abricate	TRUE	FALSE
abritamr	TRUE	FALSE
```

To generate this file, we recommend that you use the generated file `tools_status.tsv` and to simply update the last two columns.

> <hands-on-title>Review tools in your community table</hands-on-title>
>
> 1. Download the `tools.tsv` file.
> 2. Open `tools.tsv` with a Spreadsheet Software.
> 3. Review each line corresponding to a tool.
>
>    You can also just review some tools. 
> 1. Add `TRUE` to the `To keep` column if the tool should be kept, and `FALSE` if not.
> 2. Add `TRUE` or `FALSE` also to the `Deprecated` column.
> 3. Export the new table as TSV.
> 4. Submit the TSV as `tools_status.tsv` in your community folder.
> 5. Wait for the Pull Request to be merged
>
{: .hands_on}

On the Sunday following the merging of the previous pull request, a `curated_tools.tsv` file will be generated in `communities/<your community>/resources/` folder reflecting the Galaxy tool landscape for your community. You or other community members can review tools in your community from this page to make step-by-step changes or updates to the `tools_status.tsv` file as needed.

We recommend collaborative work be done with a cloud-based system such as a Google Sheets page.

Here is an overview of the files (the top three files in the table are the most important):

| Filename | Location | Generation | Function | Format | Example (microgalaxy) |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| categories | communities/<your_community>/metadata/ | Manual | Name of the categories relevant to your community, with 1 ToolShed category per row | NA | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/metadata/categories) |
| tool_status.tsv | communities/<your community>/metadata/ | Automatic (to update manually) | Table with all tool suites and a short description | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/metadata/tool_status.tsv) |
| curated_tools.tsv | communities/<your_community>/metadata/ | Automatic | Curated table with all tool suites and a short description | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/curated_tools.tsv) |
| curated_tools_w_biotools.tsv | communities/<your_community>/resources/ | Automatic | Curated table with all tool suites that have a BioTools ID | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/curated_tools_w_biotools.tsv) |
| curated_tools_wo_biotools.tsv | communities/<your_community>/resources/ | Automatic | Curated table with all tool suites without a BioTools ID (You can add them to BioTools or link them!!) | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/curated_tools_wo_biotools.tsv) |
| tools.html | communities/<your_community>/resources/ | Automatic | A list of the tools in HTML format to include in a website | HTML | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/tools.html) |
| tools_wordcloud.png | communities/<your_community>/resources/ | Automatic | A wordcloud with the tools used in the tool suites | PNG | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/tools_wordcloud.png) |

![Flowchart illustrates the steps to generate the tool’s resources for your community.](./images/codex_tools_flowchart.png "Flowchart illustrates the steps to generate the tool’s resources for your community.")

# Embed the interactive table in your community page on the Hub

The interactive table you have created can be embedded in your community page on the Hub, e.g. [microGalaxy](https://galaxyproject.org/community/sig/microbial/#tools).

> <hands-on-title>Embed your table as an iframe</hands-on-title>
>
> 1.  If not already done, fork the repository [Galaxy Hub](https://github.com/galaxyproject/galaxy-hub)
> 2. Open or create your community page: `content/community/sig/<your community>/index.md`
> 3. Add an iframe to embed the interactive table
>
>    ```
>    <iframe
>      id="inlineFrameExample"
>      title="Microbial related tools"
>      width="100%"
>      height="600"
>      frameBorder="0"
>      src="https://galaxyproject.github.io/galaxy_codex/communities/<your_community>/resources/">
>    </iframe>
>    ```
>
> 4. Replace `<your_community>` by the name of your community in `src`
> 5. Submit the changes
> 7. Wait for the Pull Request to be merged
>
{: .hands_on}

# Conclusion

You now have curated resources listing Galaxy tools available for your community, and this table can be embedded in a community page.


topic: community
tutorial: community_content
title: Creating community content
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: []
content_hash: d02a7ca543a3ce31516a329a1209b534
last_modified: 2026-05-15T22:28:10.715240
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/community_content/tutorial.html

content:
Galaxy *[Special Interest Group](https://galaxyproject.org/community/sig)* (**SIG**)s work hard to build and maintain training resources. The GTN has worked hard to acknowledge this and offer nice impact pages to communities.

Here is a list of resources that you can use!

> <comment-title></comment-title>
> - We want this material to grow (similar to the 'Creating content in markdown' tutorial) so please do add further resources that can help communities!
{: .comment}

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Maintainer Home

Maintaining resources is **fundamental** to the quality and usefulness of any software resource. We do *not* throw code over the wall!

To help topic maintainers to quickly recognise what materials need updating and fixing, topic **Maintainer Homes** were built.

> <hands-on-title>Go to your topic Maintainer Home</hands-on-title>
>
> 1. Go to any GTN training topic of interest.
> 2. Scroll down past the tutorials, and click on the "Maintainer Home" button
>
>    {% snippet topics/community/faqs/maintainer_home.md %}
>
> 3. Explore the Maintainer Home!
>    - For example, the [Single Cell Maintainer Home]({% link topics/single-cell/maintainer.md %})
>
{: .hands_on}

You may instantly see some key information missing from tutorials, or how long its been since someone checked it! Time to update some materials!

You can see an example from the Single-cell topic below:

<iframe
    src="/training-material/topics/single-cell/maintainer.html"
    width="100%"
    height="400"
    style="border: 2px solid #00008B;"
    title="Single-Cell Topic Maintainer Home">
</iframe>

# Community Home

Where the **Maintainer Home** helps you sustain your community, the **Community Home** helps you show off your community. An end-of-year gift in 2024, this page will sift through news, events, and GTN contributions for your community tag of interest (example: single-cell) and provide a beautiful visualization of your efforts.

> <hands-on-title>Go to your topic Community Home</hands-on-title>
>
> 1. Go to any GTN training topic of interest.
> 2. Scroll down past the tutorials, and click on the "Community Home" button
>
>    {% snippet topics/community/faqs/community_home.md %}
>
> 3. Explore the **Community Home**!
>    - For example, the [Single Cell Community Home]({% link topics/single-cell/community.md %})
>
{: .hands_on}

You can see an example from the Single-cell topic below:

<iframe
    src="/training-material/topics/single-cell/community.html"
    width="100%"
    height="400"
    style="border: 2px solid #00008B;"
    title="Single-Cell Topic Community Home">
</iframe>

# Topic usage statistics

Next up, you might want to know how many people are actually using your materials? Welcome to your **Topic usage statistics**! You may have already found this, actually, as it's (currently) at the bottom of the Maintainer Home.

> <hands-on-title>Go to your topic usage statistics</hands-on-title>
>
> 1. Go to any GTN training topic of interest.
> 2. Scroll down past the tutorials, and click on the "Maintainer Home" button
>
>    {% snippet topics/community/faqs/maintainer_home.md %}
>
> 3. Scroll down to the section "Statistics For Your Materials"
> 4. Explore the usage statistics!
>    - For example, the [Single Cell Usage Statistics]({% link topics/single-cell/maintainer.md %}#statistics-for-your-materials)
>
{: .hands_on}

You can see an example from the Single-cell topic below.

<iframe
    src="/training-material/topics/single-cell/maintainer.html#statistics-for-your-materials"
    width="100%"
    height="400"
    style="border: 2px solid #00008B;"
    title="Single-Cell Topic Usage Statistics">
</iframe>

# News widgets

You can also embed news into your pages, subdomains/ Galaxy Labs, or even your Matrix channels.

Follow this documentation to learn how:

1. [GTN Feeds]( {% link feeds/index.md %} )
2. Bot integration into matrix

{% snippet topics/community/faqs/matrix_news_bot.md %}

You can see an example from the Single-cell topic below.

<h3 class="mb-3">News and Events</h3>
<iframe width="100%" height="300px" src="/training-material/feeds/single-cell-month.w.html"></iframe>

# Workflow search

Want to see all the workflows tagged with your community tag across public servers? Look no further!

Follow this documentation to learn how:

1. [Galaxy Pan-Galactic Workflow Search]( {% link workflows/list.md %} )
2. [Workflow Search Querying]( {% link news/2023/11/20/workflow-search.md %} )

You can see an example from the Single-cell topic below.

<h3 class="mb-3">Public workflows</h3>
<iframe src="/training-material/workflows/embed.html?query=single-cell" height="400px" width="100%" class="gtn-embed" frameborder="1"></iframe>

# Galaxy Community Activities calendar

We host a GoogleCalendar on the [Galaxy Special Interest Group Hub page](https://galaxyproject.org/community/sig/).

{% snippet topics/community/faqs/community_activites_calendar.md %}

# Conclusion

{% icon congratulations %} Congratulations! You've made it to the end! Hopefully you think these resources are brilliant, and are making sure to tag everything (news, events, training materials, workflows, FAQs, you name it, you should tag it!) with your community tag!


topic: community
tutorial: sig_define
title: What's a Special Interest Group?
difficulty: intermediate
time_estimation: 15m
tools_json: []
requirements_json: []
tags_json: []
content_hash: f4ad4b1e76dcbab7547758037e460914
last_modified: 2026-05-15T22:28:10.715838
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/sig_define/tutorial.html

content:
In Galaxy, the term *[Special Interest Group](https://galaxyproject.org/community/sig)* (**SIG**) refers to a dedicated scientific community that crosses individual lab boundaries and wants to collaborate, share resources, support each other, and/or collectively advocate on a given theme. We have **SIGs** based on [**region**](https://galaxyproject.org/community/sig/#regional-communities), [**domain of science**](https://galaxyproject.org/community/sig/#communities-of-practice), and more. You might consider that a **SIG** covers any group of like-minded Galaxy enthusiasts not currently combined into a [**Working Group**](https://galaxyproject.org/community/wg/).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

<div class='right'><img src="../../images/mind_map.svg" alt="Person looking at a diagram with a central rectangle connected to many other nodes representing people and connections" width="25" /></div>

You can find a directory of current [**SIGs** below](https://galaxyproject.org/community/sig/).

<iframe src="https://galaxyproject.org/community/sig/" height="400px" width="800px"></iframe>

## Characteristics of SIGs

**SIGs** vary hugely. Some **SIGs** meet and collaborate frequently, while others use their **SIG** to simply provide a key contact individual to represent their group in the Galaxy community. The following list, therefore, does not necessary apply to every SIG, but can give you a general idea of what **SIGs** can get up to.

**SIGs** can be:
* Focused on user community development
* Self-regulated, which includes organising meetings, developing training material and tools, maintaining the Hub and training pages, and choosing a representative for the [Galaxy Community Board](https://galaxyproject.org/community/governance/gcb/)
* Long-term or transient
* Supported with administrative infrastructure provided by the Galaxy Project and the [GCB](https://galaxyproject.org/community/governance/gcb/), including:
  * mailing list
  * website infrastructure and a place on the Galaxy hub to present the Community
  * presentation slot at the annual GCC conference
  * Chat channel/room
* Set up to gather input from members
* Represented in the [Galaxy Community Board](https://galaxyproject.org/community/governance/gcb/) which articulates needs to the Galaxy community to influence the next innovations (known to software developers as a Roadmap process: the selection of priorities for the coming years)

# Purpose of a SIG

Taking into account the wide variety of **SIGs**, the following are often priorities for **SIGs**.

## Prevent work duplication

It’s surprisingly easy for individuals halfway across the world to need the same tool and build the same tool.
What a waste of time when there are so many cool things to do!

## Highlight new resources

If you’re working in the same research area, for instance, it can be exciting when someone builds something new that could help your research. With so many things getting added daily, it’s easy to miss this news. SIGs can point out such highlights and give you new ideas for your research or work.

## Peer review

Need someone to look at an issue? Want some feedback on a tutorial or tool? Your fellow SIG members are a lot more likely to help you if you help them, and, more importantly, if they know who you are. Cut through the 1.8k of issues on Github by making some work friends in your field.

## Build community

The bigger the community you have, the more power in developing further resources (or grant proposals!), finding collaborators, co-authoring papers, etc. Higher user counts give you more sway in decision-making and building up Galaxy further, advancing your own work. It’s a lot easier to say you need something and 100 users agree, than to ask on behalf of yourself alone.

## Many hands make light work

Users want tools, workflows, training, and events. When delivering training (like [Smorgasbord](https://gallantries.github.io/posts/2021/03/01/smörgåsbord/) global training events) or projects, explaining who you are to a grant board, or recruiting individuals to your SIG, it’s rather important that the website you point to, quite frankly, looks nice. Building tools, preparing training material, delivering events, and updating web pages takes time. If you can share these tasks amongst a **SIG**, you ensure sustainability and increase impact.

# How do Working Groups & Special Interest Groups differ?

[Working Groups **(WGs)**](https://galaxyproject.org/community/wg/) are distinct from **SIGs**.

**WGs** are:

* Focused on technical issues
* Managing important project infrastructure assets, including source code, for the long term
* Represented in the [Galaxy Technical Board](https://galaxyproject.org/community/governance/gtb/), which contributes to the roadmap (prioritisation of innovation goals in Galaxy) by providing technical input and requesting resources for technical and development activities

You can find a directory of current [**WGs** below](https://galaxyproject.org/community/wg/).

<iframe src="https://galaxyproject.org/community/wg/" height="400px" width="800px"></iframe>

# Representation in Galaxy Governance

SIGs send (at least one) representative to the [Galaxy Community Board (GCB)](https://galaxyproject.org/community/governance/gcb/), which meets throughout the year to share useful tips and practices, as well as collects ideas for promoting via the Galaxy Governance structure to enact global change across numerous groups of individuals. Global software development is tricky, and effective communication and representation is key for making things happen to benefit our SIGs.

{% snippet topics/community/faqs/governance_structure.md %}

# How to join a SIG

You can find out how to join any given SIG by going to their SIG page, which you can peruse from the [SIG directory](https://galaxyproject.org/community/sig/).

> <comment-title></comment-title>
> Being part of a **SIG** is awesome! You find people from different institutes, disciplines or regions who are all excited about the same thing. It's one of the best bits of the Galaxy Community more broadly - it's a welcoming and energetic bunch!
{: .comment}

If you do not find a SIG that represents your interest, you may want to make one!

# Conclusion

{% icon congratulations %} Congratulations! You've made it to the end! Hopefully you think **SIGs** are brilliant and maybe you have joined one, or want to start one!


topic: community
tutorial: community-lab
title: Creation of the labs in the different Galaxy instances for your community
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["Community", "SIG", "CoDex"]
content_hash: eeaac9f3246c805deb6e5417ab1fd964
last_modified: 2026-05-27T09:01:47.879593
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/community-lab/tutorial.html

content:
The **Community lab**, a centralised webpage that enables communities to rapidly aggregate, curate, integrate, display, and launch relevant tools, workflows, and training on different Galaxy servers. This user-friendly interface, built on the Galaxy framework, provides community members with data analysis capacity without requiring programming expertise. Users can run individual tools or create complex workflows, with full provenance tracking to ensure reproducibility, designed specifically for the community research (Nasr et al., 2024).
  *For example, [the microgalaxy lab (Europe)](https://microbiology.usegalaxy.eu).*


The aim of this tutorial is to create the files necessary to display the labs in each Galaxy instance.

You can also use the [Galaxy Labs engine](https://labs.usegalaxy.org.au).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Create the lab specific files (automatic)

The labs are composed of various files, some that are specific to your community and some that are common between all the labs.
To automatically create the necessary files from a set of templates, a script is provided (`sources/bin/populate_labs.sh`).

This script will create the following structure and files:
```
communities/
└── <your community>/
    └── lab/
        ├── CONTRIBUTORS
        ├── README.md
        ├── base.yml
        ├── conclusion.html
        ├── intro.html
        ├── usegalaxy.eu.yml
        ├── usegalaxy.fr.yml
        ├── usegalaxy.org.yml
        └── sections_templates/
            ├── 1_data_import_and_preparation.yml
            ├── 2_tools.yml
            ├── 3_workflows.yml
            ├── 4_tutorials.yml
            ├── 5_support_and_help.yml
            └── 6_community.yml
```


The files are created and updated automatically using GitHub actions, you just need to add a bit of code to point to your community folder.

> <hands-on-title>Generate the files using GitHub Actions</hands-on-title>
>
> 1. Go to the [Galaxy Codex repo](https://github.com/galaxyproject/galaxy_codex)
> 2. Go to the file `.github/workflows/fetch_filter_resources.yaml`
> 3. On the right, click on the pen to "Edit this file"
> 4. Scroll down
> 5. Duplicate the section called "Populate <community-name> Lab"
>
>    ```
>      - name: Populate <community-name> Lab
>        run: |
>          bash sources/bin/populate_labs.sh
>        env: 
>          COMMUNITY: <community-name>
>    ```
> 6. Replace `<community-name>` by the name of your community
> 7. Commit changes the changes to a new branch that you name "new lab for [Community-name]"
>
{: .hands_on}

After that, your commit will be reviewed (and approved) by admins.
On the following Sunday (or upon request), this action will be launched and will create the new files in the appropriate community folder (`communities/<your community>/lab/`).

# Modify the generated files to personalize your community lab

Once the files are created, you should update them as some contain template text that are not community specific. 

Files to update :

| File | Action |
|------|--------|
| `communities/<your community>/lab/CONTRIBUTORS` | Add the handles of all contributors |
| `communities/<your community>/lab/README.md` | Replace all `<your-community>` with your community name |
| `communities/<your community>/lab/base.yml` | Replace all `<your-community>` with your community name |
| `communities/<your community>/lab/intro.html` | Add a description of your community |

The following files require no changes:

| File | Note |
|------|------|
| `communities/<your community>/lab/conclusion.html` | Pre-filled, no edit needed |
| `communities/<your community>/lab/sections_templates/*` | Pre-filled, no edit needed |

The files in the section folders contain the code for each table visible in the lab.
You can check different labs for inspiration, such as the [microgalaxy lab](https://microbiology.usegalaxy.eu).

# Create community specific sections to personalize your community lab

If you want additional sections, for example, the "Microbial isolates" and "Microbiome" sections in the [microgalaxy lab](https://microbiology.usegalaxy.eu/).

> <hands-on-title>Add community specific sections</hands-on-title>
> 1. Find an appropriate template. You can use other files in your section folder, or browse other labs, such as [microgalaxy lab code](https://github.com/galaxyproject/galaxy_codex/tree/main/communities/microgalaxy/lab), and copy the raw code as a template.
> 2. Create the file in your section folder. Name it with a digit (numerical order) and a descriptive name (i.e. "7_microbial_isolate").
> 3. Copy the raw code in this newly created file.
> 4. Update the code to display what you want.
> 5. Save the file.
> 6. Open `communities/<your community>/lab/base.yml`
> 7. Add the previously created file to the sections (see [microgalaxy base file](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/biodiversity/lab/base.yml) for example)
> 8. Save `communities/<your community>/lab/base.yml`
> 9. Pre-visualize the rendering of the community lab using the following path (make sure to change the variables indicated with <> in the path) : `https://labs.usegalaxy.org.au/?content_root=https://<your-repo>/galaxy_codex/blob/<your-branch>/communities/<community-name>/lab/base.yml`
> 10. Update the files until you obtain the lab you want.
> 11. Commit the changes, create the pull request (if not done previously)
{: .hands_on}

The Pull Request will be reviewed. Make sure to respond to any feedback.

# Include the labs in the different instances

For the lab to be accessible from the different instances, you need to add files in each instance independently.

For the French instance (https://usegalaxy.fr), all the steps are described in [Issue 297](https://gitlab.com/ifb-elixirfr/usegalaxy-fr/infrastructure/-/issues/297).
You can use the [merge request made for the biodiversity lab](https://gitlab.com/ifb-elixirfr/usegalaxy-fr/infrastructure/-/merge_requests/1302) as a reference.

For the European instance (https://usegalaxy.eu), this tutorial will be updated later.
You can use the [pull request made for the biodiversity lab](https://github.com/usegalaxy-eu/infrastructure-playbook/pull/1629) as a reference.

For the American instance (https://usegalaxy.org), this tutorial will be updated later.
You can use the [pull request made for the biodiversity lab](https://github.com/galaxyproject/usegalaxy-playbook/pull/427) as a reference.

For the Australian instance (https://usegalaxy.org.au), this tutorial will be updated later.
You can use the [pull request made for the biodiversity lab](https://github.com/usegalaxy-au/infrastructure/issues/2786#issuecomment-3330995427) as a reference.

# Conclusion

You now have all the files necessary to display your lab in several instances and they should soon be available to your users.


topic: community
tutorial: community-tutorial-table
title: Creation of a Galaxy tutorial table for your community
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["Community", "SIG"]
content_hash: 0d874d32bb2949a0231f4391f4382c1d
last_modified: 2026-05-27T09:01:47.880422
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/community-tutorial-table/tutorial.html

content:
Similarly to the numerous tools available on Galaxy, the [Galaxy Training Network](https://training.galaxyproject.org/) includes numerous tutorials. This tutorial will take you through the steps to generate resources listing all the relevant tutorials and display the tutorial table on your community codex page.

The pipeline creates a table with all the tutorials. This table can be **filtered to only include tutorials that are relevant to a specific research community**.

The generated community-specific table can be used as is (e.g. downloadable tsv file) and/or embedded (e.g. into a Galaxy Hub page or Galaxy subdomain). 

The pipeline is **fully automated** and executed on a **weekly** basis. Any research community can apply the pipeline to create a table specific to their community.

The aim of this tutorial is to create a tutorial table for a community.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add your community to the Galaxy CoDex

You first need to check if your Community is in the [Galaxy CoDex](https://github.com/galaxyproject/galaxy_codex/tree/main/communities), a central resource for Galaxy communities.
If the community is already there, you can move to the next step of this tutorial.

If you community is not already included, follow this step :
> <hands-on-title>Add your community to the Galaxy CoDex</hands-on-title>
>
> You need to create a new folder in the data/community folder within Galaxy CoDex code source.
> 1. If not already done, fork the [Galaxy Codex repository](https://github.com/galaxyproject/galaxy_codex)
> 2. Go to the `communities` folder
> 3. Click on **Add file** in the drop-down menu at the top
> 4. Select **Create a new file**
> 5. Fill in the `Name of your file` field with:  name of your community + `/metadata/tutorial_tags`
>    This will create a folder for your community, a new folder for your community called metadata and a file called tutorial_tags.
> 6. Click on **Commit changes**
> 7. Fill in the commit message with something like `Add X community`
> 8. Click on `Create a new branch for this commit and start a pull request`
> 9. Create the pull request by following the instructions
>
{: .hands_on}

# Pull list of tags relevant to your community

To add tutorials in your community tutorial table, you will need to indicate a list of tags relevant to your community, and tutorials associated with this tag will be automatically pulled from the [Galaxy Training Network](https://training.galaxyproject.org/) one a week. Only tutorials with the selected tags will be added to the filtered table.
You will then be able to remove tutorials that are not relevant to your community or deprecated.

> <hands-on-title>Select tutorials tags from the Galaxy Training Network</hands-on-title>
>
> 1. Go to the [Galaxy Training Network]({% link index.md %})
> 2. Browse for tutorials that are relevant to your community
> 3. Note the tags that are associated with tutorials of interest
{: .hands_on}
> <comment-title>Tips</comment-title>
> You can also create a tag (for example, your community name) and tag all relevant tutorials with it. Then, you only need to use this tag to pull all the relevant tutorials.
{: .comment}

# Add the list relevant tags for your community in the `tutorial_tags` file

> <hands-on-title>Add the relevant tags to the tutorial_tags file</hands-on-title>
> 1. Open or create a file named `tutorial_tags` in your community metadata folder (`communities/<your community>/metadata/tutorial_tags`)
> 2. Add the name of the tags relevant to your community in the `tutorial_tags` file, with 1 tag per row
> (See above for how to identify the appropriate tags)
>
>    For example:
>    ```
>    Assembly
>    Metagenomics
>    ```
{: .hands_on}

Once you have a list of the tags that you wish to keep, you can submit this to Galaxy CoDex.

> <hands-on-title>Submit the new list of tags to Galaxy CoDex</hands-on-title>
>
> 1. Click on **Commit changes** at the top
> 2. Fill in the commit message with something like `Add tutorials tags for my community`
> 3. Click on `Create a new branch for this commit and start a pull request`
> 4. Create the pull request by following the instructions
{: .hands_on}

The Pull Request will be reviewed. Make sure to respond to any feedback.

On the sunday following the Pull Request merge, a table with all tutorial will be created in `communities/<your community>/resources/tutorials.tsv`

Here is an overview of the files :

| Filename | Location | Generation | Function | Format | Example (microgalaxy) |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| tutorial_tags | communities/<your_community>/metadata/ | Manual | Name of the tags relevant to your community, with 1 tag per row | NA | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/metadata/tutorial_tags) |
| tutorials.tsv | communities/<your community>/resources/ | Automatic | Table with all the tutorials relevant to your community | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/metadata/tutorials.tsv) |
| tutorials.html | communities/<your_community>/resources/ | Automatic | A list of the tutorials in html format to include in a website | HTML | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/tutorials.html) |

![Flowchart illustrating the steps to generate the tutorials resources for your community.](./images/codex_tuto_flowchart.png "Flowchart illustrating the steps to generate the tutorials resources for your community.")


# Embed the table in your community page on the Hub

The table you have created can be embedded in your community page on the Hub, e.g. [microGalaxy](https://galaxyproject.org/community/sig/microbial/#workflows-and-tutorials).

> <hands-on-title>Embed your table</hands-on-title>
>
> 1.  If not already done, fork the repository [Galaxy Hub](https://github.com/galaxyproject/galaxy-hub)
> 2. Open or create your community page: `content/community/sig/<your community>/index.md`
> 3. TO DO : We Will soon update this section to explain how to include the table, sorry for the inconvenience. 
> 4. Submit the changes
> 5. Wait for the Pull Request to be merged
{: .hands_on}

# Conclusion

You now have a table with tutorials available for your community, and this table is embedded in a community page.


topic: community
tutorial: tools_subdomains
title: Make your tools available on your subdomain
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["subdomain", "earth-system", "interactive-tools"]
content_hash: 1b657263996de202e31fdf46b2130ebf
last_modified: 2026-05-15T22:28:10.716109
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/tools_subdomains/tutorial.html

content:
This tutorial explains how to make your brand new tools, once they're published through a Pull Request ([check this tutorial]({%link topics/dev/tutorials/tool-from-scratch/tutorial.md %}) on how to build your tool from scracth), visible on your subdomain. Here we'll follow the example on how to make the tools visible on [Galaxy for earth System](https://earth-system.usegalaxy.eu/). You can also find the explanations on how to add your tools on this [Github page](https://usegalaxy-eu.github.io/operations/subdomains.html).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add your batch tool to the right section

> <hands-on-title>Tool sections (for batch tools) </hands-on-title>
>    - Clone the Github repository of the Galaxy server (of where your subdomain to be attached)
>    - For Galaxy Europe, fork the [usegalaxy-eu-tools repo](https://github.com/usegalaxy-eu/usegalaxy-eu-tools)
>    - Create a branch on your fork
>    - Browse through the different yaml files and select the one that correspond to your subdomain for earth system everything is made under the ecology owner.  The name of the files corresponds to a toolshed owner or community and a few of those communities with a review system and CI integration gets special trust and auto-updates.
>    - Once in the yaml you can add the section for your tool (you want to know what are the existing tool sections ? Go check the [categories defined here](https://github.com/usegalaxy-eu/infrastructure-playbook/blob/master/templates/galaxy/config/tool_conf.xml.j2))
>
> An example on how to fill in the yaml file
> ```
> - name: id_of_your_tool
>   owner: choose_the_owner_relative_to_a_github_repo
>   tool_panel_section_label: 'Choose the section where your tool belongs'
> ```
>
> ![Image of the modification to make in the ecology.yaml file](../../images/tool_subdomain/ecology_yml_tool.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

This part is only to make batch tool visible in your subdomain.

# Add a section of tools to your subdomain
> <hands-on-title>Sections for your subdomain</hands-on-title>
>    - If not already done clone the Galaxy Europe [Infrastructure-playbook repo](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master)
>    - Create a branch on your fork
>    - Go to the file **global_host_filters.py.j2** in **templates/galaxy/config/**
>    - Open it and edit it the "DOMAIN_SECTIONS" part go to your subdomain line and in the list of section add the one you need for instance `"climate_analysis"`
>
> ![Image of the modification to make in the global_host_filters.py.j2 file](../../images/tool_subdomain/add_section.png)
>
>   - Then commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}


# Add you interactive tool to the right section
> <hands-on-title>Interactive tool sections</hands-on-title>
>    - If not already done clone the Galaxy Europe [Infrastructure-playbook repo](https://github.com/usegalaxy-eu/infrastructure-playbook/tree/master)
>    - Create a branch on your fork
>    - Go to the file **tool_conf.xml.j2** in **templates/galaxy/config/**
>    - Open it and edit it by adding the xml file of the interactive tool to the section interactive tool or any other interactive tool section (to know more on how to create your interactive tool section folow [this tutorial]({%link topics/admin/tutorials/subdomain/tutorial.md %}))
>    - Below an example of how to add the interactive tool panoply to the earth system subdomain.
>
> ![Image of the modification to make in the tool_conf.xml.j2 file](../../images/tool_subdomain/add_interactive_tool.png)
>
>   - Then, commit your changes and write a nice message for the admin when you open your Pull Request.
{: .hands_on}

If you choose to create a new section for your interactive tool don't forget add this section to your subdomain !

# Conclusion

You can now start to build your workflow on your subdomain !


topic: community
tutorial: sig_create
title: Creating a Special Interest Group
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: []
content_hash: 1f1c45a2a50688d0b9bf21e2a11e3d0f
last_modified: 2026-05-15T22:28:10.715531
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/sig_create/tutorial.html

content:
So you have decided to create a new Special Interest Group (**SIG**)!

# Do you need to make a SIG?

Before you make a SIG, make sure that you have learned the basics about Galaxy. Start with our dedicated [**Get Started**](https://galaxyproject.org/get-started/) page to get a head-start of what Galaxy is about.

> <comment-title></comment-title>
> You'll find all the necessary resources and you could even participate in the [**Galaxy Mentorship Network**](https://galaxy-mentor-network.netlify.app/) to get started with the help of a Galaxy Mentor.
{: .comment}

Starting and running a **SIG** takes time and effort, so it's a good idea to make sure that:

 1. **You have a buddy** Do you already have someone who wants to join this community and help build it with you? There's no sense in starting a **SIG** of one!
 2. **You are committed** We have resources to help you and are constantly trying to make running a **SIG** easier, but it nevertheless takes time. You don't want to start a **SIG** and then fail to maintain it.

 > <warning-title>Do not contribute to software smog!</warning-title>
 > Do not create a SIG that you cannot sustain! Do not create a SIG that already exists!
 {: .warning}

Saying that, running a **SIG** can be deeply rewarding when you see people coming together across disciplines, institutes, and more to find synergy and make cool things happen together.

Let's walk you through making a new Galaxy **SIG**.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Requirements of a Special Interest Group (SIG)

Before starting a **SIG**, it's important to be aware of the (rather low) requirements for being a **SIG**:

* Must have 2 or more members
* Must maintain a page on the Community Hub with at least:
  * A description of the purpose of the SIG.
  * A clear explanation of how individuals can join and maintain membership.
  * Details of access to the SIG’s communication channels.
  * The current contact/email address for the nominated [Galaxy Community Board](https://galaxyproject.org/community/governance/gcb/) representative
* Must manage membership and internal communication for itself.
* Must use at least one public Matrix channel under the Galaxy Matrix space.
* Must allow participation across different timezones (with the exception of **Regional Communities**).

{% snippet topics/community/faqs/meeting_times.md %}

{% snippet topics/community/faqs/governance_structure.md %}

# Creating a new SIG

> <hands-on-title>Create a SIG</hands-on-title>
>
> 1. **Copy an updated SIG page** (there's a Template in that folder, work in progress!) [from the Galaxy Community Hub github folder](https://github.com/galaxyproject/galaxy-hub/blob/master/content/community/sig/). We suggest looking through the different SIG pages to see which one has the type of content you would like and copying that one.
> 2. **Update the text** to suit your new SIG
> 3. **Create a Matrix group** within the Galaxy Matrix chat forum space.
>   - Consider making a bridge to a Slack chat if you want to talk with users!
> 4. (Optional): **Create a mailing list** for your SIG: https://lists.galaxyproject.org/lists/
> 5. **Add your SIG** & any relevant links to the [SIG directory](https://github.com/galaxyproject/galaxy-hub/blob/master/content/community/sig/index.md)
> 6. Send someone to **join the Galaxy Community Board**, which has tons of resources to support you and also a chat forum on Matrix.
>
>    {% snippet topics/community/faqs/governance_gcb.md %}
>
>    {% snippet topics/community/faqs/governance_gcb_join.md %}
>
{: .hands_on}

Remember that **SIGs** vary hugely. Some **SIGs** meet and collaborate frequently, while others use their **SIG** to simply provide a key contact individual to represent their group in the Galaxy community. You don't need every link, widget or calendar we create to run a good **SIG**. Just make what fits your group's needs. Above all, think sustainably - someone has to keep things updated, and that someone likely starts with you!

# Sustainability

Remember that if you make a **SIG**, you are responsible for *maintaining* that **SIG**.

This at a minimum includes:

1. A section on the [SIG Directory on the Hub](https://galaxyproject.org/community/sig/)
2. Your individual SIG page from the Hub that is linked from that directory

This may also include:
1. Your SIG Training page
2. Mailing list

## Ending a SIG

If you collectively decide that the **SIG** is no more, please end your SIG.

> <hands-on-title>End a SIG</hands-on-title>
>
> 1. Update the [SIG Directory on the Hub](https://galaxyproject.org/community/sig/)
> 2. Remove your individual SIG page from the Hub
> 3. Notify the [Galaxy Community Board](https://galaxyproject.org/community/governance/gcb/), ideally via their chat channel
>
{: .hands_on}

# Conclusion

Did you make a SIG? {% icon congratulations %} Congratulations! Being a part of a **SIG** is one of the best bits of working with Galaxy, so well done you for enabling that for like-minded scientists!

Or did you realise that you didn't need to make a **SIG**? {% icon congratulations %} Congratulations to you as well, for preventing the software smog of creating unneeded webpages.


topic: community
tutorial: community-workflow-table
title: Creation of resources listing Galaxy workflow for your community
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: []
tags_json: ["Community", "SIG"]
content_hash: 2f71d26fe9b642f38211d789c8ee0ceb
last_modified: 2026-05-27T09:01:47.880663
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/community/tutorials/community-workflow-table/tutorial.html

content:
Similarly to the numerous tools available on Galaxy, numerous workflow are available on public Galaxy instances and in [Workflow Hub](https://workflowhub.eu/). This tutorail will take you through the steps to generate resources listing the relevant workflows and how to display them on your community codex page.

The pipeline creates a table with all the workflows associated to selected tags. This table can be **filtered to only include workflows that are relevant to a specific research community**.

The generated community-specific table can be used as it and/or embedded, e.g. into the respective Galaxy Hub page or Galaxy subdomain.

The pipeline is **fully automated** and executed on a **weekly** basis. Any research community can apply the pipeline to create a table specific to their community.

The aim of this tutorial is to create a workflow table for a community.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Add your community to the Galaxy CoDex

You first need to check if your Community is in the [Galaxy CoDex](https://github.com/galaxyproject/galaxy_codex/tree/main/communities), a central resource for Galaxy communities.
If the community is already there, you can move to the next step of this tutorial.

If you community is not already included, follow this step :
> <hands-on-title>Add your community to the Galaxy CoDex</hands-on-title>
>
> You need to create a new folder in the data/community folder within Galaxy CoDex code source.
> 1. If not already done, fork the [Galaxy Codex repository](https://github.com/galaxyproject/galaxy_codex)
> 2. Go to the `communities` folder
> 3. Click on **Add file** in the drop-down menu at the top
> 4. Select **Create a new file**
> 5. Fill in the `Name of your file` field with:  name of your community + `/metadata/workflow_tags`
>    This will create a folder for your community, a new folder for your community called metadata and a file called workflow_tags.
> 6. Click on **Commit changes**
> 7. Fill in the commit message with something like `Add X community`
> 8. Click on `Create a new branch for this commit and start a pull request`
> 9. Create the pull request by following the instructions
>
{: .hands_on}

# Pull list of tags relevant to your community

To add workflows in your community workflow table, you will need to indicate a list of tags relevant to your community, and workflows associated with these tags will be automatically pulled from public Galaxy instances and [Workflow Hub](https://workflowhub.eu/). Only workflows with the selected tags will be added to the table.
You will then be able to remove workflows that are not relevant to your community or deprecated.

> <hands-on-title>Select workflows tags from the Galaxy instances</hands-on-title>
>
> 1. Go to your favorite public Galaxy instance.
> 2. Go to the `Workflow` section
> 3. Select `Public workflows` (for example: [Public workflows on the French instance](https://usegalaxy.fr/workflows/list_published))
> 4. Browse for workflows that are relevant to your community
> 5. Note the tags that are associated with workflows of interest
{: .hands_on}

> <hands-on-title>Select workflows tags from the Workflow Hub</hands-on-title>
>
> 1. Go to [Workflow Hub](https://workflowhub.eu/)
> 2. Browse for workflows that are relevant to your community
> 3. Note the tags that are associated with workflows of interest (in the `tag` section on the left of the workflow page)
{: .hands_on}

# Add the list relevant tags for your community in the workflow_tags file

> <hands-on-title>Add the relevant tags to the workflow_tags file</hands-on-title>
> 1. Open or create a file named `workflow_tags` in your comunity metadata folder (`communities/<your community>/metadata/workflow_tags`)
> 2. Add the name of the tags relevant to your community in the `workflow_tags` file you started above. The file is split in two sections : `Public`, which should inidcate the tags used on public Galaxy instances; and `workflowhub`, which indicates the tags to use to select workflow on [Workflow Hub](https://workflowhub.eu/).
>
>    For example:
>    ```
>    public:
>    - name:microgalaxy
>    - microgalaxy
>    workflowhub:
>    - abromics
>    - amr
>    - amplicon
>    ```
>
{: .hands_on}

It is currently mandatory to have tags under both sections (public and workflowhub) or the codex script will crash and you will not get the files mentionned below.
Once you have a list of the tags that you wish to keep, you can submit this to Galaxy CoDex.

> <hands-on-title>Submit the new list of tags to Galaxy Codex</hands-on-title>
>
> 1. Click on **Commit changes** at the top
> 2. Fill in the commit message with something like `Add workflows tags for my community`
> 3. Click on `Create a new branch for this commit and start a pull request`
> 4. Create the pull request by following the instructions
>
{: .hands_on}

The Pull Request will be reviewed. Make sure to respond to any feedback.

On the sunday following merging of the pull request, a few files will be created :
- `communities/<your community>/metadata/workflow_status.tsv` : A table with all the workflows extracted with the tags mentionned above.
- `communities/<your community>/resources/curated_workflows.tsv` : A table with the same workflows as in the metadata file and additional information.
- `communities/<your community>/resources/curated_workflows.json` : A json file with the same info as the correponding table.
- `communities/<your community>/resources/tag_filtered_workflows.tsv` : At this stage, the same table as `curated_workflows.tsv`.
- `communities/<your community>/resources/tag_filtered_workflows.json` : At this stage, the same json as `curated_workflows.json`.


# Review the generated table to curate workflows

The generated table in the metadata folder (`communities/<your community>/metadata/workflow_status.tsv`) contains all the workflows associated with the tags that you selected. However, not all of these workflows might be interesting for your community.

Galaxy CoDex allows for an additional optional filter for workflows, that can be defined by the community curator (maybe that is you!).

To filter the workflows, you will update the values in the last two columns of the `communities/<your community>/metadata/workflow_status.tsv` file
- `To keep` indicating whether the tool should be included in the final table (TRUE/FALSE).
- `Deprecated` indicating whether the tool is deprecated (TRUE/FALSE).

You can modify the file directly in GitHub or download it to create a spreadsheet, easier to edit as a community.

> <hands-on-title>Modify workflows using spreadsheets</hands-on-title>
>
> 1. Download the `workflow_status.tsv` file in `communities/<your community>/metadata/`.
> 2. Open `workflow_status.tsv` with a Spreadsheet Software.
> 3. Review each line corresponding to a workflow.
>     1. Add `TRUE` to the `To keep` column if the tool should be kept, and `FALSE` if not.
>     2. Add `TRUE` or `FALSE` also to the `Deprecated` column.
> 5. Export the new table as TSV.
> 6. Submit the TSV as `workflow_status.tsv` in your `communities/<your community>/metadata/` folder.
> 7. Wait for the Pull Request to be merged
>
{: .hands_on}

On the sunday following merging of the pull request, the following files, reflecting the Galaxy tool landscape for your community, will be updated :
- `communities/<your community>/resources/curated_workflows.tsv`
- `communities/<your community>/resources/curated_workflows.json` 
- `communities/<your community>/resources/tag_filtered_workflows.tsv`
- `communities/<your community>/resources/tag_filtered_workflows.json`

You can step-by-step review all workflows in your community and update the `workflow_status.tsv` file. You could also share this file with your community members and discuss weather the workflow should be kept or not. Collaborative work could be established using google spreadsheet.


Here is an overview of the files (the top three files in the table are the most important):

| Filename | Location | Generation | Function | Format | Example (microgalaxy) |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| workflow_tags | communities/<your_community>/metadata/ | Manual | Name of the tags relevant to your community | NA | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/metadata/workflow_tags) |
| workflow_status.tsv | communities/<your community>/metadata/ | Automatic (To update manually) |  Table with all the workflows extracted with the tags mentioned above | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/metadata/tool_status.tsv) |
| curated_workflows.tsv | communities/<your_community>/resources/ | Automatic | Table containing only the curated workflows| TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/curated_workflows.tsv) |
| curated_workflows.json | communities/<your_community>/resources/ | Automatic | File containing only the curated workflows| JSON | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/curated_workflows.json) |
| tag_filtered_workflows.tsv | communities/<your_community>/resources/ | Automatic | Table with all the workflows extracted with the tags mentioned above | TSV | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/tag_filtered_workflows.tsv) |
| tag_filtered_workflows.json | communities/<your_community>/resources/ | Automatic | JSON file with all the workflows extracted with the tags mentioned above | JSON | [Example](https://github.com/galaxyproject/galaxy_codex/blob/main/communities/microgalaxy/resources/tag_filtered_workflows.json) |

![Flowchart illustrating the steps to generate the workflow resources for your community.](./images/codex_workflows_flowchart.png "Flowchart illustrating the steps to generate the workflow resources for your community.")


# Embed the table in your community page on the Hub

The table you have created can be embedded in your community page on the Hub, e.g. [microGalaxy](https://galaxyproject.org/community/sig/microbial/#workflows-and-tutorials).

> <hands-on-title>Embed your table</hands-on-title>
>
> 1.  If not already done, fork the repository [Galaxy Hub](https://github.com/galaxyproject/galaxy-hub)
> 2. Open or create your community page: `content/community/sig/<your community>/index.md`
> 3. TO DO : We Will soon update this section to explain how to include the table, sorry for the inconvenience. 
> 4. Submit the changes
> 5. Wait for the Pull Request to be merged
>
{: .hands_on}

# Conclusion

You now have a table with workflows available for your community, and this table is embedded in a community page.


