# tutorials

topic: fair
tutorial: data-management-peatlands
title: Introduction to Data Management Plan (DMP) for Peatland Research and PeatDataHub
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["fair", "open", "data management plan"]
content_hash: 57e07cb8f998c4cb0cf2dd076fa8126e
last_modified: 2026-05-15T22:28:13.005350
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/data-management-peatlands/tutorial.html

content:
Peatland monitoring and research generates data that is used for management, conservation and policy related to these ecosystems. To ensure that your project data is collected systematically and can be analysed, shared, and re-used it is important to have a data management plan.

This tutorial is a short introduction to writing a Data Management Plan (DMP) with examples for peatland monitoring projects and initiatives.

You can find more information on DMPs in the [Research Data Management Toolkit for Life Sciences]( https://rdmkit.elixir-europe.org/)

> <agenda-title></agenda-title>
>
> In this tutorial we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# What is a Data Management Plan?

A Data Management Plan (DMP) is a live document that provides a framework for managing data material during and after the research project.

A data management plan will facilitate the process of collecting and managing your peatland data and will allow you to discuss with your team how to share data, considering the requirements of your organisation and funders.

# Key components of a DMP
DMP templates vary by institution and funding agencies. 

| DMP component  | What to include|
| ------------- | ------------- |
| Descriptions of the data  |    &bull; Project details <br> &bull; Types of data <br> collected and generated <br> &bull; Origin of data <br> &bull; Format and size of data |
|Data collection/generation | &bull; Data collection methods <br> &bull; Data quality standards| 
|Data management, documentation, and curation | &bull; Storage and accessibility <br> &bull; Metadata standards and documentation <br> &bull; Curation and preservation |  
|Data sharing and access | &bull; Where the data will be shared <br> &bull; When will the data be available |
|Data security (where relevant)|&bull; Risks to data security<br> &bull; GDPR considerations|
|Capabilities| &bull; Evaluation of institutional capabilities to preserve and manage data|
|Maintaining and implementing the Data Management Plan|&bull; Actions to maintain and deliver the DMP |
|Environmental considerations|&bull; Energy cost considerations related to data management|
|Responsibilities|&bull; Agreement on the responsibilities of named individuals for data management|
|Relevant institutional, departmental or study policies on data sharing and data security| &bull; Review of institutional policies for data management|

## Hands-on: Create a DMP for peatland research
* Download the DMP template for [Peatland Research and Monitoring](https://zenodo.org/records/11185774).
* Fill in the DMP template using the prompt questions.
* Review your DMP with your team and collaborators.

# FAIR data management solutions

After you have filled in the DMP template, review how you can make your DMP FAIR.

There are several ways to set up FAIR (Findable, Accessible, Interoperable, Reusable) data management plans (DMPs) 

* Findable (F): Data description and collection or reuse of existing data.  Using metadata and community recognised standards for curation that enable someone to find the data.
* Accessible (A): Standardised authentication or authorisation (e.g. HTTP, HTTPS) for open and restricted data access.
* Interoperable (I): Documentation and data quality, and community standards for data exchange (e.g. tab-delimited text).
* Reusable (R): Storage and backup supported by legal and ethical requirements, as well as data release policies.

More information is available in the [FAIR data management tutorial]({% link topics/fair/tutorials/data-management/tutorial.md %})

# Conclusion

Creating a DMP will allow you and your institution to make the most of your peatland data. This introduction to DMP aims to highlight the usefulness of data management for peatland research and monitoring.



##  Suggested reading
* [FAIR in a nutshell]({% link topics/fair/tutorials/fair-intro/tutorial.md %}) 
* [The published FAIR Guiding Principles: Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016)]({% cite Wilkinson2016 %})
* [Recipes for data FAIRification, written by domain experts giving real-world examples: FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html)
* [Documentation and frameworks for data FAIRification. Each of the 15 FAIR principles is put into context with real data examples: GO FAIR](https://www.go-fair.org/fair-principles)
* [FAIR walkthrough using examples from across all academic disciplines: How to FAIR](https://howtofair.dk/)

## Resources
* [How to write a data-sharing agreement between collaborators](https://www.youtube.com/watch?v=iaZInoaHa04)
* [DMP online](https://dmponline.dcc.ac.uk)
* [DSWizard](https://bio.tools/tool/Data_Stewardship_Wizard)
* [WaterLands](https://waterlands.eu/)
* [PeatDataHub](https://peatdatahub.net/)


topic: fair
tutorial: ro-crate-workflow-run-ro-crate
title: Workflow Run RO-Crate Introduction
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["ro-crate"]
content_hash: ad797cb2966ab84d379ea334ff0cd026
last_modified: 2026-05-15T22:28:13.197471
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-workflow-run-ro-crate/tutorial.html

content:
The [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) working group is developing a series of [RO-Crate profiles](https://www.researchobject.org/ro-crate/profiles.html) for representing the provenance of computational workflow executions. The set includes three profiles, which represent provenance with increasing level of detail:

* [Process Run Crate](https://w3id.org/ro/wfrun/process) can be used to describe the execution of one or more tools that contribute to a generic computation;
* [Workflow Run Crate](https://w3id.org/ro/wfrun/workflow) also describes computational processes, but in this case the execution is supposed to be orchestrated by a computational workflow (e.g., a Galaxy workflow);
* [Provenance Run Crate](https://w3id.org/ro/wfrun/provenance) is based on Workflow Run Crate, but provides additional specifications for describing the details of each step's execution.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Process Run Crate

Consider a simple computation where we select the first four lines of a text file, and then we select the last three lines out of the initial selection. Suppose we start with the following input file, `lines.txt`:

```
one
two
three
four
five
```

Then we run:

```
head --lines 4 lines.txt >sel1.txt
```

The `sel1.txt` file will then be:

```
one
two
three
four
```

Now we run:

```
tail --lines 3 sel1.txt >sel2.txt
```

Obtaining the following `sel2.txt`:

```
two
three
four
```

A Process Run Crate that describes the above computation will contain the three text files, plus an `ro-crate-metadata.json` file structured as follows:

```json
{
    "@context": "https://w3id.org/ro/crate/1.1/context",
    "@graph": [
        {
            "@id": "ro-crate-metadata.json",
            "@type": "CreativeWork",
            "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},
            "about": {"@id": "./"}
        },
        {
            "@id": "./",
            "@type": "Dataset",
            "conformsTo": {"@id": "https://w3id.org/ro/wfrun/process/0.1"},
            "hasPart": [
                {"@id": "lines.txt"},
                {"@id": "sel1.txt"},
                {"@id": "sel2.txt"}
            ],
            "mentions": [
                {"@id": "#runHead"},
                {"@id": "#runTail"}
            ]
        },
        {   "@id": "https://w3id.org/ro/wfrun/process/0.1",
            "@type": "CreativeWork",
            "name": "Process Run Crate",
            "version": "0.1"
        },
        {
            "@id": "https://www.gnu.org/software/coreutils/head",
            "@type": "SoftwareApplication",
            "name": "head",
            "softwareVersion": "8.30"
        },
        {
            "@id": "https://www.gnu.org/software/coreutils/tail",
            "@type": "SoftwareApplication",
            "name": "tail",
            "softwareVersion": "8.30"
        },
        {
            "@id": "#runHead",
            "@type": "CreateAction",
            "name": "Select first four lines",
            "description": "head --lines 4 lines.txt >sel1.txt",
            "endTime": "2023-04-27T15:02:56+02:00",
            "instrument": {"@id": "https://www.gnu.org/software/coreutils/head"},
            "object": {"@id": "lines.txt"},
            "result": {"@id": "sel1.txt"},
            "agent": {"@id": "https://orcid.org/0000-0002-1825-0097"}
        },
        {
            "@id": "#runTail",
            "@type": "CreateAction",
            "name": "Select first four lines",
            "description": "tail --lines 3 sel1.txt >sel2.txt",
            "endTime": "2023-04-27T15:04:02+02:00",
            "instrument": {"@id": "https://www.gnu.org/software/coreutils/tail"},
            "object": {"@id": "sel1.txt"},
            "result": {"@id": "sel2.txt"},
            "agent": {"@id": "https://orcid.org/0000-0002-1825-0097"}
        },
        {
            "@id": "lines.txt",
            "@type": "File"
        },
        {
            "@id": "sel1.txt",
            "@type": "File"
        },
        {
            "@id": "sel2.txt",
            "@type": "File"
        },
        {
            "@id": "https://orcid.org/0000-0002-1825-0097",
            "@type": "Person",
            "name": "Josiah Carberry"
        }
    ]
}
```

The RO-Crate describes two actions, `#runHead` and `#runTail`, which represent, respectively, the execution of the `head` and `tail` programs, referenced via the `instrument` property. Each action references its input(s) via `object` and its output(s) via `result`. Note how the two actions are connected by the fact that the output of `#runHead` is the input of `#runTail`: they form an "implicit workflow", whose steps have been executed manually rather than by a higher level software tool. The actions also refer to their executor via the `agent` property. The following diagram shows the relationships between the various entities:

<img alt="head-tail diagram" src="img/head-tail-diagram.svg" width="600" />

For more information, you can see the [Process Run Crate](https://w3id.org/ro/wfrun/process) profile, which contains the full specification.


## Workflow Run Crate

Suppose now that we write a script to automate the above process:

```bash
#!/usr/bin/env bash

set -euo pipefail

die() {
    echo $1 1>&2
    exit 1
}

nargs=3
if [ $# -ne ${nargs} ]; then
    die "Usage: $0 input_file head_lines tail_lines"
fi
input_file=$1
head_lines=$2
tail_lines=$3

output_file="chunk.txt"

workdir=$(mktemp -d)
head_output="${workdir}/temp.txt"
head --lines ${head_lines} "${input_file}" >"${head_output}"
tail --lines ${tail_lines} "${head_output}" >"${output_file}"
rm -rf "${workdir}"
```

This script can be considered a workflow, since it orchestrates the execution of other applications, managing the flow of data between them, to get the final output. Note that the script is parametrized, so every time it is run we can choose a different input file and different values for the number of lines to extract with `head` and `tail`. If you name the script `head_tail.sh`, you can obtain the same result as above by running:

```
bash head_tail.sh lines.txt 4 3
```

The result will be stored in the `chunk.txt` file.

A Workflow Run Crate to represent a run of this workflow would contain the workflow itself (`head_tail.sh`), the input and output files (`lines.txt`, `chunk.txt`) and an ro-crate-metadata.json file like the following:

```json
{
    "@context": "https://w3id.org/ro/crate/1.1/context",
    "@graph": [
        {
            "@id": "ro-crate-metadata.json",
            "@type": "CreativeWork",
            "about": {"@id": "./"},
            "conformsTo": [
                {"@id": "https://w3id.org/ro/crate/1.1"},
                {"@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0"}
            ]
        },
        {
            "@id": "./",
            "@type": "Dataset",
            "conformsTo": [
                {"@id": "https://w3id.org/ro/wfrun/process/0.1"},
                {"@id": "https://w3id.org/ro/wfrun/workflow/0.1"},
                {"@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0"}
            ],
            "hasPart": [
                {"@id": "head_tail.sh"},
                {"@id": "lines.txt"},
                {"@id": "chunk.txt"}
            ],
            "license": {"@id": "http://spdx.org/licenses/CC0-1.0"},
            "mainEntity": {"@id": "head_tail.sh"},
            "mentions": {"@id": "#runHeadTail"}
        },
        {
            "@id": "https://w3id.org/ro/wfrun/process/0.1",
            "@type": "CreativeWork",
            "name": "Process Run Crate",
            "version": "0.1"
        },
        {
            "@id": "https://w3id.org/ro/wfrun/workflow/0.1",
            "@type": "CreativeWork",
            "name": "Workflow Run Crate",
            "version": "0.1"
        },
        {
            "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0",
            "@type": "CreativeWork",
            "name": "Workflow RO-Crate",
            "version": "1.0"
        },
        {
            "@id": "head_tail.sh",
            "@type": ["File", "SoftwareSourceCode", "ComputationalWorkflow"],
            "name": "Head-Tail",
            "programmingLanguage": {"@id": "#bash"},
            "input": [
                {"@id": "#input_file-param"},
                {"@id": "#head_lines-param"},
                {"@id": "#tail_lines-param"}
            ],
            "output": [
                {"@id": "#output_file-param"}
            ]
        },
        {
            "@id": "#input_file-param",
            "@type": "FormalParameter",
            "additionalType": "File",
            "name": "input_file",
            "valueRequired": "True"
        },
        {
            "@id": "#head_lines-param",
            "@type": "FormalParameter",
            "additionalType": "Integer",
            "name": "head_lines",
            "valueRequired": "True"
        },
        {
            "@id": "#tail_lines-param",
            "@type": "FormalParameter",
            "additionalType": "Integer",
            "name": "tail_lines",
            "valueRequired": "True"
        },
        {
            "@id": "#output_file-param",
            "@type": "FormalParameter",
            "additionalType": "File",
            "name": "output_file"
        },
        {
            "@id": "#bash",
            "@type": "ComputerLanguage",
            "identifier": "https://www.gnu.org/software/bash/",
            "name": "Bash",
            "url": "https://www.gnu.org/software/bash/"
        },
        {
            "@id": "#runHeadTail",
            "@type": "CreateAction",
            "name": "Run of head_tail.sh",
            "endTime": "2023-05-04T17:02:01+02:00",
            "instrument": {"@id": "head_tail.sh"},
            "object": [
                {"@id": "lines.txt"},
                {"@id": "#head_lines-pv"},
                {"@id": "#tail_lines-pv"}
            ],
            "result": [
                {"@id": "chunk.txt"}
            ],
            "agent": {"@id": "https://orcid.org/0000-0002-1825-0097"}
        },
        {
            "@id": "lines.txt",
            "@type": "File",
            "exampleOfWork": {"@id": "#input_file-param"}
        },
        {
            "@id": "#head_lines-pv",
            "@type": "PropertyValue",
            "exampleOfWork": {"@id": "#head_lines-param"},
            "name": "head_lines",
            "value": "4"
        },
        {
            "@id": "#tail_lines-pv",
            "@type": "PropertyValue",
            "exampleOfWork": {"@id": "#tail_lines-param"},
            "name": "tail_lines",
            "value": "3"
        },
        {
            "@id": "chunk.txt",
            "@type": "File",
            "exampleOfWork": {"@id": "#output_file-param"}
        },
        {
            "@id": "https://orcid.org/0000-0002-1825-0097",
            "@type": "Person",
            "name": "Josiah Carberry"
        }
    ]
}
```

Note how the crate has a `mainEntity` that points to the workflow (`head_tail.sh`). This is in conformance with the [Workflow RO-Crate](https://w3id.org/workflowhub/workflow-ro-crate/1.0) profile. In addition to Workflow RO-Crate, Workflow Run Crates also conform to the [Process Run Crate](https://w3id.org/ro/wfrun/process) profile and to the [Workflow Run Crate](https://w3id.org/ro/wfrun/workflow) profile itself, as you can see from the `conformsTo` entries.

One of the main additions with respect to Process Run Crate is the representation of parameter "slots" (which can take different values according to the specific run) via `FormalParameter`. In this case we have three input parameters:

 * `#input_file-param`, corresponding to the input file
 * `#head_lines-param`, corresponding to the number of lines to select with the `head` program
 * `#tail_lines-param`, corresponding to the number of lines to select with the `tail` program

The `#runHeadTail` action has three `object` entries, which represent the actual values used for the parameters in this specific run. The first item is the `lines.txt` file, while the other two are represented via [PropertyValue](https://schema.org/PropertyValue) instances, where the `value` property provides the values taken by the parameters. Crucially, the `exampleOfWork` property connects each parameter value to the corresponding `FormalParameter`. Similarly, the `chunk.txt` file matches the output parameter `#output_file-param`.

An example of a Workflow Run Crate for a Galaxy workflow is available at [https://doi.org/10.5281/zenodo.7785860](https://doi.org/10.5281/zenodo.7785860).

You can export a Galaxy workflow invocation as a Workflow Run Crate by following [this tutorial](https://by-covid.github.io/gtn/ro-crate-in-galaxy/).


## Provenance Run Crate

[Provenance Run Crate](https://w3id.org/ro/wfrun/provenance) further extends Workflow Run Crate, specifying how to represent the actions corresponding to individual steps in the workflow. The extent to which this can be done may depend on how the workflow is written and how the workflow engine operates. For instance, the shell script we used in the previous section does not keep the intermediate output from the execution of `head`: it's stored in a temporary directory, which is deleted at the end of the script.

A workflow management system that supports Provenance Run Crate is [StreamFlow](https://streamflow.di.unito.it/), which executes [CWL](https://www.commonwl.org/) workflows. Let's write a CWL workflow that implements our head-tail application. We start by writing `headtool.cwl`, a wrapper for the `head` tool:

```yaml
class: CommandLineTool
cwlVersion: v1.0

baseCommand: head

inputs:
  lines:
    type: int
    inputBinding:
      position: 1
      prefix: "--lines"
  input:
    type: File
    inputBinding:
      position: 2
outputs:
  output:
    type: File
    outputBinding:
      glob: head_out.txt
stdout: head_out.txt
```

We then do the same for `tail`, calling the wrapper `tailtool.cwl`:

```yaml
class: CommandLineTool
cwlVersion: v1.0

baseCommand: tail

inputs:
  lines:
    type: int
    inputBinding:
      position: 1
      prefix: "--lines"
  input:
    type: File
    inputBinding:
      position: 2
outputs:
  output:
    type: File
    outputBinding:
      glob: tail_out.txt
stdout: tail_out.txt
```

Finally, we write the workflow, `head_tail.cwl`:

```yaml
class: Workflow
cwlVersion: v1.0

inputs:
  input_file:
    type: File
  head_lines:
    type: int
  tail_lines:
    type: int
outputs:
  output_file:
    type: File
    outputSource: tail/output

steps:
  head:
    in:
      lines: head_lines
      input: input_file
    out: [output]
    run: headtool.cwl
  tail:
    in:
      lines: tail_lines
      input: head/output
    out: [output]
    run: tailtool.cwl
```

Now that we have the workflow, let's write the file that contains the parameter settings for our specific run, `params.yml`:

```yaml
input_file:
  class: File
  location: lines.txt
head_lines: 4
tail_lines: 3
```

Now, install StreamFlow (we're going to need a version with RO-Crate support). To avoid possible conflicts, consider installing in a clean [virtual environment](https://docs.python.org/3/library/venv.html).

```
pip install 'streamflow==0.2.0.dev4'
```

Now run the workflow:

```
cwl-runner head_tail.cwl params.yml
```

To generate the RO-Crate, we're going to need the workflow's name in the StreamFlow database:

```console
$ streamflow list
NAME                                  TYPE EXECUTIONS 
f6b2273d-a945-4b27-aa98-dcfe0b778f4c  cwl  1
```

In your case, the name might be different. Now generate the RO-Crate with:

```
streamflow prov f6b2273d-a945-4b27-aa98-dcfe0b778f4c
```

taking care to replace the workflow name with the actual one you got from `streamflow list`. This will generate a zip archive containing the RO-Crate. Unzip the archive to get the RO-Crate directory:

```
unzip -d f6b2273d-a945-4b27-aa98-dcfe0b778f4c{,.crate.zip}
```

If you inspect `f6b2273d-a945-4b27-aa98-dcfe0b778f4c/ro-crate-metadata.json`, you will find a similar structure to the one seen for Workflow Run Crate, but with a lot more detail. Most notably, there will be three actions: in addition to the one representing the execution of the whole workflow, there will be two more, representing the execution of `headtool.cwl` and of `tailtool.cwl`. Each of this points to the corresponding tool via `instrument`, and to its inputs and outputs via `object` and `result`. The intermediate file resulting from the `head` run (`head_out.txt`) is included, and listed both as an output of the `head` run and as an input of the `tail` run. This situation is similar to what we've seen in the [Process Run Crate example](#process-run-crate), but this time everything has been orchestrated by a workflow engine on the basis of a formal workflow description.

More information on the entities represented in a Provenance Run Crate is available in the [specification](https://w3id.org/ro/wfrun/provenance).

An example a Provenance Run Crate representing an execution of a digital pathology image annotation workflow is available at [https://doi.org/10.5281/zenodo.7669622](https://doi.org/10.5281/zenodo.7669622).


topic: fair
tutorial: ro-crate-galaxy-best-practices
title: Best practices for workflows in GitHub repositories
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["ro-crate"]
content_hash: b7f35a5f43ab9b16fda3a50f49bdb69c
last_modified: 2026-05-27T09:01:48.602132
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.html

content:
A workflow, just like any other piece of software, can be formally correct and runnable but still lack a number of additional features that might help its reusability, interoperability, understandability, etc.

One of the most useful additions to a workflow is a suite of tests, which help check that the workflow is operating as intended. A test case consists of a set of inputs and corresponding expected outputs, together with a procedure for comparing the workflow's actual outputs with the expected ones. It might be the case, in fact, that a test may be considered successful even if the actual outputs do not match the expected ones exactly, for instance because the computation involves a certain degree of randomness, or the output includes timestamps or randomly generated identifiers.

Providing documentation is also important to help understand the workflow's purpose and mode of operation, its requirements, the effect of its parameters, etc. Even a single, well structured README file can go a long way towards getting users started with your workflow, especially if complemented by examples that include sample inputs and running instructions.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn about the best practices that the Galaxy community 
> has created for workflows.
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <tip-title>Using your own workflow</tip-title>
>
> This tutorial assumes that you already have a Galaxy workflow that you want to apply best practices to. You can follow along using any workflow you have created or imported during a previous tutorial (such as [A short introduction to Galaxy]({% link topics/introduction/tutorials/galaxy-intro-short/workflows/index.md %})).
>
{: .tip}

## Community best practices

Though the practices listed in the introduction can be considered general enough to be applicable to any kind of software, individual communities usually add their own specific sets of rules and conventions that help users quickly find their way around software projects, understand them more easily and reuse them more effectively. The Galaxy community, for instance, has a [guide on best practices for maintaining workflows](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html) and a built-in Best Practices panel in the workflow editor (see the tip below).

{% snippet faqs/galaxy/workflows_best_practices.md %}

> <hands-on-title>Apply best practices for workflow structure</hands-on-title>
>
> 1. Open your workflow for editing and find the **Best Practices** panel (see the tip above).
> 1. Resolve the warnings that appear until every item has a green tick.
>
{: .hands_on}

The [Intergalactic Workflow Commission (IWC)](https://github.com/galaxyproject/iwc) is a collection of highly curated Galaxy workflows that follow best practices and conform to a specific GitHub directory layout, as specified in the [guide on adding workflows](https://github.com/galaxyproject/iwc/blob/main/workflows/README.md#adding-workflows). In particular, the workflow file must be accompanied by a [Planemo test file](https://planemo.readthedocs.io/en/latest/test_format.html) with the same name but a `-test.yml` extension, and a `test-data` directory that contains the datasets used by the tests described in the test file. The guide also specifies how to fulfill other requirements such as setting a license, a creator and a version tag. A new workflow can be proposed for inclusion in the collection by opening a pull request to the [IWC repository](https://github.com/galaxyproject/iwc): if it passes the review and is merged, it will be published to [iwc-workflows](https://github.com/iwc-workflows). The publication process also generates a metadata file that turns the repository into a [Workflow Testing RO-Crate](https://crs4.github.io/life_monitor/workflow_testing_ro_crate), which can be registered to [WorkflowHub](https://workflowhub.eu/) and [LifeMonitor](https://www.lifemonitor.eu/).


## Best practice repositories and RO-Crate

The [repo2rocrate](https://github.com/crs4/repo2rocrate) software package allows to generate a [Workflow Testing RO-Crate](https://crs4.github.io/life_monitor/workflow_testing_ro_crate) for a workflow repository that follows community best practices. It currently supports Galaxy (based on IWC guidelines), Nextflow and Snakemake. The tool assumes that the workflow repository is structured according to the community guidelines and generates the appropriate [RO-Crate](https://w3id.org/ro/crate/) metadata for the various entities. Several command line options allow to specify additional information that cannot be automatically detected or needs to be overridden.

To try the software, we'll clone one of the iwc-workflows repositories, whose layout is known to respect the IWC guidelines. Since it already contains an RO-Crate metadata file, we'll delete it before running the tool.

```bash
pip install repo2rocrate
git clone https://github.com/iwc-workflows/parallel-accession-download
cd parallel-accession-download/
rm -fv ro-crate-metadata.json
repo2rocrate --repo-url https://github.com/iwc-workflows/parallel-accession-download
```

This adds an `ro-crate-metadata.json` file at the top level with metadata generated based on the tool's knowledge of the expected repository layout. By specifying a zip file as an output with the `-o` option, we can directly generate an RO-Crate in the format accepted by WorkflowHub and LifeMonitor:

```bash
repo2rocrate --repo-url https://github.com/iwc-workflows/parallel-accession-download -o ../parallel-accession-download.crate.zip
```


## Generating tests for your workflow

What if you only have a workflow, but you don't have the test layout yet? You can use Planemo to generate it.

```bash
pip install planemo
```

As an example we will use this [simple workflow](https://github.com/crs4/life_monitor/blob/50cdb790ff125613aa07e70cb439e3a36b82d0bf/interaction_experiments/workflow_examples/galaxy/sort-and-change-case/sort-and-change-case.ga), which has only two steps: it sorts the input lines and changes them to upper case. Follow these steps to generate a test layout for it:


> <hands-on-title>Generate Workflow Tests With Planemo</hands-on-title>
> 1. Download [the workflow](https://raw.githubusercontent.com/crs4/life_monitor/50cdb790ff125613aa07e70cb439e3a36b82d0bf/interaction_experiments/workflow_examples/galaxy/sort-and-change-case/sort-and-change-case.ga) to a `sort-and-change-case.ga` file.
> 1. Download [this input dataset](https://raw.githubusercontent.com/crs4/life_monitor/50cdb790ff125613aa07e70cb439e3a36b82d0bf/interaction_experiments/workflow_examples/galaxy/sort-and-change-case/input.bed) to an `input.bed` file.
> 1. Upload the workflow to Galaxy (e.g., [Galaxy Europe](https://usegalaxy.eu/)): from the upper menu, click on "Workflow" > "Import" > "Browse", choose `sort-and-change-case.ga` and then click "Import workflow".
> 1. Rename the uploaded workflow from `sort-and-change-case (imported from uploaded file)` to `sort-and-change-case` by clicking the pencil icon next to the workflow name. 
> 1. Start a new history: click on the "+" button on the History panel to the right.
> 1. Upload the input dataset to the new history: on the left panel, go to "Upload Data" > "Choose local files" and select `input.bed`, then click "Start" > "Close".
> 1. Wait for the file to finish uploading (i.e., for the loading circle on the dataset's line in the history to disappear).
> 1. Run the workflow on the input dataset: click on "Workflow" in the upper menu, locate `sort-and-change-case`, and click on the play button to the right.
> 
>   ![Workflow Entry](img/workflow-entry.png)
> 
> 1. This should take you to the workflow running page. The input slot should be already filled with `input.bed` since there is nothing else in the history. Click on "Run Workflow" on the upper right of the center panel.
> 
>    ![Workflow Run Page](img/workflow-run-page.png)
> 
> 1. Wait for the workflow execution to finish.
> 1. On the upper menu, go to "Data" > "Workflow Invocations", expand the invocation corresponding to the workflow just run and copy the invocation's ID. In my case it says "Invocation ID: 86ecc02a9dd77649" on the right, where `86ecc02a9dd77649` is the ID.
>  
>    ![Workflow Invocation](img/workflow-invocation.png)
> 
> 1. On the upper menu, go to "User" > "Preferences" > "Manage API Key". If you don't have an API key yet, click the button to create a new one. Under "Current API key", click the button to copy the API Key on the right.
> 
>    ![API key](img/api-key.png)
> 
> 1. Run `planemo workflow_test_init --galaxy_url https://usegalaxy.eu --from_invocation INVOCATION_ID --galaxy_user_key API_KEY`, replacing `INVOCATION_ID` with the actual invocation ID and `API_KEY` with the actual API key. If you're not using the Galaxy Europe instance, also replace `https://usegalaxy.eu` with the URL of the instance you're using.
> 1. Browse the files that have been created - `sort-and-change-case-tests.yml` and `test_data/`
>
> Optionally see this tip for more details:
>
> {% snippet faqs/gtn/gtn_workflow_testing.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How do the files in `test_data/` relate to your Galaxy history?
> 2. Look at the contents of `sort-and-change-case-tests.yml`. What are the expected outputs of the test?
>
> > <solution-title></solution-title>
> >
> > 1. The files in `test_data/` correspond to the output files in the history, though some of the names are different:
> >   1. `bed_input.bed` has the same name in the history - this is the input file we uploaded
> >   2. `sorted_bed.bed` corresponds to the `Sort on data 1` step (you can confirm this by viewing the file contents)
> >   2. `uppercase_bed.tabular` corresponds to the `Change case on data 2` step (you can confirm this by viewing the file contents)
> > 2. The expected outputs are `test-data/sorted_bed.bed` and `test-data/uppercase_bed.tabular`. This means that when the workflow is run on the input (`test-data/bed_input.bed`), it is expected to produce two files that look exactly like those outputs.
> >
> {: .solution}
{: .question}

To build up the test suite further, you can invoke the workflow multiple times with different inputs, and use each invocation to generate a test, using the same command as before:

```bash
planemo workflow_test_init --galaxy_url https://usegalaxy.eu --from_invocation INVOCATION_ID --galaxy_user_key API_KEY
```

Each invocation should test a different behavior of the workflow. This could mean using different datatypes for inputs, or changing the workflow settings to produce different results.

> <hands-on-title>Generate tests for your own workflow</hands-on-title>
> 1. Create a new folder on your computer to store the workflow.
> 1. Download the Galaxy workflow you updated to follow best practices earlier in this tutorial. You can do this by going to the Workflow page and clicking {% icon galaxy-download %} **Download workflow in .ga format**.
> 1. Create a new Galaxy history, and run the workflow on some appropriate input data.
> 1. Use `planemo` to turn that workflow invocation into a test case.
{: .hands_on}


## Adding a GitHub workflow for running tests automatically

In the previous section, you learned how to generate a test layout for an example Galaxy workflow. This procedure also gives you the file structure you need to populate the GitHub repository in line with community best practices. One thing is still missing though: a GitHub workflow to test the Galaxy workflow automatically. Let's create this now.

At the top level of the repository, create a `.github/workflows` directory and place a `wftest.yml` file inside it with the following content:

```yaml
name: Periodic workflow test
on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:
jobs:
  test:
    name: Test workflow
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 1
    - uses: actions/setup-python@v1
      with:
        python-version: '3.7'
    - name: install Planemo
      run: |
        pip install --upgrade pip
        pip install planemo
    - name: run planemo test
      run: |
        planemo test --biocontainers sort-and-change-case.ga
```

Replacing `sort-and-change-case.ga` with the name of your actual Galaxy workflow. You can find extensive [documentation on GitHub workflows](https://docs.github.com/en/actions/using-workflows) on the GitHub web site. Here we'll give some highlights:

 * the `on` field sets the GitHub workflow to run:
   * automatically every day at 3 AM
   * when manually dispatched
 * the steps do the following:
   * check out the GitHub repository
   * set up a Python environment
   * install Planemo
   * run `planemo test` on the Galaxy workflow

An example of a repository built according to the guidelines given here is [simleo/ccs-bam-to-fastq-qc-crate](https://github.com/simleo/ccs-bam-to-fastq-qc-crate), which realizes the Workflow Testing RO-Crate setup for [BAM-to-FASTQ-QC](https://workflowhub.eu/workflows/220).

Your workflow is now ready to add to GitHub! If you're not familiar with GitHub, follow these instructions to create a repository and upload your workflow files: [Uploading a project to GitHub](https://docs.github.com/en/get-started/start-your-journey/uploading-a-project-to-github).


topic: fair
tutorial: med-chem-data
title: Data management in Medicinal Chemistry
difficulty: introductory
time_estimation: 1H
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "data-management", "medicinal-chemistry", "computational-chemistry"]
content_hash: 90fafe76f1a52d7c8bc52e11c9055bb1
last_modified: 2026-05-15T22:28:10.769710
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/med-chem-data/tutorial.html

content:
The development of medicinal chemistry is advancing very rapidly. Big pharmaceutical companies, research institutes and universities are working on ground-breaking solutions to help patients combat all kinds of diseases. During that development process, tons of data are generated – not only from the lab environment but also from clinical trials. Given that the discovery of more potent, safer and cheaper drugs is the ultimate goal of all research bodies, we should all focus on making the data we gather FAIR: **F**indable, **A**ccessible, **I**nteroperable, and **R**eusable to push the boundaries of drug development even further. 

With the currently available methods such as artificial intelligence, machine learning, many toolkits, software and access to various databases, managing big data is now inherently linked to medicinal chemistry and helps to make this area as efficient as it can be. 

In this tutorial, we will therefore explore some concepts related to medicinal chemistry, explore the available chemical and pharmacological databases, and perform some basic data analyses using the Galaxy interface. 

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Principles of medicinal chemistry research

## Drug-likeness
There are many factors that medicinal chemists take into account while designing new drugs. It is essential to estimate the properties of the molecule before synthesising it in the lab, so the structures of drug candidates are usually compared to the existing drugs by so-called drug-likeness. This includes intrinsic properties of a compound that will lead to favourable ADMET (**A**bsorption, **D**istribution, **M**etabolism, **E**xcretion, **T**oxicity) characteristics. Below are some properties usually assessed:
- Lipophilicity
- Size
- Polarity
- Insolubility
- Unsaturation
- Flexibility

Additionally, there have been some rules developed to help estimate drug-likeness, amongst which:
- Lipinski rule ({% cite Lipinski2001 %})
- Ghose rule ({% cite Ghose1999 %})
- Veber rule ({% cite Veber2002 %})
- Egan rule ({% cite Egan2000 %})
- Muegge rule ({% cite Muegge2001 %})
- Abbott bioavailability score ({% cite Martin2005 %})


As you see, there are a number of characteristics to consider before even synthesising the molecule in a lab. One of the online tools that help to summarise all the most important information about intrinsic properties and drug-likeness of compounds is called [SwissADME](http://www.swissadme.ch/) and is a free web tool to evaluate pharmacokinetics, drug-likeness and medicinal chemistry friendliness of small molecules ({% cite Daina2017 %}). 

Below we briefly discuss two probably the most well-known rules: Lipinski's and Veber's rules.

## The Lipinski rule of 5

This rule of thumb developed back in 1997 by Christopher Lipinski and colleagues tries to predict the likelihood that a given small molecule can be orally active. The Lipinski rule of 5 favours molecules as potential oral drug candidates if:
-	the molecular mass is less than 500
-	the calculated logarithm of the octanol−water partition coefficient (clogP) is less than 5
-	they have up to 5 H-bond donors
-	they have up to 10 H-bond acceptors

However, nowadays there are more and more drugs being developed which don’t comply with those rules and regardless are still effective. There are voices from the scientific community, pointing out that “We are in danger of repeating our past mistakes if we assume these new modalities are not 'drug-like' and cannot be oral drugs because they are not [rule of 5] compliant” (Michael Shultz from Novartis, cited in {% cite Mullard2018 %} where the authors re-assess the rule of 5). Then, in {% cite OHagan2014 %} we read "This famous "rule of 5" has been highly influential in this regard, but only about 50 % of orally administered new chemical entities actually obey it." 


## Veber's rule

In ({% cite Veber2002 %}) we read that the commonly applied molecular weight cutoff at 500 does not itself significantly separate compounds with poor oral bioavailability from those with acceptable values. The authors explain that on average both the number of rotatable bonds and polar surface area or hydrogen bond count tend to increase with molecular weight which may in part explain the success of the molecular weight parameter in predicting oral bioavailability. Their observations suggest that compounds will have a high probability of good oral bioavailability if they meet the two criteria: 
- 10 or fewer rotatable bonds - increased rotatable bond count has a negative effect on the permeation rate
- a polar surface area no greater than 140 Å <sup>2</sup> (or 12 or fewer H-bond donors and acceptors) - reduced polar surface area correlates better with increased permeation rate than does lipophilicity


# Big data for drug discovery

Have you ever wondered how new drugs are developed? It is actually quite a long and complex pathway and not only takes lots of time but also resources. The scheme below will help you understand the scale of that process.

![A scheme showing drug discovery pipeline: starting from target identification, hit, lead, lead optimisation, pre-clinical testing, clinical evaluation in phases I - III, and finally regulatory approval. Lead optimisation also includes iterative steps of designing, making, testing and analysing the proposed compounds.]({% link topics/computational-chemistry/images/medchem.png %} "A scheme showing drug discovery pipeline")

As you can see, to get just one drug to the market, often thousands of structures are assessed to find the most effective and safe compound. That's lots of information being generated in the whole process! The information can be stored in repositories and databases so that subsequent drug discovery process in a similar area is easier, faster and cheaper. Managing big data and using machine learning and computational chemistry methods is crucial in the lead optimisation step. By reviewing chemical databases and studying molecular docking simulations, we can save much time and resources to synthesise only those compounds that are the most promising based on *in-silico* methods. Even after synthesising the optimised 'leads', we still get lots of data from tests and analyses. By using appropriate cheminformatic tools and software, we can use the results to drive forward this iterative process of lead optimisation.


## Ten Vs of big data for drug discovery

Hopefully, now you have an idea of what the process of drug discovery looks like. To understand why managing data in medicinal chemistry is so challenging, let's have a closer look at the "Ten Vs" of big data {% cite Zhao2020 %}:

- **V**olume: size of data. 
- **V**elocity: speed of new data generation. 
- **V**ariety: various formats of data. 
- **V**eracity: quality of data. 
- **V**alidity: authenticity of data. 
- **V**ocabulary: terminology of data. 
- **V**enue: platform of data generation. 
- **V**isualization: view of data. 
- **V**olatility: duration of data usefulness. 
- **V**alue: the potential of data usefulness to reduce the cost of drug discovery and development.

Medicinal chemists must keep in mind those features while both searching for data and publishing it. In this way we can focus on the main aspects of data management, try to improve the availability and normalisation of data, and be aware of limitations of repositories and inconsistencies in data quality. 

## FAIR MedChem
As you saw before, there are many properties that the compounds can be searched by, such as molecular weight, H-bond donors or acceptors, polarity, etc. Sometimes there is a need to assess other molecular properties but it might happen that not all the information is included in the database. This is one of the reasons why new repositories are being developed – they are more specific and gather particular properties of interest. How much easier the life of scientists could be if the relevant data was publicly available, well-ordered and contained the needed metadata? By submitting the data with the necessary information to the repository is a good way to make the data FAIR. This will make it:
- **F**indable, as the data will be given specific identifiers
- **A**ccessible, as the data will be available online, open and free where possible
-	**I**nteroperable, as the repository will often enforce the use of formalised, consistent language
-	**R**eusable, as the data will be released under a license with detailed provenance
In the repositories 


# Chemical and pharmacological databases

Currently, there are lots of publicly available databases storing information about hit molecules, chemical structures and properties, drug targets, pharmaceuticals… In the recent paper {% cite Zhao2020 %} the authors collated relevant databases to advance computer-aided drug discovery and divided them into several main groups:
- Chemical collections
- Drug / drug-like compounds
- Drug targets, including genomics and proteomics data
- Biological data from assay screening, metabolism, and efficacy studies
- Drug liabilities and toxicities
- Clinical databases

Below you will find the databases listed under corresponding categories, all taken from {% cite Zhao2020 %} paper. 

There are many more databases available, and many are still being developed. They are often quite specific and contain certain types of compounds (eg. [PROTACs](http://cadd.zju.edu.cn/protacdb/) or are aimed at a particular disease. If the listed databases are not specific enough for your research, you can try luck by searching smaller, more specific databases. 

## Chemical collections

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [Enamine REAL Database](https://enamine.net/hit-finding/compound-collections/real-database) | Tool used to find new hit molecules using large-scale virtual screening and for searching analogues to hit molecules | >700 million compounds that comply with ‘rule of 5’ and Veber criteria |
| [ZINC](http://zinc.docking.org/) | Contains compound information including 2D/3D structure, purchasability,   target, and biology-related information | >230 million compounds in 3D formats and >750 million compounds for analogue searching |
| [PubChem](https://pubchem.ncbi.nlm.nih.gov) | Contains chemical molecule (mostly small molecule) information, including chemical structures, identifiers, chemical and physical properties,   biological activities, safety and toxicity data | 97 million compounds, 236 million substances, 268 million bioactivities |
| [ChemSpider](https://www.chemspider.com/) | Free chemical structure database providing fast access to >78 million   structures, properties, and associated information | >78 million compound structures |
| [SCUBIDOO](http://kolblab.org/scubidoo/index.php) | Freely accessible database that currently holds 21 million virtual products originating from a small library of building blocks and collection of robust organic reactions | 21 million virtual products |
| [ChEMBL](https://www.ebi.ac.uk/chembl/) | Manually curated database of bioactive molecules with drug-like properties; brings together chemical, bioactivity, and genomic data to aid translation of genomic information into effective new drugs. | >1.9 million compounds, 1.1 million pieces of assay information |
| [TCM-Mesh](http://mesh.tcm.microbioinformatics.org/) | Integration of a database and a data-mining system for network pharmacology analysis of all respects of traditional Chinese medicine,   including herbs, herbal ingredients, targets, related diseases, adverse effect, and toxicity | 383 840 compounds, 6235 herbs |
| [Super Natural II](http://bioinf-applied.charite.de/supernatural_new/index.php) | Contains natural compounds, including information about corresponding 2D   structures, physicochemical properties, predicted toxicity class and potential vendors | 325 508 natural compounds |
| [BIAdb](https://webs.iiitd.edu.in/raghava/biadb/index.html) | Comprehensive database of benzylisoquinoline alkaloids, containing information about 846 unique benzylisoquinoline alkaloids | ~846 unique benzylisoquinoline alkaloids |

## Drug / drug-like compounds

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [AICD](http://956023.ichengyun.net/AICD/index.php) | Anti-Inflammatory Compounds Database (AICD) deposits compounds with potential antiinflammation activities | 79 781 small molecules |
| [DrugBank](https://go.drugbank.com/) | Unique bioinformatics and cheminformatics resource that combines detailed drug data with comprehensive drug target information | 13 441 drug entries |
| [ReFRAME](https://reframedb.org/) | Screening library of 12 000 molecules assembled by combining three databases (Clarivate Integrity, GVK Excelra GoStar, and Citeline   Pharmaprojects) to facilitate drug repurposing | 12 000 molecules |
| [SuperDRUG2](http://cheminfo.charite.de/superdrug2/) | Contains approved/marketed drugs with regulatory details, chemical structures (2D and 3D), dosage, biological targets, physicochemical properties, external identifiers, adverse effects, and PK data | >4600 active pharmaceutical ingredients |
| [Drugs@FDA database](https://www.fda.gov/drugs/drug-approvals-and-databases/drugsfda-data-files) | Information about drugs from FDA | ~23 391 drug application records |
| [e-Drug3D](https://chemoinfo.ipmc.cnrs.fr/MOLDB/index.php) | Contains 1930 molecular structures approved by FDA between 1939 and 2019   with a molecular weight <2000 | 1930 drugs |

## Drug targets, including genomics and proteomics data

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [BindingDB](https://www.bindingdb.org/bind/index.jsp) | Public, web-accessible database of measured binding affinities, focusing chiefly on interactions of proteins considered to be candidate drug targets with ligands that are small,   drug-like molecules | 1 756 093 binding data, for 7371 protein targets and 780 240 small molecules |
| [Supertarget](http://insilico.charite.de/supertarget/index.php?site=home) | An extensive web resource for analyzing drugtarget interactions. | 332 828 drug-target interactions |
| [Ligand Expo](http://ligand-expo.rutgers.edu/index.html) | Provides chemical and structural information about small molecules within structure entries of Protein Data   Bank. | 30 440 entries of ligand |
| [PDBeChem](https://www.ebi.ac.uk/pdbe-srv/pdbechem/) | Consistent and enriched library of ligands, small molecules, and monomers referenced as residues and hetgroups in PDB entries | >29 922 ligands |
| [PDBbind-CN](http://www.pdbbind-cn.org/) | Provides essential linkage between energetic and structural information of biomolecular complexes, which is helpful for various computational and statistical studies on molecular recognition in biological systems | 19 588 biomolecular complexes |
| [STITCH](http://stitch.embl.de/) | Database integrating information about interactions from metabolic pathways, crystal structures, binding experiments, and drug–target relationships | Interactions between 300 000 small molecules and 2.6 million proteins from 1133 organisms |
| [BioGRID](https://thebiogrid.org/) | The Biological General Repository for Interaction Datasets is an open-access database on protein, genetic, and chemical interactions for humans and all major model organisms | 1 753 686 protein and genetic interactions, 28 093 chemical associations and 874 796 posttranslational modifications from major model organisms |
| [Binding MOAD](http://bindingmoad.org/) | Created from a subset of Protein   Data Bank (PDB), containing every high-quality example of ligand-protein binding. | 36 047 protein-ligand structures, and 13 353 binding data |
| [GPCRdb](https://gpcrdb.org/) | Contains data from GPCRs, including crystal structures, sequence alignments, and receptor mutations; can be visualized in interactive diagrams; provides online analysis tools | 15 149 proteins, and 144 917 ligands |
| [Guide to Pharmacology](https://www.guidetopharmacology.org/) | IUPHAR/BPS guide to pharmacology is an open-access, expert-curated database of molecular interactions between ligands and their targets. | 2937 targets, and 9859 ligands |
| [GLASS](https://zhanglab.ccmb.med.umich.edu/GLASS/) | GPCR-Ligand Association (GLASS)   database is a manually curated repository for experimentally validated   GPCR–ligand interactions; along with relevant GPCR and chemical information,   GPCRligand association data are extracted and integrated into GLASS from literature and public databases | ~ 277 651 unique ligands and 3048 GPCRs |

## Biological data from assay screening, metabolism, and efficacy studies

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [HMDB](https://hmdb.ca/about) | Freely available electronic database containing detailed information about small-molecule metabolites found in the human body | 114 162 metabolite entries |
| [SMPDB](http://smpdb.ca/) | Small Molecule Pathway Database (SMPDB) is an interactive, visual database containing >30 000 small-molecule pathways found in humans only | >30 000 small-molecule pathways |
| [TTD](http://db.idrblab.net/ttd/) | Therapeutic Target Database (TTD) is a database providing information about known and explored therapeutic protein and nucleic acid targets,   targeted disease, pathway information and corresponding drugs directed at each of these targets | 2589 targets, and 31 614 drugs |
| [BioCyc](https://biocyc.org/) | Collection of 7615 pathway/genome databases; each database in BioCyc collection describes genome and metabolic pathways of a single organism | 7615 pathway/genome databases |
| [BiGG](http://bigg.ucsd.edu/) | Metabolic reconstruction of human metabolism designed for systems biology simulation and metabolic flux balance modelling | 2004 proteins, 2766 metabolites, and 3311 metabolic and transport reactions |
| [BRENDA](https://www.brenda-enzymes.org/) | Main collection of enzyme functional data available to scientific   community | At least 40 000 different enzymes from >6900 different organisms |
| [Reactome](https://reactome.org/) | Curated, peer-reviewed knowledgebase of biological pathways, including metabolic pathways, and protein trafficking and signalling pathways | >9600 proteins, 9800 reactions, and 2000 pathways for humans |
| [BioModels Database](https://www.ebi.ac.uk/biomodels/) | Repository of computational models of biological processes; models   described from literature are manually curated and enriched with   cross-references | 6753 patient-derived genome-scale metabolic models, 112 898 metabolic   models etc. |
| [KEGG](https://www.genome.jp/kegg) | Database resource that integrates genomic, chemical, and systemic functional information | 18 652 metabolites |
| [CARLSBAD](http://carlsbad.health.unm.edu/carlsbad/?mode=home) | Database and knowledge inference system that integrates multiple bioactivity data sets to provide researchers with novel capabilities for mining and exploration of available structure-activity relationships (SAR)   throughout chemical biology space. | 932 852 CARLSBAD activities, 890 323 unique structure–target pairs, 3542   targets, 435 343 unique structures |
| [WOMBAT](http://dud.docking.org/wombat/) | Contains 331 872 entries, representing 1966 unique targets, with bioactivity annotations | 268 246 unique structures |
| [Open NCI Database](https://cactus.nci.nih.gov/ncidb2.2/) | Maintained by the National Cancer Institute; contains small-molecule information such as names, biological activities, and structures; a useful resource for researchers working in cancer/AIDS fields | >250 000 compounds |
| [NPACT](http://crdd.osdd.net/raghava/npact/) | Provides information on the plant-derived natural compounds, including structure, properties (physical, elemental, and topological), cancer type,   cell lines, inhibitory values (IC50, ED50, EC50, GI50), molecular targets,   commercial suppliers, and drug-likeness of compounds | 1574 entries |
| [PKPB_DB](https://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=204443) | Contains physiological parameter values for humans from early childhood through senescence; intended to be used in physiologically based (PB) PK modelling; also contains similar data for animals (primarily rodents) | N/A |

## Drug liabilities and toxicities

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [T3DB](http://www.t3db.ca/) | Unique bioinformatics resource that combines detailed toxin data with comprehensive toxin target information | 3678 toxins |
| [DrugMatrix](https://ntp.niehs.nih.gov/data/drugmatrix/) | One of world’s largest toxicogenomic reference resources | ~ 600 drug molecules and 10 000 genes |
| [ACToR](https://actor.epa.gov/actor/home.xhtml) | Includes computational toxicology information about compounds, including   HTS, chemical exposure, sustainable chemistry (chemical structures and physicochemical properties) and virtual tissue data | >500 000 chemicals |
| [SkinSensDB](https://cwtung.kmu.edu.tw/skinsensdb/) | Contains curated data from published AOP-related skin sensitization   assays | 710 unique chemicals |
| [SIDER](http://sideeffects.embl.de/download/) | Contains information on marketed medicines and their recorded adverse drug reactions, including frequency, drug and adverse effect classifications | 1430 drugs with 5868 side effect information |
| [LTKB Benchmark Dataset](https://www.fda.gov/science-research) | Contains drugs with potential to cause drug-induced liver injury in humans; established using FDA-approved prescription drug labels;   liver-toxicity-knowledge-base-ltkb/ltkb-benchmark-dataset | 287 prescription drugs |
| [CTD](http://ctdbase.org/) | Comparative Toxicogenomics Database (CTD) is a premier public resource for literature-based, manually curated associations between chemicals, gene products, phenotypes, diseases, and environmental exposures | 13 378 unique chemicals and related information |

## Clinical databases

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [ClinicalTrials.gov](https://clinicaltrials.gov/) | Database of privately and publicly funded clinical studies conducted around the world | ~ 324 429 research studies in all 50 US states and 209 countries |
| [AACT database](https://aact.ctti-clinicaltrials.org/) | Publicly available relational database that contains all information   (protocol and result data elements) about every study registered in   ClinicalTrials.gov. Content is downloaded from ClinicalTrials.gov daily and loaded into AACT | ~ 324 429 research studies in all 50 US states and 209 countries |
| [EORTC Clinical Trials Database](https://www.eortc.org/clinical-trials-database/) | Contains information about EORTC clinical trials and clinical trials from other organizations with EORTC participation | N/A |
| [Exposome-Explorer](http://exposome-explorer.iarc.fr/) | Contains detailed information on the nature of biomarkers, populations and subjects where measured, samples analyzed, methods used for biomarker analyses, concentrations in biospecimens, correlations with external exposure measurements, and biological reproducibility over time | 908 biomarkers |
| [PharmaGKB](https://www.pharmgkb.org/) | A pharmacogenomics knowledge resource that encompasses clinical information about drug molecules | 733 drugs with their clinical information |


# Data management in chemistry using Galaxy

## Chemical file formats

Depending on the type of analysis you are going to perform, you will work with different file formats. Below are the most important data types, commonly used in cheminformatics, that you will likely see in Galaxy:
- SDF (.sd, .sdf) - stands for Structure-Data Format, and SDF files wrap the molfile (MDL Molfile) format. Stores information about the chemical structure and associated data of compounds in plain text. Files in SDF format can encode single or multiple molecules that are then delimited by lines consisting of four dollar signs. SDF files are formatted ASCII files that store information about the positions of the individual atoms (either in 2D or 3D space) that make up the molecule. The data on connectivity and hybridization state are also encoded, although their use is less frequent and often inconsistent.
- MOL (.mol) - an MDL Molfile for holding information about the atoms, bonds, connectivity and coordinates of a molecule. 
- MOL2 (.mol2) - a Tripos Mol2 file is a complete, portable representation of a SYBYL molecule. It is an ASCII file which contains all the information needed to
reconstruct a SYBYL molecule
- CML (.cml) - Chemical Markup Language (ChemML or CML) is an approach to managing molecular information using tools such as XML and Java. It supports a wide range of chemical concepts including molecules, reactions, spectra and analytical data, computational chemistry, chemical crystallography and materials
- InChI (IUPAC International Chemical Identifier) - a textual identifier for chemical substances, designed to provide a standard way to encode molecular information. The identifiers describe chemical substances in terms of layers of information — the atoms and their bond connectivity, tautomeric information, isotope information, stereochemistry, and electronic charge information ({% cite Heller2015 %})
- SMILES (.smi) - the **s**implified **m**olecular-**i**nput **l**ine-**e**ntry **s**ystem (SMILES) is a specification in the form of a line notation for describing the structure of chemical species using short ASCII strings. A linear text format which can describe the connectivity and chirality of a molecule ({% cite Weininger1988 %}) 
- PDB - the Protein Data Bank (PDB) file format is a textual file format describing the three-dimensional structures of molecules held in the [Protein Data Bank](https://www.rcsb.org/), now succeeded by the mmCIF format. It contains a description and annotation of protein and nucleic acid structures including atomic coordinates, secondary structure assignments, as well as atomic connectivity. In addition, experimental metadata is stored. ({% cite Berman2007 %})
- GRO (.gro) - a plain text file storing spatial coordinates and velocities (if available) of atoms during a molecular dynamics simulation, utilised by [GROMACS](https://manual.gromacs.org/archive/5.0.3/online/gro.html)

To visualise the structures held by the files with the positions of the individual atoms, you can use (NGL Viewer)[https://nglviewer.org/ngl/].

If you plan to work with molecular dynamics simulations, there are also some MD trajectory file formats that you might want to get familiar with. In Galaxy, you can convert between xtc, trr, dcd and netcdf files using a tool called {% tool [MDTraj file converter](toolshed.g2.bx.psu.edu/repos/chemteam/md_converter/md_converter/1.9.6+galaxy0) %}.

## Galaxy tools 

In Galaxy Chemical Toolbox there are dozens of tools that can be used for various analyses. Below we will only show a few, mostly related to data import, format conversion and some functions linked to what was discussed previously. 

Let's start with importing publicly available data. [Protein Data Bank](https://www.rcsb.org/) stores thousands of three-dimensional structural data of proteins nucleic acids and other biological molecules.

> <hands-on-title>Import PDB file from Protein Data Bank</hands-on-title>
>
> 1. Open [Protein Data Bank](https://www.rcsb.org/)
> 2. Type your search term in the box *"Enter search term(s), Entry ID(s), or sequence"*: `MAO B`
> ![A screenshot showing a search box in the upper right corner of the Protein Data Bank website.]({% link topics/computational-chemistry/images/pdb_search.jpg %} "Search box of the Protein Data Bank")
>3. Choose the structure of interest and copy its accession code: 2BK3
> ![A screenshot showing some of the resulting structures with their accession codes.]({% link topics/computational-chemistry/images/pdb_results.jpg %} "How to get accession code of the resulting structures.")
>4. Switch to Galaxy
>5. {% tool [Get PDB file](toolshed.g2.bx.psu.edu/repos/bgruening/get_pdb/get_pdb/0.1.0) %} with the following parameters:
>    - *"PDB accession code"*: `2BK3`
>
{: .hands_on}

This is often the first step of docking studies that you can learn about in other tutorials, for example in [Protein-ligand docking tutorial]({% link topics/computational-chemistry/tutorials/cheminformatics/tutorial.md %}).

You can import any molecule to Galaxy using SMILES notation. Below we show an example with benzenesulfonyl chloride and ethylamine. 

> <hands-on-title>Import SMILES</hands-on-title>
>
> 1. Copy SMILES of your molecule(s) of interest. In this example, we use benzenesulfonyl chloride (SMILES: C1=CC=C(C=C1)S(=O)(=O)Cl) and ethylamine (SMILES: CCN).
> 2. Click on the **{% icon galaxy-upload %} Upload data** button in the tools panel on the left-hand side.
> 3. Click on the **{% icon galaxy-wf-edit %} Paste/Fetch data** button twice. Two boxes will appear.
> 4. In the first box, under **"Name"** section, enter `Benzenesulfonyl chloride` and under **"Type"**: `smi`
> 5. Below paste the SMILES of benzenesulfonyl chloride: `C1=CC=C(C=C1)S(=O)(=O)Cl`
> 6. In the second box, under **"Name"** section, enter `Ethylamine` and under **"Type"**: `smi`
> 7. Below paste the SMILES of ethylamine: `CCN`
> 8. Click the **"Start"** button, then close the dialogue box. Your files are being added to your history!
>    
{: .hands_on}

As you learned from the previous section, there are many different formats used in computational chemistry. Of course, Galaxy allows you to interconvert between them - just have a look below!

> <hands-on-title>Convert the file format</hands-on-title>
>
> 1. {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Molecular input file"*: click on {% icon param-files %} *Multiple datasets* icon and choose both `Benzenesulfonyl chloride` and `Ethylamine`
>    - *"Output format"* - you have many options to choose from! Not only the formats but also the associated parameters. Just scroll the list and pick the one that is relevant to your downstream analysis. Here we use `MDL MOL format (sdf, mol)`
> 2. Rename {% icon galaxy-pencil %} the corresponding files `Benzenesulfonyl chloride SDF` and `Ethylamine SDF`. Before renaming the files, make sure to check either the starting dataset numbers or the atoms in the file in order not to confuse the two files! 
>
{: .hands_on}

We intentionally chose to work on benzenesulfonyl chloride and ethylamine. Do you know why? Well, if you think about the synthesis of sulfonamides, that's exactly what you need - a sulfonyl chloride and an amine. So... let's do some computational synthesis! 

> <hands-on-title>Run the reaction</hands-on-title>
>
> 1. {% tool [Reaction maker](toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_rxn_maker/ctb_im_rxn_maker/1.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `Benzenesulfonyl chloride SDF`
>    - *"Reagent options"*: Sulfonamide
>    - {% icon param-file %} *"Reagent file"*: `Ethylamine SDF`
>
{: .hands_on}

> <tip-title> Reaction maker tool works also with SMILES!</tip-title>
>
> Above we converted the *.smi* file into *.sdf* to show how {% icon tool %} **Compound conversion** tool works. However, it is worth pointing out that {% icon tool %} **Reaction maker** works also with SMILES files and the conversion happens automatically so that your *.smi* dataset is imported as *.sdf* input.
>
{: .tip} 

> <details-title>Which reactions I can run using this tool?</details-title>
>
> {% icon tool %} **Reaction maker** works with more reaction types, such as:
> - Amides synthesis
> - Nucleophilic aromatic substitution (SNAr mechanism)
> - Urea
> - Suzuki coupling
> - Sulfonamide
> - Reductive amination
> - N-alkylation
> - Ether coupling
> - Ester coupling
> - Benzimidazole
> - Triazole
> - Benzoxazole
> - Sarah Cu
> - Sarah Quat Am
>
{: .details}


The output is the *.txt* log file and the SDF file with the product molecule. If you have a look at that file, you'll see that there is just a list of atoms with their corresponding coordinates. To draw the resulting structure, we can use another tool.

> <hands-on-title>Visualisation of compounds</hands-on-title>
>
> 1. {% tool [Visualisation](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_svg_depiction/openbabel_svg_depiction/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: `SDF output for Reaction maker`
>    - *"Property to display under the molecule"* - there are many properties that you can choose from! Please note that sometimes if not enough information is provided, then the property might not be displayed. Let's choose a basic but useful property: `Molecular weight`
>    - *"Format of the resulting picture"*: `SVG`
>
{: .hands_on}

Since we spoke about the drug-likeness in the previous section, let's see how it works in practice, on the example of our "synthesised" molecule.

> <hands-on-title>Estimate the drug-likeness</hands-on-title>
>
> 1. {% tool [Drug-likeness](toolshed.g2.bx.psu.edu/repos/bgruening/qed/ctb_silicos_qed/2021.03.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecule data in SDF or SMILES format"*: `SDF output for Reaction maker`
>    - *"Method"* - two possible methods to weight the features are available - one based on max. weight (QEDw,max), and the other on mean weight (QEDw,mo). There is also an option to leave features unweighted (QEDw,u) and we'll choose this one: `unweighted (QEDw,u)`
>    - *"Include the descriptor names as header"*: {% icon param-toggle %} `Yes`
>
{: .hands_on}

> <details-title>What are the assessed properties?</details-title>
>
> The eight properties used are molecular weight (MW), octanol–water partition coefficient (ALOGP), number of hydrogen bond donors (HBDs), number of hydrogen bond acceptors (HBAs), molecular polar surface area (PSA), number of rotatable bonds (ROTBs), number of aromatic rings (AROMs) and number of structural alerts (ALERTS).
>
{: .details}

In medicinal chemistry, we often base the new structures on scaffolds of existing drugs. Therefore, it is quite useful to be able to find similar structures - below is an example of how to do it in Galaxy.

> <hands-on-title>Search ChEMBL database for similar compounds</hands-on-title>
>
> 1. {% tool [Search ChEMBL database](toolshed.g2.bx.psu.edu/repos/bgruening/chembl/chembl/0.10.1+galaxy4) %} with the following parameters:
>    - *"SMILES input type"*: `File`
>    - {% icon param-file %} *"Input file"*: `Benzenesulfonyl chloride`
>    - *"Search type"*: `Substructure`
> 2. Rename {% icon galaxy-pencil %} the output file `Benzenesulfonyl chloride substructures`
>    
{: .hands_on}


> <tip-title>Search ChEMBL database tool works also with SDF!</tip-title>
>
> Even though the input format of the above tool is SMILES, it can automatically interconvert between *.sdf* and *.smi* formats, so you don't have to use {% icon tool %} **Compound conversion** tool beforehand. 
> 
{: .tip} 

> <details-title>Parameters of Search ChEMBL database tool</details-title>
>
>  {% icon tool %} **Search ChEMBL database** tool allows for searching the ChEMBL database for compounds which resemble a SMILES string. Two search options are possible:
> - similarity (searches for compounds which are similar to the input within a specified Tanimoto cutoff)
> - substructure (searches for compounds which contain the input substructure)
>   
> Results can be filtered for compounds which are
> - approved drugs
> - biotherapeutic
> - natural products
> - fulfil all of the Lipinski rule of five criteria
>
{: .details}

Let's repeat this step, but with Lipinski's Rule of Five which you learned about in the previous section of this tutorial!

> <hands-on-title>Search ChEMBL database for similar compounds using the Lipinski rule of five</hands-on-title>
>
> 1. Click the {% icon galaxy-refresh %} **Repeat** button on the previous dataset.
> 2. You will be redirected to {% tool [Search ChEMBL database](toolshed.g2.bx.psu.edu/repos/bgruening/chembl/chembl/0.10.1+galaxy4) %}. Leave all the parameters as before, except for one:
>    - *"Filter for Lipinski's Rule of Five"*: {% icon param-toggle %} `Yes`
> 2. Rename {% icon galaxy-pencil %} the output file `Benzenesulfonyl chloride Lipinski substructures`
>
{: .hands_on}


> <question-title></question-title>
>
> How many molecules have been filtered out?
>
> > <solution-title></solution-title>
> >
> > If you click on the dataset, you will see a short summary of how many molecules have been found.
> > When we didn't apply any filters, the tool found 45 molecules. After applying Lipinski's Rule of Five filter, 36 molecules satisfied the rules. 
> > ![Image showing the preview of the output datasets from searching benzenesulfonyl chloride substructures in the ChEMBL database, with and without Lipinski's Rule of Five filter.]({% link topics/computational-chemistry/images/substructures.png %} "Preview showing the number of molecules found in the ChEMBL database, before and after applying Lipinski's Rule of Five filter")
> >
> {: .solution}
>
{: .question}


## Galaxy workflow summary

Those are just a few tools, but they offer many parameters that you can tune depending on your analysis. Additionally, we provide you with an [example history](https://usegalaxy.eu/u/j.jakiela/h/data-management-in-medicinal-chemistry-tutorial) and the dedicated [workflow](https://usegalaxy.eu/u/j.jakiela/w/data-management-in-medicinal-chemistry-workflow) of this small tutorial. You are more than welcome to explore other tools in the Galaxy Chemical Toolbox though!

# Data-driven medicinal chemistry - yesterday, today and tomorrow

## Why do we need big data in med-chem?
In the article by {% cite Lusher2014 %} the role of the medicinal chemist is to make decisions about which of the infinite possibilities of new compounds should be made next. As the amount and variety of data on which to base these decisions grows, so too must the data analysis skills of the medicinal chemist. The authors claim that modern medicinal chemists should be able to recognise sources of relevant information, prepare raw data, use statistical tools, extract meaningful information, interpret results, recognise potential problems and make visualisations to communicate their findings to improve the quality of compounds being produced in research. According to the authors, this situation will require improved education and increased access to data and information management tools. This is where Galaxy comes in, offering not only a platform for data analysis with a plethora of the most commonly used tools in the field but also educational materials allowing anyone to learn and excel in analysis. 
 
The privilege of applying computational methods in medicinal chemistry pipelines allows to focus on the most promising compounds and to remove unsuitable ones before the stage of chemical synthesis. Not only is this approach more efficient but also more sustainable and can reduce the costs of the synthetic stage of drug discovery ({% cite Brown2018 %}).

Another important role that big data plays in drug discovery is target identification (and validation) which might be currently one the biggest challenges in medicinal chemistry. By introducing genomic data in drug discovery, we can produce more specific and effective medicines. Understanding the underlying causes of the disease and the biological targets are crucial in designing new drugs and hence should be inherently linked with this process, where possible. 

## Data challenges 
It is important to understand the limitations of the databases we use since if data is not well understood, the processing and analysis drawn may ultimately be flawed, following the ‘garbage in, garbage out’ principle. Here are some challenges mentioned in {% cite Brown2018 %} that you can explore more in-depth by referring to that article.
-	Errors
-	Reproducibility
-	Standardisation
-	Formatting of data
-	Information held in silos
-	Loss of contextual information
-	Limited and biased data

## Machine learning in med-chem
Given the huge number of molecules in the various databases, applying machine learning in cheminformatics and building predictive models of the physiochemical properties of molecules is more and more popular. It helps to predict absorption, distribution, metabolism, excretion, toxicity (ADMET), likelihood of interaction with the drug targets as well as off-target effects. It can be useful in elucidating complex protein-protein or drug-drug interaction networks, in understanding structure-activity relationships (SARs) and when combined with data from the ‘omics’ revolution. This approach also helps to identify bioisosteres and introduce the idea of scaffold hopping in molecular design ({% cite Brown2018 %}).

<!---
TO FURTHER IMPROVE THE TUTORIAL:
expand this section, include examples
-->

## Knowledge sharing
It is not only about gathering and managing the data but also building a culture of knowledge sharing. We work in big, international teams with well-established methods and well-equipped labs. There are so many opportunities for knowledge capture and exchange! The challenge emerges here though – how to efficiently connect and help each other? This is the question that many pharmaceutical companies try to answer, such as Merck & Co. who describe their approach to knowledge management and report on the multiple enduring and complementary teams and initiatives to capture and share knowledge ({% cite Beshore2022 %}).

tools_json:
["MDTraj file converter", "toolshed.g2.bx.psu.edu/repos/chemteam/md_converter/md_converter/1.9.6+galaxy0", "Get PDB file", "toolshed.g2.bx.psu.edu/repos/bgruening/get_pdb/get_pdb/0.1.0", "Compound conversion", "toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0", "Reaction maker", "toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_rxn_maker/ctb_im_rxn_maker/1.1.4+galaxy0", "Visualisation", "toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_svg_depiction/openbabel_svg_depiction/3.1.1+galaxy0", "Drug-likeness", "toolshed.g2.bx.psu.edu/repos/bgruening/qed/ctb_silicos_qed/2021.03.4+galaxy0", "Search ChEMBL database", "toolshed.g2.bx.psu.edu/repos/bgruening/chembl/chembl/0.10.1+galaxy4"]


topic: fair
tutorial: bioimage-metadata
title: FAIR Bioimage Metadata
difficulty: introductory
time_estimation: 15m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["Data management", "Metadata", "Bioimaging"]
content_hash: 181e376092e598178ca87fbe67ba0298
last_modified: 2026-05-27T09:01:48.578249
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/bioimage-metadata/tutorial.html

content:
# FAIR Bioimaging

Submitting your data to a repository is a good way to make the data FAIR. This will make it:
- **F**indable, as the data will be given specific identifiers
- **A**ccessible, as the data will be available online, open and free where possible
-	**I**nteroperable, as the repository will often enforce the use of formalised, consistent language
-	**R**eusable, as the data will be released under a license with detailed provenance

# Examples of bioimage data repositories

But the question remains: where can I submit my data? Currently the main repositories where you can submit your images are: BioImage Archive, Electron Microscopy Public Image Archive (EMPIAR) and Image Data Repository (IDR). Let's have a look at the questions below to explore those repositories more in depth!

> <question-title></question-title>
>
> Listed below are three examples of Bioimage Data Repositories:
> - [IDR: Image Data Repository](https://idr.openmicroscopy.org/)
> - [EMPIAR: Electron Microscopy Public Image Archive](https://www.ebi.ac.uk/empiar/)
> - [BioImage Archive](https://www.ebi.ac.uk/bioimage-archive/)
>
> Visit their websites and find out what their scope is or what sorts of datasets they accept.
>
> > <solution-title></solution-title>
> >
> > - **IDR: Image Data Repository**: Curated datasets of cell and tissue microscopy images
> > - **EMPIAR: Electron Microscopy Public Image Archive**: Cryo-EM, Scanning Electron Microscopy, Soft X-ray tomography
> > - **BioImage Archive**: Everything else and some overlap with IDR and EMPIAR
> {: .solution}
>
{: .question}

> <tip-title>Repositories everywhere</tip-title>
>
> As well as these repositories, your Institute may have their own repository. For example, at the Warwick University, there is also [OMERO](https://warwick.ac.uk/fac/sci/med/research/biomedical/facilities/camdu/training/omero-warwick-guide_2.pdf) and [WRAP](https://wrap.warwick.ac.uk/).
>
{: .tip}


> <comment-title></comment-title>
> The repositories we are looking at in this course are for bioimage data, not medical data. There are other specialist repositories available if you have medical data.
{: .comment}

# Things to consider when choosing a repository

Now we know what repositories are available, but how to decide which one is best given the files we want submit? Try to work through the below Question box and find the answer!

> <question-title></question-title>
>
> Choose one repository from above and look through its documentation. Try to find:
> 1. What data formats are accepted?
> 2. What license is recommended to publish the data?
> 3. Are there specific instructions for large datasets?
>
> > <solution-title></solution-title>
> >
> > - **IDR: Image Data Repository**:
> >   1. The IDR uses the Bio-Formats library for reading imaging data. Bio-Formats supports over 150 proprietary and open file formats (see the [full list](https://bio-formats.readthedocs.io/en/stable/supported-formats.html)).
> >   2. It is strongly recommended that submitters make their datasets available under [CC-BY](https://creativecommons.org/licenses/by/4.0/) license.
> >   3. As specified on the [IDR website](https://idr.openmicroscopy.org/about/submission.html), dataset size is typically not an issue, but for sizes significantly larger than 1000 GB special planning may be needed.
> > - **EMPIAR: Electron Microscopy Public Image Archive**:
> >   1. Provide image data in the formats in which they are uploaded, but recommended is the use of common formats in the field including MRC, MRCS, TIFF, DM4, IMAGIC, SPIDER, MRC FEI, RAW FEI and BIG DATA VIEWER HDF5.
> >   2. All data in EMPIAR is freely and publicly available to the global community under the [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) license.
> >   3. As specified on the [EMPIAR page](https://www.ebi.ac.uk/empiar/deposition/manual/#manIntro), typically having more than 4000 files in a directory has a tendency to slow down access considerably. It is recommended in this case to sub-divide the directory into subdirectories with no more than 4000 files each. If you have a single file larger than 1 TB, contact EMPAIR in advance.
> >      To find out more, check the [FAQ page](https://www.ebi.ac.uk/empiar/faq).
> > - **BioImage Archive**:
> >   1. The BioImage Archive accepts all image data formats, although formats readable by [Bio-Formats library](https://bio-formats.readthedocs.io/en/stable/supported-formats.html), are preferable.
> >   2. According to [BioImage Archive Policies](https://www.ebi.ac.uk/bioimage-archive/help-policies/), all new data directly submitted to the BioImage Archive will be made available under a [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) licence, datasets brokered/imported from other resources may have other licenses though.
> >   3. There are different submission methods depending on data size:
> >      - Less than 50 GB total size, less than 20GB per file – use submission tool
> >      - Up to 1TB total size – use FTP
> >      - Anything larger – use Aspera
> >
> >        To find out more, check the [FAQ page](https://www.ebi.ac.uk/bioimage-archive/help-faq/).
> {: .solution}
>
{: .question}


# What metadata to collect

Whichever repository you choose, you will be required to upload some metadata along with your data. In an ideal world, you would remember everything about your data when you submit it. In reality, this is unlikely, the data could have been collected over a long time period or by different people. To overcome these challenges, it is best to collect metadata alongside imaging experiments, don’t leave it all to the end!
However, this raises further challenges. At the time of data acquisition, you probably won’t know which repository you will submit it to, what the study results will be, or even who will be the target audience for the data. So what metadata do you need to collect?

Currently, there is no standard for bioimages, so here is the general outline how to proceed:
- If you have chosen a repository, use their template/guidelines.
- Otherwise, use [REMBI](https://www.nature.com/articles/s41592-021-01166-8). These are published guidelines which are explained in detail in the [REMBI tutorial]({% link topics/fair/tutorials/bioimage-REMBI/tutorial.md %}). REMBI is useful as it should cover most of the metadata requirements of the repositories, even if you haven’t decided which one you want to use yet.
- For medical images, see the [DICOM](https://www.dicomstandard.org/) standard.

# How to store metadata

Metadata should be stored somewhere that it can be viewed and edited by collaborators. This helps everyone to stay on the same page with regard to what data should and has been collected. This is also useful if different people contribute to different aspects of the image acquisition, e.g. maybe one person prepared the sample, another imaged it under the microscope, and a third person did the post-processing. Each person can then update the metadata related to their part of the study.

Recommendations for storing metadata:
- If a repository has been chosen, use their template (if provided).
- Use a Delimited text file format, e.g. .csv, .tsv. You can use spreadsheet software and save to this format. Try to use a plain format, e.g. avoid merged/split cells.
-  Use data management software suitable for your data, e.g. [OMERO](https://www.openmicroscopy.org/omero/).

# Further steps
{% icon congratulations %} Congratulations on successfully completing this tutorial! If you want to know more about FAIR data management, we provide training that you can find in the [FAIR Data, Workflows, and Research]({% link topics/fair %}) section on GTN. For more detials on FAIR data management in bioimaging, see the [REMBI tutorial]({% link topics/fair/tutorials/bioimage-REMBI/tutorial.md %}), and if you want to dive into the imaging analysis straight away, feel free to choose one of the Galaxy [tutorials]({% link topics/imaging %})! Additionally, [Global BioImaging](https://globalbioimaging.org/) offers [training courses](https://globalbioimaging.org/international-training-courses/repository/image-data) dedicated to image data management, sharing, reuse, and image data repositories that were mentioned in this tutorial.


topic: fair
tutorial: zenodo-in-galaxy
title: Uploading Data to Zenodo from Galaxy
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["fair", "zenodo"]
content_hash: 6c1d2214b3bf56a129088a45faa83350
last_modified: 2026-05-27T09:01:48.605422
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/zenodo-in-galaxy/tutorial.html

content:
Discover a more streamlined approach to research data management with Galaxy's integration with Zenodo.

> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction

Zenodo can be connected to Galaxy, offering a streamlined experience in managing and analyzing your research data. With this integration, you can export research results directly from Galaxy to Zenodo, and import files from Zenodo into Galaxy for reproducible analysis workflows.

## Zenodo

<img style="float: right; margin-left: 1rem;" alt="Zenodo Logo" src="../../images/zenodo-gradient-logo.svg" />

[Zenodo](https://zenodo.org/) is an open repository for all scholarship, enabling researchers from all disciplines to share and preserve their research outputs, regardless of size or format. Free to upload and free to access, Zenodo makes scientific outputs of all kinds citable, shareable and discoverable for the long term.

It's worth noting that [Zenodo, in October 2023, migrated to InvenioRDM as its underlying technical platform](https://blog.zenodo.org/2023/10/13/2023-10-13-zenodo-rdm/). This move signals a commitment to enhanced features and scalability, further bolstering the integration's potential benefits for researchers.

# Getting started

## Setting up your PAT (Personal Access Token)

To be able to upload files and browse protected records, you need to create an account and set up your PAT (Personal Access Token). To create a new token:

> <hands-on-title>Create Your Personal Access Token</hands-on-title>
>
> 1. Open [Zenodo](https://zenodo.org) and `Log in` or create an account.
> 
> 2. Go to your `user settings` in the top right corner and select the `Applications` tab.
>
> 3. Then, click on `New token` and give it a name and the necessary permissions.
>
>    ![Zenodo PAT](../../images/production-zenodo-pat.png)
>
> > <comment-title></comment-title>
> > You will only be able to view and copy the access token when it is first created, so you should copy and securely store the token at this time. If you lose your copy of the access token, then you will need to generate a new one.
> >
> {: .comment}
> 
{: .hands_on}

## How to connect Zenodo to Galaxy 25.0

The new `Manage Your Repositories` section is available under `User → Preferences`. Here’s how it works:

> <hands-on-title>Connecting to Galaxy</hands-on-title>
> 
> 1. Navigate to `User → Preferences → Manage Your Repositories`. If you haven't set up any integrations yet, you'll see an empty list.
> ![Manage Your Remote File Sources](../../images/1-empty-file-sources-list.png)
>
> 2. Click `+ Create` to configure a new integration. You'll see a list of available integrations, including `S3, Dropbox, InvenioRDM, Zenodo`, and more (depending on your Galaxy server).
> ![Create New Integration](../../images/2-create-new.png)
> 
> 3. Select `Zenodo` to configure it and enter your **credentials or relevant connection details**. (This will be the access token created in the previous section if using the Zenodo sandbox.)
> ![Configure Zenodo](../../images/3-zenodo-template-filled.png)
> 
> 4. Once set up, the Zenodo integration, in this case named `My Zenodo`, will appear in the list where you can manage or delete it.
> ![Zenodo Integration](../../images/4-instance-created-zenodo.png)
> 
> 5. The Zenodo integration, along with any other integrations that you have set up, will appear first when browsing import/export locations. You can find them by clicking on the `Upload` button in the activity bar.
> ![Browse Remote Files](../../images/5-upload-import-zenodo.png)
{: .hands_on}


> <details-title>Do you want to add a different repository to Galaxy?</details-title>
> 
> There are a number of additional repository types that you can connect to Galaxy and use to import and export data.
>
> {% snippet faqs/galaxy/manage_your_repositories.md %}
>
{: .details}

# Importing records and files into Galaxy

Once you have connect Zenodo to Galaxy, you will be able to import records and files stored in Zenodo into Galaxy.

> <hands-on-title>Importing Files</hands-on-title>
> 
> 1. Open the Upload tool, and select `Choose remote files`
> ![Upload Tool](../../images/upload-tool.png)
>
> 2. Search for the Zenodo instance. Remember that this will only appear if the Zenodo plugin is configured to connect to Zenodo in your Galaxy instance.
> ![Choose Remote Files](../../images/choose-remote-files-zenodo.png)
>
{: .hands_on}

Once you have selected Zenodo, you can browse public records and import them into your Galaxy history. You can choose to import the full record or individual files in the same way you would import files from any other remote source.

# Exporting your Galaxy history

You can export your history directly to Zenodo. A benefit of publishing your history on Zenodo is that Zenodo will generate a DOI for your history. You can then include this DOI in your publication and it can be used by other researchers to cite your work.

> <hands-on-title>Exporting a History</hands-on-title>
> 
> 1. From the history menu {% icon galaxy-history-options %}
, you can select `Export History to File`
>
> 2. Then choose `to My Zenodo`. 
> 
> 3. You can decide whether to create a new record or upload the history to an existing draft record.
> ![Export History to Zenodo](../../images/export-history-zenodo.png)
>
{: .hands_on}

You can always edit the record metadata directly from the Zenodo web interface. Once you are satisfied with the record, you can publish it, generating a DOI that others can use to cite your research.

# Conclusion

In this tutorial, we have connected Zenodo to a Galaxy instance. We have also imported files and records from Zenodo into Galaxy and exported a Galaxy History to Zenodo. 

To see how to connect any InvenioRDM-compatible respository to Galaxy, see the tutorial, [Integrating InvenioRDM-compatible Repositories with Galaxy]({% link topics/fair/tutorials/inveniordm-integration/tutorial.md %}).

# References

1. [InvenioRDM and Zenodo integration in Galaxy](https://galaxyproject.org/news/2024-05-03-inveniordm-integration/)
2. [Updates to your InvenioRDM and Zenodo integration in Galaxy
   ](https://galaxyproject.org/news/2025-03-10-inveniordm-integration-update/)


topic: fair
tutorial: fair-clinical
title: Making clinical datasets FAIR
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "open", "data stewardship"]
content_hash: 4c9c75304f70a747132ee604a17c1320
last_modified: 2026-05-15T22:28:13.144020
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-clinical/tutorial.html

content:
The life science community is generally very good at sharing omics data on platforms such as [GEO](https://www.ncbi.nlm.nih.gov/geo/) and [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress).  However, the metadata and clinical data associated with the omics datasets are often incredibly sparse.  Tracking down the meta- and clinical data of omics datasets can be time-consuming for both data owner and researcher and is a significant hurdle when trying to explore enriched omics data.

In this tutorial, you will learn the benefits of making clinical datasets FAIR (Findable, Accessible, Interoperable, and Reusable) and how to make them FAIR.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Clinical datasets

> <hands-on-title>How would you describe what are clinical datasets? - {% icon time %} 2 minutes - Silent reflection</hands-on-title>
>
> * consider in general terms
> * consider more specifically
>   
{: .hands_on}

In general, the most common clinical data used in research are clinical reports (from trials and observational studies) and individual patient data.  In this section, we will explore some of the different types of clinical data.

## Metadata that accompanies transcriptomic data
In GEO and ArrayExpress the elements that we are describing as clinical data are the Characteristics or Source Characteristics.

> <hands-on-title>Explore GEO and ArrayExpress for characteristics - {% icon time %} 5 minutes</hands-on-title>
>
> 1. Click on the link below for ArrayExpress and open the section for Source Characteristics  
>    [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-12829 )
> 
> 2. Take a look at the Experimental Factors.  Notice that there are very few characteristics available for this dataset.
>
> 3. Click on the link below for GEO  
>    [GEO](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE65391)
>
> 4. Under the Samples section, click on one of the healthy controls.  Here you will see that this dataset has many characteristics.  Notice, however
>    that there is not much information gathered on the healthy controls
>
> 5. Click on one of the samples with systemic lupus erythematosus (SLE).  Notice here that there is more data collected on SLE subjects.
>
> 6. Explore ArrayExpress and/or GEO for datasets within the domain(s) that you are familiar with.  Take note of the availability of characteristic data for the different datasets
>
{: .hands_on}

## Interventional clinical trials
In these trials, usually, the participants have a condition or disease, and the research question is related to the effectiveness of a treatment.  Some of the participants are given the treatment(s) under investigation, and the other participants are given either a placebo or non-treatment drug.  Usually, these are under double-blind conditions, both participant and doctor do not know whether they are receiving/dispensing treatment or a placebo.  In addition, there may also be a set of control participants who do not have the condition or disease.  Data is usually collected at baseline or screening before any treatment or placebo is given, and then at least one time point in the future.  The breadth and type of data collected can vary across trials, and this can be limiting when trying to pool data from different trials.  The data collected could be minimal, with basic metadata to accompany the sample or imagining data.  But the data collection could also could be quite extensive, including quality of life questionnaires filled out by the patient.

An example of an interventional clinical trial is [TRACTISS](https://bmcmusculoskeletdisord.biomedcentral.com/articles/10.1186/1471-2474-15-21), where the aim of the study was to establish whether the drug rituximab improves clinical outcomes for Primary Sjögren's Syndrome patients.

## Observational clinical trials
In these trials, data is collected at least once, there may not be a follow-up collection.  The research question is usually related directly to the disease or condition, for example, understanding the biomarkers of a disease.  As with interventional clinical trials, the breadth of data collected varies on the trial.  There is often a control group but not always.  

[PRECISEADS](https://www.imi.europa.eu/projects-results/project-factsheets/precisesads) is an example of an observational clinical trial where the study's aim is molecular reclassification to find clinically useful biomarkers for systemic autoimmune diseases.


## Registries
Registries usually record (over a period of time), the disease progression of patients with the same disease or condition.  Often referred to a hub/collection point once the diagnosis is confirmed.  Patients are continually added to the registries and therefore registries can provide a large pool of patients with relatively similar data collected.  As these are registries of patients with a particular condition, there are no control participants.

Here are links to two registry projects.  Both websites describe the projects, the aims and the outputs of the studies.

[UK Sjögren's registry](https://www.sjogrensregistry.org/index.php)  
[UK JIA Biologics Registers](https://sites.manchester.ac.uk/bcrdbspar/) 

## Electronic health records (EHR)
These are generally the data collected when a person has contact with the health services. In England, this can be quite fragmented, as each service provider (i.e. GP, hospital, social care, etc.) collects data as an individual organisation, using one of a number of accredited packages for collecting and storing electronic health records.  

There are strict controls for researchers applying to access this data.  The research purpose must be approved before access is given, and the researchers usually receive data relevant to their research question.  In addition, the data received is usually anonymised to avoid the ability to identify individuals from the data.

[comment]: <> (This part may need rewording. My understanding is that EHRs are provided by: GPs where two main platforms are EMIS Web and TPP SystmOne and; NHS trusts with many different suppliers.  NHS England provides advice on accredited suppliers.)
[comment]: <> (SN - have rewritten most of the original paragraph and hope this covers the point above)

In the UK, Clinical Practice Research Datalink ([CPRD](https://cprd.com)) collects anonymised patient data from a network of GP practices links this data to a range of other health-related data and provides a longitudinal, representative UK population health dataset. 

The [Health Data Research Innovation Gateway](https://www.healthdatagateway.org/) can help researchers to find and discover UK health datasets, tools and other resources to further health research.

> <hands-on-title>Explore HDRUK Innovation Gateway for datasets - {% icon time %} 5 minutes</hands-on-title>
> 
> 1. Click on Datasets in the [Health Data Research Innovation Gateway](https://www.healthdatagateway.org/)
> 
> 2. In the filters, expand Publisher and in Search Filter type CPRD and enter  
> 
> 3. Select CPRD  
> 
> 4. Expand Phenotype and search for rheumatoid, and select Rheumatoid Arthritis  
> 
> 5. Take a look at the information about the two datasets  
> 
> 6. Clear the filters, and search for datasets related to your areas of interest  
>
{: .hands_on}

# The importance of making clinical datasets FAIR

## Associations between clinical features and omics
In some recent work in rheumatoid arthritis [The RA-MAP Consortium, 2022](https://www.nature.com/articles/s41597-022-01264-y), it was demonstrated that simple patient demographics such as sex, ethnicity and age are all drivers of expression variation in addition to disease activity.  Principal component driver plots highlighted critical associations between diverse clinical features and omics.  This shows how rich clinical information may be key to analysis in some, if not, many diseases.  

![Image of Unsupervised PCA Driver analysis of mRNA for whole blood, CD8 T-cells and CD14 Monocytes, showing that the clinical features Sex, Ethnicity and Age have some degree of association with principal components 1-5. ](../../images/RAMAP_graphic.png "Example of Unsupervised PCA Driver analysis of multi-omic compartments from the RA-MAP study, showing clinical features and their degree of association with Principal Components 1-5.  Image from: [RA-MAP, molecular immunologival landscapes in early rheumatoid arthritis and healthy vaccine recipients](https://www.nature.com/articles/s41597-022-01264-y#Tab1)")

> <hands-on-title>Do the datasets have all these characteristics - {% icon time %} 5 minutes</hands-on-title>
>
> For your domain, look at datasets on ArrayExpress and GEO and see how many of them have sex, age, ethnicity/race, and some measure of disease activity
>
{: .hands_on}

## Powering up cohorts for analyses and machine learning
For many diseases (or across a family of diseases), there is a need for well-powered cohorts with hundreds to thousands of participants, for analysis purposes.  FAIR principles have been widely promoted for omics datasets, but as shown, there are key challenges to meta-analysis.  This is illustrated by an example of collating 17 public rheumatoid arthritis studies, which was done as pre-analysis for the RA-MAP study.  The clinical data in most cases were sparse and incomplete.  For example, of the curated 17 studies, three did not include sex, 5 did not include age and only one study included ethnicity or race.  The limitations in the availability of clinical data substantially diminish the value of public data sets and on the surface to not adhere to FAIR principles.

[comment]: <> (Is there a link to the rheumatoid arthritis study example?)
[comment]: <> (SN - no link as this was pre-work before analysis, have changed the text to explain this)

Similarly, in order to make the most of machine learning techniques, datasets need to be reasonably large.  

## Time-saving
Usually, after finding potentially interesting datasets on GEO or ArrayExpress, the next step in trying to enrich these public datasets is to contact the PI of the study to request additional data.  From experience, the PI is usually happy to share additional data, especially if there have been previous collaborations.  However, even given the willingness to share, there is a time-consuming process of contacting the person who has the data and reaching agreement on what can and will be shared.  This at best will take weeks, but most likely months, especially if there is a lapse in communication between parties.

By implementing some of the techniques described in the next section can help release rich and FAIR data and metadata in a timely fashion. 

# Practical techniques to make clinical data FAIR
There are certain privacy concerns when making clinical data freely available through public libraries such as GEO and ArrayExpress. Data needs to be anonymised, or at least pseudo-anonymised.  

> <hands-on-title>What data would be problematic to share? - {% icon time %} 5 minutes</hands-on-title>  
>
> Take a look at the data below and consider which columns and data may be problematic to share  
> ![Image of a spreadsheet with the columns Variable Name, Data Item, Definition, Format, Validation Rule, Codes, Labels and Comments.  The spreadsheet gives examples of contents for each of these columns for ID, gender, age at onset, ethnicity and RF](../../images/Patient_data.png "An example of a data dictionary")
>
> > <solution-title></solution-title>  
> > * Remove columns C (FIRST), D (LAST), H (ADDRESS) and L (ZIP) as these are clear identifiers of the person  
> > * In terms of location, it is preferable to keep this as broad as possible.  In this case, keep column J (STATE) but drop columns I (CITY) and 
> > K (COUNTY).  It is debatable whether to keep them or not  
> > * Column B (BIRTHDATE) is another clear identifier but could be replaced by age  
> > * It is preferable not to keep dates (Columns (DATEOFASSESSMENT) and N (DATEOFDISEASEONSET))  
> >
> > We will discuss date handling in the next section  
> {: .solution}  
{: .hands_on}  

## Converting dates to time periods 
Avoid including dates and convert these into time periods.  For example, use date of birth and date of assessment to calculate Age (at assessment or baseline).  Similarly, calculate Age at Diagnosis, and Age at Onset, if dates are given.

If there are more than one visits, and if these are not within roughly defined time periods, for example, 4 weeks, or 12 weeks, calculate the number of days between the visits.

> <hands-on-title>Which columns would you change to time periods from dates? - {% icon time %} 10 minutes</hands-on-title>
> 
>  For the dataset below identify the column you would change to time periods and convert them to time periods
> ![Image of spreadsheet with columns ID, Birth Date, Race, Ethnicity, Gender, State, Date of Assessment and Date of Disease Onset as the columns, with 10 lines of data filled in below for each of these columns](../../images/Patient_dates.png "Patient data with dates")
>
> > <solution-title></solution-title>  
> > * Replace BIRTHDATE with AGE at DATEOFASSESSMENT  
> > * Replace DATEOFDISEASEONSET with AGEATONSET 
> > * Remove BIRTHDATE, DATEOFASSESSMENT and DATEOFDISEASEONSET
> > 
> > ![Image of spreadsheet with columns ID, Race, Ethnicity, Gender, State, Age and Age at Onset, with 10 lines of data filled in below for each of these columns](../../images/Datesremoved.png "Patient data with dates removed")  
> >
> {: .solution}  
{: .hands_on}  

## Comprehensive metadata for omics datasets
When publishing omics datasets to libraries such as GEO and ArrayExpress, consider at least including Age, Sex, Race/Ethnicity, and if possible some disease activity measure.  Publishing as much as possible is great for researchers to access enriched datasets, and also reduces fielding queries and requests to the PI.

## Data dictionaries
There are cases when PIs would rather not have the clinical datasets on free-to-access platforms.  In these situations, the clinical datasets can still be made FAIR.  By having clear documentation of the datasets, these documents can be published and attributed to the project or consortia.  For each individual clinical dataset, it is useful to have a data dictionary.

A data dictionary is the metadata of the dataset.  It should include details such as data type, value range or allowed values, any relationship to other data elements and their meaning or purpose.  The details of the data dictionary often originate from how the data was input either at the source or into the database.

Here’s an example of a data dictionary used on a project

![Image of an example of a data dictionary in a spreadsheet format](../../images/ExampleDataDict1.png "Example of a data dictionary")

> <tip-title>Key things to note</tip-title>  
>
> * The variable name is often the short form name of the data item  
> * The definition describes the data item more fully  
> * In the data dictionary example shown above, data items are encoded, and so their format is numeric, with constraints
> * The validation rule usually refers to instructions given when data was originally input, but it could be the calculation formula for the data item  
> * Codes and Labels here are the encoding definitions  
{: .tip}

> <hands-on-title>Create a data dictionary for the dataset below - {% icon time %} 10 minutes</hands-on-title>
>
> ![Image of a spreadsheet with columns ID, Race, Ethnicity, Gender, Age and Age at Onset, with 10 rows filled in below](../../images/DataDictionaryHandsOn.png "Create a data dictionary for this dataset")
>
> > <solution-title></solution-title>
> > Here is one example of a suitable data dictionary for the above dataset
> >
> > ![Image of a spreadsheet with the columns Variable Name, Data Item, Definition, Format, Validation Rule, Codes, Labels and Comments.  The spreadsheet gives examples of contents for each of these columns based on the previous image, for ID, race, ethnicity, gender, age and age at onset](../../images/DataDictSolution.png "Example of a data dictionary for the above dataset")
> >
> {: .solution}  
{: .hands_on}  

A key benefit of data dictionaries is that the information captured is generally not confidential or patient-sensitive and, therefore, there are few if any restrictions to sharing them.

## Data catalogues
Data catalogues are useful for teams, projects or consortia where there are many datasets being used and generated by the group.  Data catalogues are the metadata for the individual datasets, and they can provide context and provenance for each dataset within the group of datasets.

Here is an example of a data catalogue

![Image of a spreadsheet example of a data catalogue](../../images/DataCat.png "Example of a data catalogue")

You may not want to capture all these metadata, and you may want to include other items such as phenotype, links to any publications related to the dataset, and links to the published dataset.

Additionally, if there are complex data-sharing agreements, this information could be added to the catalogue (for internal use).

When projects and consortia are live, the corresponding data catalogue is a work in progress alongside the project and needs to be updated as required.  Although data catalogues can be somewhat time-consuming to set up, this is more than made up with the time saved looking for details about datasets, especially as people switch in and out of the team.

As with data dictionaries, data catalogues rarely capture any information that needs to be restricted access and usually can be shared in full.  The exception potentially may be the contact person's email address.  

More information about data catalogues can be found here  
 [Using a data catalogue within a research team](https://tess.elixir-europe.org/materials/rdmbites-using-a-data-catalogue-within-a-research-team)

## Publishing on Zenodo
Data dictionaries and data catalogues include a wealth of information about the clinical datasets they reference, and in general, contain no sensitive or confidential data.  [Zenodo](https://www.zenodo.org) is an open repository for research outcomes hosted by CERN.  It is free to use and each upload is citeable through a Digital Object Identifier (DOI).  Uploads are instantly available, with version control features.  Publishing on platforms such as Zenodo can help make your data findable.

Here, you can see published data dictionaries for a consortium
![Image of a screenshot of the CLUSTER Consortium page on the Zenodo website.  There are two entries for data dictionaries of datasets](../../images/CLUSTER_Zenodo.png "Screenshot of CLUSTER Consortium community page on Zenodo website")

To upload, create a login using GitHub ID, ORCID, or email.  Click on Upload (at the top of the page, left of centre), and follow the instructions.

## Managing Data Access Requests
Regardless of whether or not you publish data dictionaries and data catalogues, it is worth considering the mechanisms when receiving data access requests.  

When a project is ongoing, if there is a Data Management Committee (DMC), the responsibility for data access requests would be with the DMC.  It is really helpful to have a centralised point to provide data governance including guidelines for data sharing.

It is really worthwhile clarifying the process of handling data access requests to minimise effort and delays for both the data holder and the data requester.  This process may need to be revised as the project ends.

> <tip-title>Points to consider when handling data access requests</tip-title>  
> * Who will be the point of contact and the email address (e.g. create a general project email that is handled by the project administration team)
> * Expected length of time before requester receives a response
> * Involve the institution's legal team 
>   - Terms and conditions of use of data and/or samples
>   - Publication guidelines
> * Standardise the process with a Data Access Request Form  
{: .tip}

An example of a data access process is shown below
![Image of a four step process to request for data access.  Step 1 email the project.  Step 2 Discuss with the project what data or samples are being requested.  Step 3 approval by relevant committees.  Step 4 Decision communicated to requestor](../../images/DataAccessProcess.png "Example of a data access process")

This is an example of a [data and sample access request form](https://zenodo.org/record/8344400) that you might find useful as a starting point to create one for your project.

In the UK, for members of the UK Health Data Research Alliance, the HDRUK Innovation Gateway may also be a way to manage the data access request process.  This requires the metadata of the health datasets to be loaded to the Gateway web portal.  Researchers with an account can then search the portal and request access to datasets. The request access process and also creating collections of metadata are described in more detail in the [Data Custodian Playbook](https://discourse.healthdatagateway.org/t/data-custodian-playbook/980)

This work was funded by the ELIXIR-UK: FAIR Data Stewardship training UKRI award (MR/V038966/1).

[comment]: <> (need to also include acknowledgement of CLUSTER funding)


topic: fair
tutorial: fair-data-registration
title: Data Registration
difficulty: intermediate
time_estimation: 40M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 2ba6ea4c1d34b908c8c7cb92e7c91820
last_modified: 2026-05-15T22:28:13.144628
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-data-registration/tutorial.html

content:
The concept of data registration is defined as well as ways in which data registration can be achieved.  Learners will be able to describe why indexed data repositories are important as well as resources enabling you to choose a searchable repository.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}




# Data registration and the FAIR Principles
Data registration relates to the following 3 FAIR Principles (Table 3.1).  
We will discuss and signpost these in this Episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. **(meta)data are assigned a globally unique and persistent identifier**<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes<br>F4. **(meta)data are registered or indexed in a searchable resource**                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary<br> A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. **(meta)data are released with a clear and accessible data usage license**<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        

Table 3.1: The 15 FAIR Guiding Principles. Principles relating to data registration in **black**.

# What is data deposition and registration?

Data deposition and registration refer to the process of uploading data to a searchable resource and providing appropriate metadata to facilitate its discoverability.
For example, a data repository, where data and metadata can be uploaded, may enable it to be discovered, preserved and accessed.  Here we use the general term data repository to describe any online storage location that can host deposited (meta)data.

In the context of FAIR, data deposition relates to a number of the Guiding Principles. Firstly,  _“(meta)data are registered or indexed in a searchable resource, Indexed in a searchable resource: a resource where (meta)data are organised so that they can be queried based on defined fields, ”_ (FAIR Principle F4).  Searchable (indexed) metadata enables humans and computers to query and discover data of interest, though this depends on what is indexed.  Here, indexing refers to a process that occurs within the architecture of the data repository (local indexing) where metadata are organised so that they can be queried based on a defined field.  It is worth noting that community resources focused on a particular domain (for example, the human database in [Ensembl](https://www.ensembl.org/Homo_sapiens/Info/Index)) are better indexed for a particular community, rather than generic repositories (for example, [Zenodo](https://zenodo.org/)) which may not index the community specific components, and may focus on higher level metadata.  Indexing by an internet search engine is another example of this.  Google (and other search engines, such as Yahoo and Yandex) have an agreed vocabulary ([schema.org](https://schema.org/)), within web pages, that are ‘scraped’ and indexed.  While the focus of this vocabulary was originally intended for commercial products, community-specific efforts to facilitate discipline-specific indexing are underway (for example, [Bioschemas](https://faircookbook.elixir-europe.org/content/recipes/findability/seo/bioschemas-data-page.html)). 

# Why should I upload my data to a data repository?

Data repositories are generally preferred to file storage systems (such as Dropbox) or sharing data on an ad hoc basis since they often better support FAIR best practice.  Repositories will assign citable, _“globally unique and persistent identifiers”_ (FAIR Principle F1) to data, and in some cases enable a data submitter to apply a data usage licence through association with the resource (FAIR Principle R1.1).  

Although not exclusively, data repositories support the creation of metadata through curation interfaces providing drop-downs and text fields for metadata entry and validation.  Often in the case of a domain or data-specific data repository, such as BioStudies shown in the previous Episode, drop-downs for metadata curation will link community-endorsed vocabularies (FAIR Principle R1.3).

# Types of data repository

General public data repositories, such as [Zenodo](https://zenodo.org/), are multidisciplinary and permit registration and upload of open and closed access (meta)data.  Metadata curation is relatively high level and made searchable via indexing.  Relating to data in the Life Sciences, Zenodo is often used to publish and provide citable URLs to supplementary data within articles, usually in instances where a domain repository does not exist.

Institutional repositories work similarly and provide an online archive for hosting, indexing and preserving research output specific to an institution.  Typically these house more than data, providing a repository often for documents and articles.  Institutions will have their own systems supported locally or buy into company solutions. 

Discipline-specific repositories cater for communities and datatypes, and typically provide web interfaces to annotate rich metadata at the point when data are submitted.  Examples of these belong to the suite of data repositories at the [European Bioinformatics Institute (EBI)](https://www.ebi.ac.uk/submission/) where rich metadata creation is supported by teams of curators.


> <question-title></question-title>
>
> An example of a discipline-specific repository is [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress) database.  ArrayExpress stores data from high-through functional genomics assays, such as RNAseq, ChIPseq and expression microarrays.
> The data submission interface of ArrayExpress is called [Annotare](https://www.ebi.ac.uk/fg/annotare/login/).  Without creating a login, what help is given to a person looking to submit a dataset for the first time?
>
> > <solution-title></solution-title>
> >
> > Both a [submission guide](https://www.ebi.ac.uk/fg/annotare/help/index.html) and [YouTube video](https://www.youtube.com/watch?v=ANr2PTVy7JA) is provided.
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> Finding more help on how to upload data to specific repositories
> The [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html) is an online open resource housing specific ‘how to’ guides or recipes.  Use the FAIR Cookbook to find two recipes for “depositing data to Zenodo” and “registering datasets with Wikidata”, respectively.
>
> > <solution-title></solution-title>
> >
> > Open the **Findability** pulldown on the left hand banner to find recipes for the following:
> > [Depositing to generic repositories - Zenodo use case](https://faircookbook.elixir-europe.org/content/recipes/findability/zenodo-deposition.html) and [Registering Datasets in Wikidata](https://faircookbook.elixir-europe.org/content/recipes/findability/registeringDatasets.html).
> >
> {: .solution}
{: .question}




> <question-title></question-title>
>
> Choosing the right data repository for your data
> [FAIRsharing](https://fairsharing.org/) helps researchers identify suitable data repositories, standards and policies relating to their data.  Use this resource to identify data repositories for proteomic data.
>
> > <solution-title></solution-title>
> >
> > Access the search bar for the [FAIRsharing database registry](https://fairsharing.org/search?fairsharingRegistry=Database).  Search for proteomics and select “repository” under “Record Type”.
> > ![A screenshot of FAIRsharing showing the results for a search for all proteomics repositories.](../../images/figure3-1-fairsharing.png "FAIRsharing allows you to search for specific record types that are relevant for your area.")
> >
> {: .solution}
{: .question}


# Useful Resources
- Registries and lists of public repositories: [FAIR Cookbook](https://fairsharing.org/search?page=1&recordType=repository) and [nature](https://www.nature.com/sdata/policies/repositories) journal
- Publishing your data: [RDMkit](https://rdmkit.elixir-europe.org/data_publication)
- Using Bioschemas to embed metadata into webpages: [FAIR Cookbook Bioschemas](https://faircookbook.elixir-europe.org/content/recipes/findability/seo/bioschemas-datacatalog.html)


topic: fair
tutorial: ro-crate-intro
title: RO-Crate - Introduction
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: ["type: none", "type: none", "type: external", "type: \"internal\""]
tags_json: []
content_hash: 92e85555d9c4c4ab67f9f9fbcbfc9b92
last_modified: 2026-05-27T09:01:48.603437
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-intro/tutorial.html

content:
This tutorial assumes you have already completed [An overview of the RO-Crate concept and its implementations](https://gallantries.github.io/video-library/videos/ro-crates/intro/slides/) and have a basic understanding of working with JSON.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



In this tutorial, meant to be read along with the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.1/), we'll walk through the initial steps for creating a basic RO-Crate. You are invited to replicate the below steps on your local computer.


## Making a folder into an RO-Crate

In the simplest form, to describe some data on disk, an _RO-Crate Metadata File_ is placed in a folder alongside a set of files or folders.

First create a new folder `crate1/` and add a single file `data.csv` to represent our dataset:

```
"Date","Minimum temperature (°C)","Maximum temperature (°C)","Rainfall (mm)"
2022-02-01,16.0,28.4,0.6
2022-02-02,16.3,17.2,12.4
```

Next, to turn this folder into an RO-Crate we need to add the _RO-Crate Metadata File_, which has a fixed filename.
Create the file `ro-crate-metadata.json` using [Visual Studio Code](https://code.visualstudio.com/) or your favourite editor, then add the following JSON:

```json
{
  "@context": "https://w3id.org/ro/crate/1.1/context",
  "@graph": [

  ]
}
```

Your folder should now look like this:

![Folder listing of crate1, including data.csv and ro-crate-metadata.json](../../images/ro-crate-intro/crate1-folders.svg "Any folder can be made into an RO-Crate by adding <code>ro-crate-metadata.json</code>")

The presence of the reserved `ro-crate-metadata.json` filename means that `crate1` (and its children) can now be considered to be an **RO-Crate**. We call the top-level folder of the crate for the **RO-Crate Root** and can now refer to its content with relative file paths.

We also need to make some declaration within the JSON file to turn it into a valid _RO-Crate Metadata Document_, explained in the next session.


## JSON-LD preamble

The preamble of `@context` and `@graph` are JSON-LD structures that help provide global identifiers to the JSON keys and types used in the rest of the RO-Crate document. These will largely map to definitions in the [schema.org](http://schema.org/) vocabulary, which can be used by RO-Crate extensions to provide additional metadata beyond the RO-Crate specifications. It is this feature of JSON-LD that helps make RO-Crate extensible for many different purposes -- this is explored further in the specification's [appendix on JSON-LD](https://www.researchobject.org/ro-crate/1.1/appendix/jsonld.html). In short, only JSON keys (_properties_) and types defined this way can be used within the RO-Crate Metadata Document.

However, in the general case it should be sufficient to follow the RO-Crate JSON examples directly without deeper JSON-LD understanding. The RO-Crate Metadata Document contains a flat list of _entities_ as JSON objects in the `@graph` array. These entities are cross-referenced using `@id` identifiers, rather than being deeply nested. This is one major difference from JSON structures you may have experienced before.  The `@type` keyword associates an object with a predefined type from the JSON-LD context. Almost any property can alternatively be used with an `[]` array to provide multiple values.

The rest of this tutorial, and indeed most of the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.1/), specify which entities can be added to the `@graph` array.


## RO-Crate Metadata descriptor

The first JSON-LD _entity_ to add has the `@id` value of `ro-crate-metadata.json` to describe the JSON file itself:


```json
{
    "@id": "ro-crate-metadata.json",
    "@type": "CreativeWork",
    "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},
    "about": {"@id": "./"}
}
```

This required entity, known as the [RO-Crate Metadata Descriptor](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#ro-crate-metadata-file-descriptor), helps this file self-identify as an RO-Crate Metadata Document, which is conforming to (`conformsTo`) the RO-Crate specification version 1.1. Notice that the `conformsTo` URL corresponds to the `@context` URL version-wise, but they have two different functions. The context brings the defined terms into the metadata document, while the conformance declares which RO-Crate conventions of using those terms are being followed.


> <tip-title>RO-Crate versions</tip-title>
> This tutorial is written for RO-Crate 1.1, the RO-Crate website will list the [current specification version](https://www.researchobject.org/ro-crate/specification.html) -- RO-Crates can generally be upgraded to newer versions following [semantic versioning](https://semver.org/) conventions, but check the [change log](https://www.researchobject.org/ro-crate/1.1/appendix/changelog.html) for any important changes. The next development version of the specification, indicated with a `-DRAFT` status, may still be subject to changes and should only be used with caution.
{: .tip}


## RO-Crate Root

Next we'll add another entity to the `@graph` array, to describe the [RO-Crate Root](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#direct-properties-of-the-root-data-entity):


```json
{
    "@id": "./",
    "@type": "Dataset",
    "hasPart": [

    ]
}
```

By convention, in RO-Crate the `@id` value of  `./` means that this entity describes the folder in which the RO-Crate metadata file is located. This reference from `ro-crate-metadata.json` is therefore semantically marking the `crate1` folder as being the RO-Crate Root.


{: .note }
This example is a folder-based RO-Crate stored on disk, and therefore absolute paths are avoided, e.g. in case the root folder is moved or archived as a ZIP file. If the crate is being served from a Web service, such as a data repository or database where files are not organized in folders, then the `@id` might be an absolute URI instead of `./` -- this is one reason why we point to the root entity from the metadata descriptor, see section [Root Data Entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html) for details.

## Describing the root entity

When describing the [root entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#direct-properties-of-the-root-data-entity), the properties generally apply to the whole of the crate. For instance it is a good idea to give a description of why these resources are gathered in a crate, as well as giving the crate a name and license for FAIR reuse and citation.

> <question-title></question-title>
>
> Try to add the `name`, `description` and `datePublished` properties, and for `license` as a cross-reference, use [SPDX](https://spdx.org/licenses/) license list to find the identifier for Creative Commons Zero or another license of your choice:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@id": "./",
> >   "@type": "Dataset",
> >   "hasPart": [ ],
> >   "name": "Example crate",
> >   "description": "I created this example by following the tutorial",
> >   "datePublished": "2023-05-22T12:03:00+0100",
> >   "license": { "@id": "http://spdx.org/licenses/CC0-1.0"}
> > }
> > ```
> {: .solution}
{: .question}


> <tip-title>License identifiers</tip-title>
> In the above solution, the identifier for CC0-1.0 <http://spdx.org/licenses/CC0-1.0> is slightly
> different from their listed web page URI <https://spdx.org/licenses/CC0-1.0.html> -- the former
> is chosen to align with [SPDX JSON-LD identifiers](https://github.com/spdx/license-list-data/tree/main/jsonld),
> which unfortunately are not shown directly on their website as _permalinks_.
> It is not a requirement in RO-Crate to use permalinks for `@id` of entities like licenses,
> it is nevertheless best practice to propagate permalinks where known.
{: .tip}

> <tip-title>Choosing a license</tip-title>
> Choosing a license appropriate for your dataset can be non-trivial, particularly if third-party data/software and multiple organizations are involved. See [FAIR Cookbook on licensing](https://faircookbook.elixir-europe.org/content/recipes/reusability/ATI-licensing.html). It is worth noting that an RO-Crate permits data entities to have a `license` different from the overall Crate license. It is still recommended to choose an overall Crate license that can legally apply across all the content in the RO-Crate Root.
{: .tip}


## About cross-references

In a RO-Crate Metadata Document, entities are cross-referenced using `@id` reference objects, rather than using deeply nested JSON objects.
In short, this _flattened JSON-LD_ style (shown below) allows any entity to reference any other entity, and RO-Crate consumers can directly find all the descriptions of a given entity as a single JSON object.


![JSON block with id `ro-crate-metadata.json` has some attributes, `conformsTo` RO-Crate 1.2, and `about` referencing id `./`. In second JSON block with id <code>./</code> we see additional attributes such as its name and description.](../../images/ro-crate-intro/introduction-figure-1.svg "showing RO-Crate Metadata descriptor's <code>about</code> property pointing at the RO-Crate Root entity with matching <code>@id</code>")


> <question-title></question-title>
>
> Consider the root Data Entity `./`, and add such a cross-reference to the file `data.csv` using the _property_ called `hasPart`:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "./",
> >     "@type": "Dataset",
> >     "hasPart": [
> >       {"@id": "data.csv"}
> >     ],
> >     "…": "…"
> > }
> > ```
> {: .solution}
{: .question}


The RO-Crate root is always typed `Dataset`, though `@type` may in some cases have additional types by using a JSON array instead of a single value.  Most entities can have such more specific types, e.g. chosen from [schema.org type list](https://schema.org/docs/full.html).

> <question-title></question-title>
> 1. Navigate the schema.org type list to find a subtype of `CreativeWork` that is suitable for a learning resource..
> 2. Modify the root entity's `@type` to be an array.
> 3. Add the type name for learning resource at the end of the array.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "./",
> >     "@type": ["Dataset", "LearningResource"],
> >     "hasPart": [
> >       {"@id": "data.csv"}
> >     ],
> >     "…": "…"
> > }
> > ```
> {: .solution}
{: .question}

The root has several metadata properties that describe the RO-Crate as a whole, considering it as a Research Object of collected resources. The section on [root data entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html) details further the required and recommended properties of the root `./`.




## Data entities

A main type of resources collected in a Research Object is _data_ -- simplifying, we can consider data as any kind of file that can be opened in other programs. These are aggregated by the Root Dataset with the `hasPart` property. In this example we have an array with a single value, a reference to the entity describing the file `data.csv`.

> <tip-title>Referencing external resources</tip-title>
> RO-Crates can also contain data entities that are folders and Web resources, as well as non-File data like online databases -- see section on [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html).
{: .tip}

We should now be able to follow the `@id` reference for the corresponding _data entity_ JSON block for our CSV file, which we need to add to the `@graph` of the RO-Crate Metadata Document.


> <question-title></question-title>
>
> 1. Add a declaration for the CSV file as new entity with `@type` declared as `File`.
> 2. Give the file a human-readable `name` and `description` to detail it as _Rainfall data for Katoomba in NSW Australia, captured February 2022_.
> 3. To add this is a CSV file, declare the `encodingFormat` as the appropriate [IANA media type](https://www.iana.org/assignments/media-types/#text) string.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "data.csv",
> >     "@type": "File",
> >     "name": "Rainfall Katoomba 2022-02",
> >     "description": "Rainfall data for Katoomba, NSW Australia February 2022",
> >     "encodingFormat": "text/csv"
> > },
> > ```
> >
> {: .solution}
{: .question}


It is recommended that every entity has a human-readable `name`; as shown in the above example, this does not need to match the filename/identifier. The `encodingFormat` indicates the media file type so that consumers of the crate can open `data.csv` in an appropriate program, and can be particularly important for less common file extensions frequently encounted in outputs from research software and instruments.

For more information on describing files and folders, including their recommended and required attributes, see section on [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html).

> <question-title></question-title>
>
> 1. Consider if the file content of `data.csv` is not covered by our overall license (CC0), but [Creative Commons BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (which only permits non-commercial use)
> 2. To override, add an  `license` cross-reference property on this particular data entity
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "data.csv",
> >     "@type": "File",
> >     "name": "Rainfall Katoomba 2022-02",
> >     "description": "Rainfall data for Katoomba, NSW Australia February 2022",
> >     "encodingFormat": "text/csv",
> >     "license": { "@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/" }
> > },
> > ```
> {: .solution}
{: .question}



## Contextual entities

Entities that we have added under `hasPart` are considered _data entities_, while entities only referenced from those are considered _contextual entities_ -- they help explain the crate and its content.

> <tip-title>Adding entities to the JSON array</tip-title>
> You may have noticed we're adding incrementally to the `@graph` array. It is important to remember the `,` between each entity, **except** for the final entity in the JSON array; and likewise for the properties within the JSON object for each entity. This is an artefact of the strict [JSON](https://www.json.org/) file format rules to simplify parsing. The order of the entities within the `@graph` JSON-LD array and the order of the keys within a JSON object is _not significant_. The _graph_ content is given by the `@id` cross-references.
{: .tip}

You may notice the subtle difference between a _data entity_ that is conceptually part of the RO-Crate and is file-like (containing bytes), while a _contextual entity_ is a representation of a real-life organization that can't be downloaded: following the URL, we would only get its _description_. The section [contextual entities](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html) explores several of the entities that can be added to the RO-Crate to provide it with a **context**, for instance how to link to authors and their affiliations.  Simplifying slightly, a data entity is referenced from `hasPart` in a `Dataset`, while a contextual entity is referenced using any other defined property.

## Detailing licenses

We have previously declared two different `license` cross-references. While following the URLs in this case explain the licenses well, it is also best practice to include a very brief summary of contextual entities in the RO-Crate Metadata Document. This is more important if the cross-reference do not use a permalink and may change over time. As a minimum, each referenced entity should have a `@type` and `name` property. It is also possible to add `url` for more information

> <question-title></question-title>
>
> Add a contextual entity for each of the two licenses, see the [licensing](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html#licensing-access-control-and-copyright) section for details:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
> >     "@type": "CreativeWork",
> >     "name": "CC BY-NC-SA 4.0 International",
> >     "description": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International"
> > },
> > {
> >     "@id": "http://spdx.org/licenses/CC0-1.0",
> >     "@type": "CreativeWork",
> >     "name": "CC0-1.0",
> >     "description": "Creative Commons Zero v1.0 Universal",
> >     "url": "https://creativecommons.org/publicdomain/zero/1.0/"
> > },
> > ```
> {: .solution}
{: .question}


An additional exercise is to try to unify the two entites so that both use spdx identifiers, remembering to update the corresponding `license` cross-references when changing the `@id`. However, not all licenses have a direct SPDX identifier.


## Author

Moving back to the RO-Crate root `./`, let's specify who are the authors of the crate.

> <question-title></question-title>
>
> 1. Add yourself as an [`author`](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html#people) of the crate using the type `Person`
> 2. Include your preferred name.
> 3. If you don't have an [ORCID](https://orcid.org/), you may use either the URL of your main home page at your institution, or a crate-local identifier like `#alice`.
> 4. Include your `affiliation` as a string value.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@id": "./",
> >   "@type": "Dataset",
> >   "author": {"@id": "https://orcid.org/0000-0002-1825-0097"},
> >   "…": "…"
> > },
> > {
> >   "@id": "https://orcid.org/0000-0002-1825-0097",
> >   "@type": "Person",
> >   "name": "Josiah Carberry",
> >   "affiliation": "Brown University"
> > }
> > ```
> {: .solution}
{: .question}


> <tip-title>Who can be authors of an RO-Crate?</tip-title>
> When we say someone is an author of a crate, it means they have contributed something substansively to its content (typically the data). Agreement on what is considered authorship on a dataset can be tricky; you may decide some people would be better represented as `contributor`. One advantage of RO-Crate is that authorship can be declared explicitly also on each data entity, so it can be clearer where each person have contributed (e.g. a statistician is author of an R script).  This means that generally the authors of the crate can be a broader, more inclusive list than perhaps traditionally recognized as academic authorship.
{: .tip}

> <question-title></question-title>
>
> 1. "Unroll" your `affiliation` of the person as cross-reference to another contextual entity, typed as an `Organization`.
> 2. You can use [ROR](https://ror.org/) to find an identifier for most educational/research institutions, or you can use the main web page of your organization as its `@id`.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@id": "https://orcid.org/0000-0002-1825-0097",
> >   "@type": "Person",
> >   "name": "Josiah Carberry"
> > },
> > {
> >   "@id": "https://ror.org/05gq02987",
> >   "@type": "Organization",
> >   "name": "Brown University",
> >   "url": "http://www.brown.edu/"
> > }
> > ```
> {: .solution}
{: .question}



The reuse of existing identifiers is important for both persons and organization from a FAIR perspective, as their names may not be globally unique.

> <question-title></question-title>
>
> 1. Now imagine you are going to publish the RO-Crate on your institution's web pages.
> 2. Cross-reference the same Organization entity with `publisher` from the RO-Crate Root entitity:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "./",
> >     "@type": "Dataset",
> >     "publisher": {"@id": "https://ror.org/05gq02987"},
> >     "…": "…"
> > }
> > ```
> {: .solution}
{: .question}


## Validating and visualizing

As we made this RO-Crate Metadata File by hand, it's good to check for any JSON errors, such as missing/extra `,` or unclosed `"` quotes. Try pasting the file content into the [JSON-LD Playground](https://json-ld.org/playground/). It should show up any errors, for example:

```
JSON markup - SyntaxError: JSON.parse: expected `','` or `']'` after array element
at line 29 column 5 of the JSON data
```

Modify the JSON file in your editor to fix any such errors. You can also use editor commands such as _Format Document_ to ensure you have consistent spacing, indentation and brackets.

If the document passes without errors in the JSON-LD Playground, you should see output under _Expanded_ looking something like:

```json
[
  {
    "@id": "ro-crate-metadata.json",
    "@type": [
      "http://schema.org/CreativeWork"
    ],
    "http://schema.org/about": [
      {
        "@id": "./"
      }
    ],
    "http://purl.org/dc/terms/conformsTo": [
      {
        "@id": "https://w3id.org/ro/crate/1.1"
      }
    ]
  },
  {
    "@id": "./",
    "@type": [
      "http://schema.org/Dataset"
    ],
    "…": "…"
  }
]
```

This verbose listing of the JSON-LD shows how the `@context` has correctly expanded the keys, but is not particularly readable. Try the _Visualized_ tab to see an interactive rendering of the entities:

![Visualized in the JSON-LD Playground](../../images/ro-crate-intro/jsonld-playground-visualized.png)

As the RO-Crate Metadata Document is valid JSON-LD it is also possible to process it using Linked Data technologies such as triple stores and SPARQL queries. It is beyond the scope of this tutorial to explain this aspect fully, but interested readers should consider how to [handle relative URI references](https://www.researchobject.org/ro-crate/1.1/appendix/relative-uris.html).
As an example, try the _Table_ button and notice that the entities with relative identifiers are not included. This is because when converting to RDF you need absolute URIs which do not readily exist when a crate is stored on disk, we've not decided where the crate is to be published yet.

## Advanced: Converting JSON-LD to triples

Try modify the graph's `@context` within the [JSON-LD Playground](https://json-ld.org/playground/) (don't modify the `ro-crate-metadata.json` on disk), and revisit the _Table_ rendering.

```json
{
  "@context": [
    "https://w3id.org/ro/crate/1.1/context",
    { "@base": "arcp://uuid,deffa754-c764-4e04-aabf-e600c6200553/" }
  ],
  "…": "…"
}
```

![Triples table in the JSON-LD Playground](../../images/ro-crate-intro/jsonld-playground-table.png)

Above `arcp://uuid,deffa754-c764-4e04-aabf-e600c6200553/` is a randomly generated identifier to represent the RO-Crate root, and now the JSON-LD Playground can show all the triples from the metadata file. You can likewise use the _N-Quads_ button to convert the metadata file to the [RDF N-Quads](http://www.w3.org/TR/n-quads/) format. Most RDF libraries and stores have JSON-LD support, but may need to specify a base URI as we did above, making a new UUID for each imported RO-Crate.


## HTML preview

An RO-Crate can be distributed on disk, in a packaged format such as a zip file or disk image, or placed on a static website. In any of these cases, an RO-Crate can have an accompanying HTML version (`ro-crate-metadata.html`) designed to be human-readable.

![Example dataset for RO-Crate specification](../../images/ro-crate-intro/ro-crate-preview-example.png)

> <question-title></question-title>
>
> Try navigating the [preview of the running example](rainfall-1.2.1/ro-crate-preview.html) and find:
>
> 1. What is the license of the rainfall CSV?
> 2. What is the affiliation of the crate's author?
> 3. What does the Validity Check inspect
> 4. What is not covered by this check?
>
> > <solution-title></solution-title>
> > 1. CC BY-NC-SA 4.0 International
> > 2. Brown University
> > 3. The context, and for root dataset: existance, valid identifier, name, description, license and date published.
> > 4. The other entities were not checked, e.g. the `affiliation` of the author.
> {: .solution}
{: .question}


## Advanced: Generating HTML preview

If you have [Node JS](https://nodejs.org/) installed, you can install and try the [ro-crate-html](https://www.npmjs.com/package/ro-crate-html) tool on your `crate1` folder:

```bash
npm i -g ro-crate-html
rochtml crate1/ro-crate-metadata.json
```

Or if you have [Docker](https://www.docker.com/), something like:

```bash
docker pull simleo/rochtml
docker run -v $(pwd):/crate -it simleo/rochtml -c ro-crate-preview_files /crate/ro-crate-metadata.json
```

The above will generate a `ro-crate-preview.html` file within your RO-Crate Root. Experiment with modifying the Metadata file and re-generating the HTML preview to see how your rendering differs from the [running example](rainfall-1.2.1/ro-crate-preview.html).

## Next steps

You have completed making a basic RO-Crate. You may try any of the following:

- Add and describe additional data entities, adding additional properties for [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html)
- Follow the RO-Crate specification for additional [contextual entities](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html) you can add to the crate
- Try briefly describing [provenance](https://www.researchobject.org/ro-crate/1.1/provenance.html) or [software](https://www.researchobject.org/ro-crate/1.1/workflows.html) for any additional data entities you have added.

> <question-title>Complete RO-Crate Metadata Document</question-title>
>
> The final RO-Crate Metadata Document constructed in this tutorial should look something like:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@context": "https://w3id.org/ro/crate/1.1/context",
> >   "@graph": [
> >     {
> >       "@id": "ro-crate-metadata.json",
> >       "@type": "CreativeWork",
> >       "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},
> >       "about": {"@id": "./"}
> >     },
> >     {
> >       "@id": "./",
> >       "@type": ["Dataset", "LearningResource"],
> >       "hasPart": [
> >         {"@id": "data.csv"}
> >       ],
> >       "name": "Example dataset for RO-Crate specification",
> >       "description": "Official rainfall readings for Katoomba, NSW 2022, Australia",
> >       "datePublished": "2023-05-22T12:03:00+0100",
> >       "license": {"@id": "http://spdx.org/licenses/CC0-1.0"},
> >       "author": { "@id": "https://orcid.org/0000-0002-1825-0097" },
> >       "publisher": {"@id": "https://ror.org/05gq02987"}
> >     },
> >     {
> >       "@id": "data.csv",
> >       "@type": "File",
> >       "name": "Rainfall Katoomba 2022-02",
> >       "description": "Rainfall data for Katoomba, NSW Australia February 2022",
> >       "encodingFormat": "text/csv",
> >       "license": {"@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/"}
> >     },
> >     {
> >       "@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
> >       "@type": "CreativeWork",
> >       "name": "CC BY-NC-SA 4.0 International",
> >       "description": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International"
> >     },
> >     {
> >       "@id": "http://spdx.org/licenses/CC0-1.0",
> >       "@type": "CreativeWork",
> >       "name": "CC0-1.0",
> >       "description": "Creative Commons Zero v1.0 Universal",
> >       "url": "https://creativecommons.org/publicdomain/zero/1.0/"
> >     },
> >     {
> >       "@id": "https://orcid.org/0000-0002-1825-0097",
> >       "@type": "Person",
> >       "name": "Josiah Carberry",
> >       "affiliation": {
> >         "@id": "https://ror.org/05gq02987"
> >       }
> >     },
> >     {
> >       "@id": "https://ror.org/05gq02987",
> >       "@type": "Organization",
> >       "name": "Brown University",
> >       "url": "http://www.brown.edu/"
> >     }
> >   ]
> > }
> > ```
> {: .solution}
{: .question}



# Copyright and license

© Copyright 2021-2023 University of Technology Sydney, The University of Manchester UK and RO-Crate contributors.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

## Changes

This tutorial has been adapted from the revised [RO-Crate introduction for 1.2](https://www.researchobject.org/ro-crate/1.2-DRAFT/introduction.html) by Stian Soiland-Reyes. Changes include:
* Rewriting to tutorial style with exercises
* Adaptation to Galaxy Training Material rendering
* Explicit links to RO-Crate specifications
* Example changed to different organization and license


topic: fair
tutorial: data-management
title: FAIR data management solutions
difficulty: intermediate
time_estimation: 10M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: f51bd9e2be825906b9bd9971bf228996
last_modified: 2026-05-15T22:28:13.005655
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/data-management/tutorial.html

content:
The FAIR (Findable, Accessible, Interoperable, Reusable)  data stewardship created the foundation for sharing and publishing digital assets, especially data. This apply to machine accessibility and emphasize that all digital assets should share data in a way that will enable maximum use and reuse.

This tutorial is a short introduction to the FAIR data management framework. You can find out more at the [FAIR Pointers](https://elixir-uk.github.io/FAIR-Pointers/ep1/index.html) online course.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data management planning
In recent years we have notice a data explosion. Number of sequence records in each release of [GenBank](https://www.ncbi.nlm.nih.gov/genbank/statistics/), from 1982 to the present, doubled in size approximately every 18 months. Great amounts of data available followed by expanding range of tools and computational solutions result in reproducibility crisis. Having **data management plan (DMP)** in place is essential to achieve FAIR data management. DMPs are often described as living documents and should be updated according to changing circumstances.

There are several ways to set up FAIR (Findable, Accessible, Interoperable, Reusable) data management plans (DMPs) :
  - Findable (F): Data description and collection or reuse of existing data
  - Accessible (A): Standardised authentication or authorisation (e.g. HTTP, HTTPS)
  - Interoperable (I): Documentation and data quality
  - Reusable (R): Storage and backup supported by legal and ethical requirements

## Data description and collection or reuse of existing data
Reusing legacy datasets from institutional repositories or the digital libraries data collections can be FAIRified retrospectively. Support for data collection and development, throughout the life cycle can be provided and followed by change management and capacity improvement.

Multi-part FAIR research need a way of wrapping up, describing and sharing to promote the reuse of data. **Data sharing agreements** define the purpose of data sharing. Reference roles and responsibilities; specifies the purpose and legal requirements, e.g. for data security.

An institutional aim should be to create an integrated view and context over fragmented resources using their **persistent identifiers (PIDs)** and **metadata**. To make datasets findable, these metadata need to be as widely available as possible.

Enhancing reproducibility, quality and transparency by ensuring information flow and showcasing secondary use is also a part of data management. Promoting hands-on data experience and events activities built an collaborative environment for reproducible science.


## Documentation and data quality
Having access to local knowledge and encouraging best practises at the departmental level is a smart way to offer direction on a variety of standards and methods. In order to implement FAIR data practises within an institution, resources and infrastructure are needed. To increase the possibility of data reuse, several FAIR requirements can be satisfied using freely available guidelines e.g. [RDMkit](https://rdmkit.elixir-europe.org/), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html), [ELIXIR-UK DaSH Fellowship](https://sites.google.com/view/navigation-portal-fellowship/home?authuser=0) initiative and repositories e.g. [Zenodo](https://zenodo.org/), [Harvard Dataverse](https://dataverse.harvard.edu/) and [figshare](https://figshare.com/).

## Storage and backup
Systems for storage, backup and collaboration depend upon technical infrastructure.The **'3-2-1 rule'**, a recommendation for saving three copies of the research data—two locally and one off-site—is a standard backup strategy for research data. Data, metadata, and other research artefacts, such as ontologies, software, documentation, and papers, must all be kept in locations where they are adequately safeguarded, backed up, and accessible to maximise their potential for reuse. Appropriate access management is essential, in addition to backup and restoration services that protect researchers against data loss, theft, malfunctioning computers or storage media, and accidental deletion or inadvertent alterations to the data.

The fundamental component of infrastructure required for the FAIR research data lifecycle are repository services. They allow access to the data, a persistent identifier, and the descriptive metadata that support interoperability. Repositories can include basic data storage, resource finding, managing access and use of private information, facilitating peer review of information related to publications or services requiring digital preservation, and more.

The [OpenAIRE](https://www.openaire.eu/opendatapilot-repository-guide) repository guide advises users to check the availability of a suitable repository in following order:

1. The most effective option (if available) is to maintain the data in accordance with acknowledged discipline-specific criteria using an established, dedicated (external) data archive or repository that caters specifically to the study domain.
2. Making use of institutional data repositories is the second-best option.
3. If none of those options is practical, a free data repository should be used.

Up-to-date lists of available registered data repositories can be found at [re3data](https://www.re3data.org/) and [FAIRsharing](https://fairsharing.org/search?fairsharingRegistry=Database).

## Legal and ethical requirements
Institutional support network (data stewards, ethics boards, IP, legal and financial offices) need to guide researchers in safeguarding data management responsibilities and resources.

# Conclusion
You will have the advantage of saving time and resources by planning how to FAIRify your data in the early phases of your research endeavour. To put this into action, a data management strategy, or DMP, must be written. A DMP is also where you outline your data collection, storage, processing, sharing, and disposal procedures. Planning the management and FAIRification of your data reduces the possibility of issues down the road, whether they be practical, legal, or technical.

Keep in mind that creating FAIR data is a complex process. Consider how you can make your data FAIR one step at a time at each stage of the creation, collection, documentation, storage, sharing, archiving, and preservation processes. The framework for the rest of your study planning is laid by incorporating your data documentation. Imagine you want to use a dataset that was generated by another researcher and how you would like it to be found. Hope this quick introduction to FAIR data management solutions will help you improve not only your experience with data but also influence others by using your guidance and FAIRified resources.


topic: fair
tutorial: fair-rna
title: FAIRification of an RNAseq dataset
url: https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-rna/tutorial.html
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 0005cf43f641a1ed9b1d8d5abee1afe8
last_modified: 2026-05-15T22:28:13.157512
workflows_json: []
content:
[RNA sequencing](https://en.wikipedia.org/wiki/RNA-Seq) is chosen here as an example of how to FAIRify data for a popular assay in the Life Sciences. RNAseq data can be shared and curated in designated public repositories using established ontologies (and controlled vocabularies) for describing protocols and biological material (metadata).

Two international repositories are commonly used to locate and download RNAseq (meta)data: [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress) and [GEO](https://www.ncbi.nlm.nih.gov/geo/).  Other repositories for raw sequence data exist (e.g. SRA, ENA, DDBJ), but ArrayExpress and GEO specifically house and index expression data , including rich metadata detailing samples, data processing and final results files such as gene expression matrices.

By submitting data to a public repository, it becomes openly accessible, searchable and annotated with rich metadata, by the submitter and curation team. Note, both repositories belong to the [FAIRsharing database registry](https://fairsharing.org/search?fairsharingRegistry=Database), which can help you find public repositories for all types of Life Science data.

This lesson will take you through a publicly available RNAseq dataset in ArrayExpress and show you how it meets FAIR principles using the checklist published in 2016 {% cite Wilkinson2016 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Example of FAIRification when data is uploaded to a public repository

We will use the following human RNAseq dataset for this learning material [ArrayExpress:E-MTAB-8316](https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-8316). This link hosts all metadata and links to downloadable raw and transformed data, shown as a schematic below.

![A schematic of RNAseq data and metadata within ArrayExpress](../../images/fair_rna_1.png "A schematic of RNAseq data and metadata within ArrayExpress. Data shown are raw FASTQ files and summary gene transcript count matrices.")


# Finding and accessing an RNAseq dataset
In this section, four of the FAIR Guiding Principles are put into practice:
- **(F1) (Meta)data are assigned a globally unique and persistent identifier**
- **(F4) (Meta)data are registered or indexed in a searchable resource**
- **(A1) (Meta)data are retrievable by their identifier using a standardised communications protocol**
- **(A1.1) The protocol is open, free, and universally implementable**


## (F1) (Meta)data are assigned a globally unique and persistent identifier

> <question-title></question-title>
>
>  The RNAseq dataset we are looking at belongs to a study published in Nature Medicine {% cite Alivernini2020 %}.  Can you find the globally unique and persistent identifier for the RNAseq data and data descriptions (metadata), within the text of the publication? 
>
> > <solution-title></solution-title>
> >
> > Under "Data and code availability', towards the end of the manuscript, we are told the following. 
> >
> > ![The data availability section within the RNAseq publication](../../images/fair_rna_2.png "The data availability section within the RNAseq publication showing links to access primary data.")
> >
> {: .solution}
{: .question}



Note that the dataset's unique and persistent (unchanging) identifier is E-MTAB-8316, (F1) and could also be expressed as the full [URL](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316. Note that there are two other identifiers given to 2 single cell RNAseq datasets published as part of the same study. For this lesson, we are only considering the bulk RNAseq dataset.

Note also, the same record can be accessed using a resolution service, such as [identifiers.org](https://identifiers.org/),  that allows URLs to be regularised using a namespace (arrayexpress) and the identifier local to the database ([E-MTAB-8316](https://identifiers.org/arrayexpress:E-MTAB-8316))



## (A1) (Meta)data are retrievable by their identifier using a standardised communications protocol (A1.1) The protocol is open, free, and universally implementable 

> <question-title></question-title>
>
>  Use the following [URL](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316) to access this dataset. The URL, in this case, is a concatenation of the URL of the data repository and the unique identifier: 
>
> > <solution-title></solution-title>
> >
> > You should see the following webpage hosted at the EBI (www.ebi.ac.uk). In the context of FAIR, you have now accessed this RNAseq data using a weblink employing https, where https is a standardised communications protocol or “data getting method” that is open, free and universally implementable (A1, A1.1).  Note that the title of this RNAseq record and the title of the companion publication are different, which means the two cannot be linked by title.  Instead, the accession (persistent ID) is used to connect the data and companion publication.
> >
> > Many databases, including ArrayExpress, also have programmatic ways to access and download data.  For advanced users, ArrayExpress [host a REST server](https://www.ebi.ac.uk/biostudies/arrayexpress/help#programmatic) for this purpose.  Other packages written by third parties, such as [ffq](https://github.com/pachterlab/ffq), also permit this function.
> > ![RNAseq data record](../../images/fair_rna_3.png "The RNAseq data record showing the Primary ID circled in red.")
> >
> {: .solution}
{: .question}


## (F4) (Meta)data are registered or indexed in a searchable resource

> <question-title></question-title>
>
>  Location the dataset using the persistent ID in a different way.  Use the search menu in ArrayExpress to find the dataset again.   Use the following link to (access ArrayExpress)[https://www.ebi.ac.uk/biostudies/arrayexpress/studies/] and then type E-MTAB-8316 into the search bar.
>
> > <solution-title></solution-title>
> >
> > Note here that data access is gained through searching the database whereas previously we used a direct URL.  We are able to search ArrayExpress with the persistent ID  because the RNAseq dataset and its metadata are indexed (F4).
> >
> > ![ArrayExpress/BioStudies search bar](../../images/fair_rna_4.png "The ArrayExpress/BioStudies search bar with the query circled in red.")
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> Alternatively, use the same search menu in ArrayExpress to search again using the words "macrophage rheumatoid arthritis" and selecting "rna-seq of coding rna" on the left-hand search bar. 
>
> > <solution-title></solution-title>
> >
> > Here, the dataset we want is not the first in the list but appears later in the search results. In this example, data access is gained through searching metadata (data about the experiment) (F4), and not the persistent ID as we did previously.
> >
> > ![ArrayExpress/BioStudies search bar for the metadata query](../../images/fair_rna_5.png "The ArrayExpress/BioStudies search bar showing a metadata query circled in red.")
> >
> {: .solution}
{: .question}


# Learning about the experiment and dataset from reading the metadata

In this section, the Findability Principles are put into practice:

- **(F1) (Meta)data are assigned a globally unique and persistent identifier**
- **(F2) Data are described with rich metadata**
- **(F3) Metadata clearly and explicitly include the identifier of the data they describe**

## (F1) (Meta)data are assigned a globally unique and persistent identifier (F3) Metadata clearly and explicitly include the identifier of the data they describe

> <question-title></question-title>
>
> Start looking at the metadata given on dataset webpage. Try to find the unique, persistent identifier in the [record](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316) (F1) (Meta)data are assigned a globally unique and persistent identifier.  
>
> > <solution-title></solution-title>
> >
> > The persistent identifier is circled in red and is the first thing we see in the record (F1). All metadata (descriptions about the data) and the actual raw data files are linked from this page. Here metadata clearly and explicitly include the identifier of the data they describe (F3).
> >
> > ![RNAseq data record showing the Primary ID (persistent identifier)](../../images/fair_rna_6.png "The RNAseq data record showing the Primary ID (persistent identifier) circled in red.")
> >
> > Here data are described with rich metadata (F2), which allows a person to reuse data appropriately by reducing ambiguity around what the data mean or how they are derived. This principle applies also to other principles in  FAIR such as (R1) Meta(data) are richly described with a plurality of accurate and relevant attributes.
> >
> {: .solution}
{: .question}

## (F2) Data are described with rich metadata
Indexing rich metadata allows a person to easily locate a dataset of interest, since it is made searchable within ArrayExpress. Metadata is added by the person submitting the data and is further curated by the ArrayExpress database. Metadata curation is performed via a web-based submission interface called [Annotare](https://www.ebi.ac.uk/fg/annotare/login/) which aids rich curation through using community ontologies and controlled vocabularies.


> <question-title></question-title>
>
> Familiarise yourself with the [page layout](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316): there are links to all protocols, data, sample metadata and assay type. How many samples are in this dataset?
>
> > <solution-title></solution-title>
> >
> > There are 12 samples in this dataset.
> >
> > ![The RNAseq data record showing the number of samples (assays)](../../images/fair_rna_7.png "The RNAseq data record showing the number of samples (assays) circled in red.")
> >
> {: .solution}
{: .question}



> <question-title></question-title>
>
> What data provenance can you find? i.e. what processes have been performed to create the data linked from this page? Are there any metadata which might be absent?
>
> > <solution-title></solution-title>
> >
> > The final 2 protocols within the protocol table drawdown detail all data transformations for the raw and transformed data. 
> >
> > ![The protocol table from the RNAseq data record](../../images/fair_rna_8.png "The protocol table from the RNAseq data record showing the final 2 data protocols circled in red.")
> >
> {: .solution}
{: .question}


When recording data provenance, ambiguities can be introduced.  True provenance can be achieved though through publishing scripts and software versions, enabling other researchers to repeat all data transformation processes. 


# Metadata and community standards

In this section, we will consider Interoperability and data Reuse:

- **(I1) (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.**
- **(I2) (Meta)data use vocabularies that follow FAIR principles.**
- **(R1) (Meta)data are richly described with a plurality of accurate and relevant Attributes.**
- **(R1.3) (Meta)data meet domain-relevant community standards.**


Interoperability is mentioned in our companion (FAIR Pointers course)[https://fellowship.elixiruknode.org/latest/carpentries-course-fair-pointers]  where we consider merging datasets using a common controlled vocabulary.  

## (I1) (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.
Commonly when adding metadata to a dataset a published ontology is used.  In its simplest form, an ontology is a dictionary of terms you use to annotate a piece of data. For example, the NCBI taxonomy database is something you may have used to annotate species within a dataset. An ontology, though, will also define relationships between terms. So in the taxonomy example, the term "Homo sapiens" will belong to parent terms such as Primate, Mammal and so on. Importantly, by using an ontology you can ensure you are using interoperable and searchable terms with your data.

## (I2) (Meta)data use vocabularies that follow FAIR principles.  (R1) (Meta)data are richly described with a plurality of accurate and relevant Attributes.
Published ontologies are linked from the ArrayExpress data submission tool, [Annotare](https://www.ebi.ac.uk/fg/annotare/login/), so this work is done for you. Although, if you wish to start using ontologies to annotate your data at the point it is produced, the [FAIRsharing Standards registry](https://fairsharing.org/search?fairsharingRegistry=Standard) is a place to start. Importantly, when an ontology is published it too becomes findable and reusable in the context of the FAIR principles **(I2)**.
Metadata using published ontologies permit interoperability since you can match identical annotations across data and databases **(I1 & I2)**. Additionally, they are created, refined and used by communities of practice **(R1, R1.3)**.


> <question-title></question-title>
>
> Look at the page again. Identify any metadata that belongs to an existing, published ontology. Note, we have mentioned one already: "Homo sapiens" as part of [taxonomy](https://www.ncbi.nlm.nih.gov/taxonomy).
>
> > <solution-title></solution-title>
> >
> > There are many.  There are ontologies used for Source Characteristics such as ‘Developmental stage’,  ‘Disease’, ‘Organism part’, ‘Cell type’, and so on. These annotations are curated at the point of data submission.
> >
> > ![published ontologies metadata](../../images/fair_rna_9.png "Metadata within the record using controlled terms from published ontologies.")
> >
> > There are also others that are not so obvious. Under "Protocols', a protocol ontology is used under the column ‘Type’.  By selecting the EFO link-and-arrow following the protocol annotation, you are taken to the FAIR (Experimental factor ontology)[https://www.ebi.ac.uk/ols/ontologies/efo] at the EBI. 
> >
> >![Experimental Factor Ontology protocol table from the RNAseq data record](../../images/fair_rna_10.png "The protocol table from the RNAseq data record showing the final 2 data protocols circled in red with links to the Experimental Factor Ontology.")
> >
> {: .solution}
{: .question}

## (R1.3) (Meta)data meet domain-relevant community standards.


![fair RNAseq MINSEQ](../../images/fair_rna_11.png "The MINSEQ Score table from the RNAseq data.")

Notice that the "MINSEQE" star rating on the left hand banner, indicates a level of Community Compliance in terms of (meta)data submission.   The community  standard in this case is MINSEQE (Minimum Information about a Sequencing Experiment Checklist) which is recorded in FAIRsharing {% cite https://doi.org/10.5281/zenodo.5706412 %}.  This example receives a maximum 5-star score demonstrating that data and metadata are available for experiential design, protocols and  variables as well as processed and raw data.

At a more advanced level of community compliance, the ArrayExpress database uses a domain specific metadata format: MAGE-TAB, also identified in [FAIRsharing](https://fairsharing.org/FAIRsharing.ak8p5g).  MAGE-Tab is a tab-delimited file containing all relevant metadata for the experiment.  This is a machine-readable version of the webpage we are looking at.



> <hands-on-title>Locate the MAGE-TAB file containing sample metadata, and open in Excel.</hands-on-title>
>
> Notice how all data on the webpage are also contained in this machine-readable format.  You are looking to download the SDRF file (Sample and Data Relationship Format).
>
> ![fair RNAseq links](../../images/fair_rna_12.png "Links to machine-readable metadata in the RNAseq data record circled in red.")
{: .hands_on}


Finally, this ArrayExpress record links to the raw sequence data housed in the ENA (European Nucleotide archive).  We see this in an exercise later in this course.  The ENA record in turn links to ENA SRA XML format (community recognised), which gives another machine-readable interoperability format.


> <hands-on-title>Machine Readable RNA XML formats</hands-on-title>
> 
> Have a quick look at the machine readable RNA XML formats.  Notice how data are contained within tags very similar to HTML.
>
> - [ENA:PRJEB34761](https://www.ebi.ac.uk/ena/browser/api/xml/PRJEB34761)
> - [ENA:ERP117716](https://www.ebi.ac.uk/ena/browser/api/xml/ERP117716)
>
{: .hands_on}


# Downloading data and metadata for reuse
- **(R1.1) (Meta)data are released with a clear and accessible data usage license.**
- **(R1.2) (Meta)data are associated with detailed provenance.**

## (R1.1) (Meta)data are released with a clear and accessible data usage license
Data licensing makes it clear to any person looking to reuse a dataset, who the data belongs to and how it can be used.  Open licences (available to all) are often used to maximise data reuse, though different stipulations can be applied to these licences to restrict/enforce certain downstream activities, such as reuse of data for commercial purposes.

The [Creative Commons](https://creativecommons.org/) licences are used commonly for life science data, of which there are 7: **CC0, CC BY, CC BY-SA, CC BY-NC, CC BY-ND, CC BY-NC-SA, CC BY-NC-ND**

CC0 is the most permissive licence where "no rights are reserved'.  This means the data creator releases the data without restriction to the public domain.

The second CC BY licence is often written as CC BY 4.0 to reflect the most current (4th) version of the licence.  Here, anyone can share and adapt data for any purpose, including commercial CC BY.

CC BY 4.0 is virtually identical to CC0, except by using CC BY, legal rights are retained to acknowledge the originator of the data.


> <question-title></question-title>
>
> Look at the following site detailing the (Creative Commons licences)[https://creativecommons.org/share-your-work/cclicenses/].  What is the effect of adding the following letters to the CC BY licence; SA, NC and ND?
>
> > <solution-title></solution-title>
> >
> > ![fair RNAseq licences](../../images/fair_rna_13.png "Screenshots from the Creative Commons licences definitions pages for SA, NC and ND.")
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> Look for the licensing options for [arrayexpress](https://www.ebi.ac.uk/biostudies/arrayexpress) at the bottom of the page under [Licensing](https://www.ebi.ac.uk/licencing).  Which licence is used?
>
> ![fair RNAseq licence footer](../../images/fair_rna_14.png "The footer from the RNAseq data record showing the link to the Licensing conditions circled in red.")
>
> > <solution-title></solution-title>
> >
> > CC0 is used.
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> How could I download the data or metadata  from this page?
>
> > <solution-title></solution-title>
> >
> > Machine-readable metadata can be downloaded from the “MAGE-TAB Files” link.  There are 2 files: Investigation Design Format (experimental overview) and the Sample/Data Relationship Format (sample/data descriptions).  Note this information can be accessed on the right hand banner as well.
> >
> > Processed data can be downloaded also, in this case there is a link to a read-count matrix.  Sequencing data from the NEA.
> >
> > Access to raw FASTQ data is given via the reference to the ENA database on the right hand banner under “Linked Information”.
> >
> > ![fair RNaseq metadata and rawdata](../../images/fair_rna_15.png "Links to metadata, raw data and processed data in the RNAseq record, circled in red.")
> >
> {: .solution}
{: .question}


## (R1.2) (Meta)data are associated with detailed provenance.
The term "data provenance" relates to information about how and why a piece of data was produced. Often the individual producing the data and details of its production (protocols) are recorded as part of provenance.

> <question-title></question-title>
>
> What provenance metadata can you find on [Arrayexpress:E-MTAB-8316](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316)?
>
> > <solution-title></solution-title>
> >
> > We are looking for information that could help us understand how data have been created.  We have been given the name of the data creator noting that there is an option too for the author to submit an ORCID.  Data creation and transformations are detailed under Protocols.
> >
> > ![fair RNaseq submitter](../../images/fair_rna_16.png "Showing submitter and data provenance metadata in the RNAseq record, circled in red.")
> >
> {: .solution}
{: .question}


topic: fair
tutorial: fair-persistent-identifiers
title: Persistent Identifiers
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 33a6e09eedc6813f9e319f79d56321c5
last_modified: 2026-05-15T22:28:13.155137
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-persistent-identifiers/tutorial.html

content:
The persistent identifier is defined.  Learners will be able to define the structure of an identifier and explain its importance.  

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Persistent identifiers and the FAIR Principles
Data Identifiers relate to the following 5 FAIR Principles (Table 5.1).  We will discuss and signpost these in this episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | **F1. (meta)data are assigned a globally unique and persistent identifier**<br>F2. data are described with rich metadata (defined by R1 below)<br>**F3. metadata clearly and explicitly include the identifier of the data it describes** <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | **A1. (meta)data are retrievable by their identifier using a standardized communications protocol** <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>**A2. metadata are accessible, even when the data are no longer available** |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>**I3. (meta)data include qualified references to other (meta)data**                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. (meta)data are released with a clear and accessible data usage license<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        


Table 5.1: The 15 FAIR Guiding Principles.  Principles relating to data identifiers are highlighted in **black**.

# Using persistent identifiers (PIDs)

Identifiers are an important theme within the FAIR principles, arguably being foundational; they are considered two of the pillars for the FAIR principles since they are crucial for Findable (F) and Accessible (A) principles.

Identifiers are an eternal reference to a digital resource such as a dataset and its metadata. They provide the information required to reliably identify, verify and locate your research data. 

Commonly, a persistent identifier is a unique record ID in a database, or a unique URL that takes a researcher to the data in question in a database. Persistent identifiers (PIDs) have to be unique so that only one dataset can be identified by this identifier. In addition to the identifier being unique, it needs to be persistent. When depositing or hosting data, you should ensure the longevity of this persistence meets your requirements, which may require reading specific database policies regarding identifier policy.  

Since FAIR permits the withdrawal of data, the FAIR Principles combat the potential for broken URLs by stating:  _“Metadata are accessible, even when the data are no longer available.”_  (FAIR Principle A2).  This means the link (PID) remains valid, displaying all the original metadata of the record even though the data is no longer available.

**It is important to note that when you upload your data to a public repository, the repository will create this ID for you automatically**.

Based on [how to FAIR](https://howtofair.dk/how-to-fair/persistent-identifiers/), many resources can help you find databases to assign PIDs to your data. One of these resources is [FAIRsharing](https://fairsharing.org/) something we’ve already encountered in the previous episodes. [FAIRsharing](https://fairsharing.org/), provides a list of databases grouped by domains and organisations.

# The structure of persistent identifiers

To ensure that a PID is globally unique, commonly it is supplied as a unique URL. For the case of a record in a biological database, the use of a URL ensures that the database record ID is associated with the database name or often some derivation of this. This is often enough to ensure the uniqueness of the PID for any future scenario. 

Commonly, for things like publications, a DOI is used for the PID, where DOI stands for Digital Object Identifier. An example is shown below where the PID is constructed from 3 pieces of information: the resolver service, the prefix (namespace) and the suffix (dataset ID).

**Resolver service**: the domain/service/institution hosting the PID e.g. [https://www.doi.org] (doi.org)

**Prefix**: a unique number referring to the publisher. This is also known as the namespace.

**Suffix**: the unique dataset number

![DOI_structure](../../images/figure5-1.png "The structure of a DOI")

For biological data, PIDs usually require a resolver that can deal with multiple resolving locations, which means that if a database changes its name or internal structure, the new and old variations of the PID remain valid and take the user to the (meta)data.  One commonly used resolver service is called [identifiers.org](https://docs.identifiers.org/) which maintains a list of database namespaces (prefix’s) as a persistent record.  If a database changes its name, it keeps the original namespace operational or alternatively arranges for redirection from the original.

Examples of using identifiers.org to construct a PID are given below for 2 different databases, [Ensembl](https://www.ensembl.org/Mus_musculus/Gene/Summary?g=ENSMUSG00000031887;r=8:105984918-105991241) and [WikiPathways](https://www.wikipathways.org/pathways/WP1243.html), respectively. The **namespace** is given as the database name in these examples.

![PID_structure](../../images/figure5-2.png "The structure of a PID using the identifiers.org resolver service")

> <question-title></question-title>
>
> Access the preprint for [Identifiers for the 21st century: How to design, provision, and reuse persistent identifiers to maximize utility and impact of life science data](https://www.google.com/url?q=https://doi.org/10.1101/117812&sa=D&source=docs&ust=1681921501619473&usg=AOvVaw0iHyEgvEzDniMbIcvhokQk).  This paper makes 10 recommendations for PID best practice. 
>
> <p> Locate the lesson for “Do not reassign or delete identifiers”.  Which PID is used as an example of a “tombstone page”?  Which FAIR Principle does this relate to? </p>
>
> > <solution-title></solution-title>
> >
> > Tombstone PID: [https://www.uniprot.org/uniprotkb/A0AV18](https://www.uniprot.org/uniprotkb/A0AV18/history)
> > <p> </p> This relates directly to the FAIR Principle A2. _“Metadata are accessible, even when the data are no longer available”_ Note that other FAIR Principles are also illustrated.
> > <p>  </p> The Tombstone page is _“retrievable by their identifier using a standardised communications protocol”_(FAIR Principle A1), which in this case is http(s).  The page contains metadata which _“are assigned a globally unique and persistent identifier”_ (FAIR Principle F1).  Also the metadata _“clearly and explicitly includes the identifier of the data they describe”_ (FAIR Principle F3) noting that the identifier itself is featured on the webpage.
> > <p>  </p> Additionally, the webpage features a link to the updated UniProt record thereby _“metadata include qualified reference to other (meta)data”_ (FAIR Principle I3)
> >
> {: .solution}
{: .question}


# Useful Resources
- More on identifiers: [RDMkit](https://rdmkit.elixir-europe.org/identifiers.html) and [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/findability/identifiers.html)
- Nick Juty, Sarala M. Wimalaratne, Stian Soiland-Reyes, John Kunze, Carole A. Goble, Tim Clark; Unique, Persistent, Resolvable: Identifiers as the Foundation of FAIR. Data Intelligence 2020; 2 (1-2): 30–39. {% cite Juty2020 %}


topic: fair
tutorial: fair-origin
title: FAIR and its Origins
difficulty: intermediate
time_estimation: 40M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 3bbde779f497947ed103a051e85a7c1a
last_modified: 2026-05-15T22:28:13.154426
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-origin/tutorial.html

content:
FAIR as an acronym, its origins and its guiding principles are introduced.  Learners will be able to explain the difference between FAIR and open data and contextualise the main principles of FAIR around the common characteristics of identifiers, access, metadata and registration.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



# FAIR

The concepts underlying the FAIR principles are intuitively grounded in good scientific practice, though formalising them as easily measurable is not straightforward.  Being completely ‘FAIR’ is somewhat of an ideal, where it is often more pragmatic to be ’FAIR enough’ for a particular purpose or use case.

FAIR’s primary goal is to maximise data reuse by researchers.  The FAIR principles enable reuse by helping researchers share and manage their data, though FAIR is not limited to data alone and can be applied to services, software, training and workflows.  The word FAIR is an acronym, derived from its major components: ‘F’indable, ‘A’ccessible, ‘I’nteroperable, and ‘R’eusable which form the foundation of the FAIR Guiding Principles. 

**Findable** means that data and its metadata can be found/discovered by humans and computers.  Part of this is making rich metadata and keywords available to search engines and data repositories, so that companion data can be discovered.

**Accessible** means that once discovered, data and metadata can be accessed/downloaded by humans and computers.  Typically this means the commitment of the resource to its long term hosting and availability, with a suitable licence, and in appropriate format.

**Interoperable** means that data and metadata are supplied in formats that can be easily used and interpreted by humans and computers. The file formats and terms (vocabularies) used can be integrated easily with other datasets and software.

**Reusable** means that metadata is rich, enabling appropriate reuse.  Commonly FAIR will encourage the use of community standards for data curation.

These 4 major components form headings for the 15 FAIR Guiding Principles shown in **Table 1.1**

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a globally unique and persistent identifier<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. (meta)data are released with a clear and accessible data usage license<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        |

Table 1.1: The FAIR guiding principles as described in Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) {% cite Wilkinson2016 %}. 


> <question-title></question-title>
>
> Look at the wording of the FAIR principles in **Table 1.1**. Which terms are used more than once?  Which terms are you seeing for the first time?
>
> > <solution-title></solution-title>
> >
> > Shared terms are: **data** (F1, F2, F4, A1, A2, I1, I2, I3, R1, R1.1, R1.2, R1,3), **metadata** (F1, F2, F3, F4, A1, A2, I1, I2, I3,  R1, R1.1, R1.2, R1,3), **identifier** (F1, F3, A1), **protocol** (A1, A1.1, A1.2). <br> Not so obviously shared terms are: **vocabularies** (I1, I2), **access** (A1, A1.1, A1.2, R1.1) <br> Terms which you may be seeing for the first time are: metadata, persistent identifier, searchable resource, standardised communications protocol, authentication and authorisation procedure, knowledge representation, vocabularies, qualified references, data usage license, provenance, domain-relevant community standards.
> >
> {: .solution}
{: .question}


The aim of this course is to put these principles into context and familiarise learners with  terms used commonly around FAIR.  A glossary of terms appears at the bottom of this episode as a point of reference.

# Open data and FAIR 

![FAIR_is_not_the_same_as_Open_data](../../images/figure1-2_fair-vs-open.png "FAIR is not the same as Open data")
 

Since FAIR promotes data sharing, it is often misunderstood as Open data.
  
The [Open Data handbook](https://opendatahandbook.org/) defines Open data as **“data that can be freely used, reused and redistributed by anyone.”**  Commonly, FAIR data is open however FAIR compliance does not mandate access without restriction. Instead, for instances where FAIR data is subject to restricted access, the conditions of access need to be stated to be compliant with FAIR, for example access around sensitive data.  Often in FAIR data, people adhere to the philosophy of **“as open as possible, and as closed as necessary”** thereby maximising opportunities to reuse data.

> <question-title></question-title>
>
> Under which circumstances would restricted or closed  access to FAIR data be advisable?
>
> > <solution-title></solution-title>
> >
> > Where data is sensitive or subject to intellectual property.  Protecting sensitive data overrules mandating research data should be open access. <br> Note though that in most cases, people following FAIR principles will be looking to share their data openly. Also note that sensitive data can be released through anonymisation and in many cases subject to controlled access by the authority of the principal investigator or data access committee. 
> >
> {: .solution}
{: .question}


# What is meant by FAIRification and FAIRness of data? 
  
**FAIRification** is the process of making your data FAIR compliant by applying the 15 Guiding Principles shown in **Table 1.1**.  The extent to which you apply these principles defines the **FAIRness** of your data.  In other words, FAIRness refers to the extent by which your data is FAIR and implies some implicit means of measuring its compliance.

# FAIR’s origins

A [report from the European Commission Expert Group on FAIR data](https://zenodo.org/record/1285272#.ZD_9KhXMIqt) describes the origins of FAIR and its development in 2014-2015 by a FORCE11 Working Group.  The following exercise dips into this report and asks you to investigate some of FAIR’s history and foundation. 


> <question-title></question-title>
>
> Read page 11 of [the European Commission report](https://zenodo.org/record/1285272#.ZD_9QxXMIqu), under the heading “Origins and definitions of FAIR”. What benefit did the FORCE11 Working Group see to coining the word FAIR?
>
> > <solution-title></solution-title>
> >
> > The report states: “a FORCE11 Working Group coined the FAIR data definition, latching onto an arresting and rhetorically useful acronym. The wordplay with fairness, in the sense of equity and justice, has also been eloquent in communicating the idea that FAIR data serves the best interests of the research community, and the advancement of science as a public enterprise that benefits society.”
> >
> {: .solution}
{: .question}


# Course structure
  
During the remainder of this course we will put the FAIR Guiding Principles into context using the 4 FAIR characteristics of **metadata**, **data registration**, **access** and **identifiers**, and devote an episode to each of these.  Whilst we teach, we will map content to the appropriate FAIR principle and define all relevant terms identified.
 
# Glossary of FAIR terms
  
All terms in this glossary are mentioned in the FAIR Guiding Principles (Table 1.2) and referenced in the following episodes of this course.
  
  
  | The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a **globally unique and persistent identifier**<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the **identifier** of the data it describes <br>F4. (meta)data are registered or **indexed in a searchable resource**                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their **identifier** using a **standardized communications protocol** <br>A1.1 the protocol is open, free, and **universally implementable**<br>A1.2 the protocol allows for an **authentication and authorization procedure**, where necessary <br>A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable **language for knowledge representation**. <br>I2. (meta)data use **vocabularies** that follow FAIR principles<br>I3. (meta)data include **qualified references** to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br> R1.1. (meta)data are released with a clear and accessible **data usage license**<br>R1.2. (meta)data are associated with detailed **provenance**<br>R1.3. (meta)data meet domain-relevant **community standards**            

Table 1.2: Terms used by the FAIR Principles that appear in this glossary are highlighted in **black**.

**Globally unique and persistent identifier**: a reference to a digital resource such as a dataset, a document, a database, etc, usually given as a URL, that takes the user to that resource.  The persistent identifier (PID) is unique, globally, in the sense that it is not used to identify any other digital resource.  The PID is persistent, in that it enables a resource to be located long term, preferably permanently.  For more see: [How to FAIR](https://howtofair.dk/how-to-fair/persistent-identifiers/) and [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/findability/identifiers.html), episode 5 (identifiers).

**Indexed in a searchable resource**: repositories and catalogues that can be queried (search box) are examples of searchable resources.  Indexing refers to processes within the architecture of the data repository where (meta)data are organised so that they can be queried based on defined fields.  Indexing by an internet search engine (for example, Google) is another example of an indexed and searchable resource.  For more see: [GO FAIR](https://www.go-fair.org/fair-principles/f4-metadata-registered-indexed-searchable-resource/).
  
**Standardised communications protocol**: a method that connects two computers and ensures secure data transfer. Examples of this include the hypertext transfer protocol ([http](https://en.wikipedia.org/wiki/HTTP)(s)) and the file transfer protocol ([ftp](https://en.wikipedia.org/wiki/File_Transfer_Protocol)) that permit data to be requested and downloaded by selecting a link on a webpage, or launched from within a script, for example using an (application programming interface) API. For more see the [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/).
  
**Universally implementable**: in the context of a standardised communications protocol such as http(s), universally implementable means it can be used by a number of resources and software, for example Firefox, Chrome and Unix commands such as wget.  For more see the [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/).
  
**Authentication and authorisation procedure**: data repositories with authentication and authorisation procedures generally require a login (and password) to access (meta)data.  For more see the [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/).

**Language for knowledge representation**: in the context of (meta)data exchange between computers, (meta)data should be in formats that are universally recognised (interoperable standards).  For more see [GO FAIR](https://www.go-fair.org/fair-principles/i1-metadata-use-formal-accessible-shared-broadly-applicable-language-knowledge-representation/) and [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/introduction/FAIR-and-knowledge-graphs.html).
  
**Vocabularies**: (or **controlled vocabulary**) is a dictionary of terms you can use when producing (meta)data.  Controlled vocabularies are often shared between databases and communities so by using them you can allow data from different sources to be merged (interoperable), based on a shared understanding of the concepts.   Ontologies are related to vocabularies, where terms in the vocabulary are organised by relations between them.  A commonly used ontology is the [NCBI taxonomy](https://www.ncbi.nlm.nih.gov/taxonomy) where the term ‘Homo sapiens’ belongs to a hierarchy of parent terms such as ‘Primates’ and ‘Mammalia’.   The ontology defines the vocabularies and the parent/child relationships.   For more see Ten simple rules for making a vocabulary FAIR {% cite Cox2021 %}.
  
**Qualified references**: are terms used to describe relationships to pieces of (meta)data.  For more see [GO FAIR](https://www.go-fair.org/fair-principles/i3-metadata-include-qualified-references-metadata/).
  
**Data usage license**: describes the legal rights on how others use your data.  For more details on considerations in selecting a licence, or to find out more about types of licence that are available, see [RDMkit](https://rdmkit.elixir-europe.org/licensing). 
  
**Data provenance**: refers to metadata describing the origin of a piece of data, including information such as version, original location of the data, and usually an audit trail up to the current version.  For more see [RDMkit](https://rdmkit.elixir-europe.org/data_provenance#how-to-document-and-track-your-data-provenance).
  
**Community standards**: are standard guidelines used to structure and exchange data, usually supported by community-developed resources and/or software.  In the context of (meta)data, community standards relate often to the standardised ontologies used by a domain of research, and minimum information guidelines allowing data to be interoperable.  For more see [FAIRDOM](https://fair-dom.org/community_standards).
  
**Machine-readable**: though this term is not referenced in the FAIR principles, it is often discussed within the FAIR context. Machine-readable (meta)data is supplied in a structured format that can be read by a computer.  For more see [Open Data Handbook](https://opendatahandbook.org/glossary/en/terms/machine-readable/) and [RDMkit](https://rdmkit.elixir-europe.org/machine_actionability).
  
**(Meta)data**: is shorthand for ‘metadata and data’.

# Useful Resources
- The published FAIR Guiding Principles: Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) {% cite Wilkinson2016 %}.
- Recipes for data FAIRification, written by domain experts giving real-world examples: [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html)
- Documentation and frameworks for data FAIRification.  Each of the 15 FAIR principles is put into context with real data examples: [GO FAIR](https://www.go-fair.org/fair-principles/)
- FAIR walkthrough using examples from across all academic disciplines: [How to FAIR](https://howtofair.dk/)


topic: fair
tutorial: ro-crate-submitting-life-monitor
title: Submitting workflows to LifeMonitor
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ro-crate"]
content_hash: 35746fc56bb520b1fa027ec57cfdc799
last_modified: 2026-05-15T22:28:13.194822
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-submitting-life-monitor/tutorial.html

content:
[{LM}](https://www.lifemonitor.eu/) is a service to support the
sustainability and reusability of published computational workflows. The
LifeMonitor helps alleviate the burden of maintaining workflows over time
through **automation**.

This document will show you **how to use the
[LifeMonitor](https://www.lifemonitor.eu/) service and its [GitHub
app](https://github.com/apps/lifemonitor)** on a workflow in a GitHub
repository.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Why use the LifeMonitor?

Keeping workflows reusable takes work! Like all software, even the best
workflows can break over time if left unmaintained: for instance, there could be
a regression or an API change in an unpinned dependency, or an external resource
that the workflow relies upon might be moved to a different URL. Periodic
automated **testing** is a fundamental practice that helps expose such problems,
giving workflow maintainers a chance to intervene and fix them.  The
[LifeMonitor](https://www.lifemonitor.eu/) helps you monitor and periodically
execute the automated tests for your workflow.

Moreover, workflows should be findable to be reusable.
[LifeMonitor](https://www.lifemonitor.eu/) helps automate the generation of
[RO-Crate metadata](https://www.lifemonitor.eu/workflow_testing_ro_crate) as
well as registering new releases of your workflow with the
[WorkflowHub](https://workflowhub.eu).

Finally, [LifeMonitor](https://www.lifemonitor.eu/) can apply automated checks
to your workflow repository to help you follow [community best
practices]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}).


## Pre-conditions

We assume you **published a Galaxy workflow** to a GitHub repository following the
[*Galaxy Community best
practices*]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#community-best-practices).

In addition, we assume you have followed the same best practices to [**create
tests** for your
workflow]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#generating-tests-for-your-workflow) using
[Planemo](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html#tests),
as well as configuring a [GitHub Actions
workflow]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#adding-a-github-workflow-for-running-tests-automatically) to automatically
run those tests.  Don't worry if this sounds like a lot: the [best
practices]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}) document provides straightforward
instructions.

### Running example

As a running example, we're going to use a simple "sort and change case"
workflow: it merely takes a text file, sorts the lines and swaps the case of all
the letters. You can access and inspect this example workflow, its tests and
its GitHub Action through [this link](https://github.com/by-covid/gtn/tree/main/docs/submitting-life-monitor/sort-and-change-case-workflow).


## Installing the LM GitHub App

The [{LM} GitHub App](https://github.com/apps/lifemonitor) provides the most
straightforward way to use {LM} with your GitHub-hosted workflow. The app can do
the following things.

* Examine your repository and apply a series of automated *checks*.
* Suggest pull requests (PRs) to make changes or additions to bring the workflow
  repository closer to conforming to best practices.
* Open issues to let you know about problems detected by the checks:
  * in some cases, you can interact with the LifeMonitor bot through the issues,
      to provide information or issue commands.
* Register new releases/versions of the workflow with both the [LifeMonitor
  service](https://app.lifemonitor.eu/) (for test monitoring) and the
  [WorkflowHub](https://workflowhub.eu/) workflow registry.


Follow the instructions provided by the [LifeMonitor
documentation](https://www.lifemonitor.eu/lm_wft_best_practices_github_app#installation)
to complete the app installation on your repository, and:

* we recommend that you enable the LifeMonitor "Issue Checks";
* we also recommend that you enable the automatic upload to the
    [WorkflowHub](https://workflowhub.eu/).

> <tip-title>Accounts Required</tip-title>
> Note that completing the installation steps may require you to create an
> account with both LifeMonitor and WorkflowHub.
> We recommend **using [LS Login](https://lifescience-ri.eu/ls-login/)** to authenticate with both
> services.
{: .tip}

> <tip-title>GitHub Connection Required</tip-title>
> You will also need to connect your GitHub account to your LifeMonitor account.
{: .tip}

## Configure your LifeMonitor account

To fully enable the features provided by LifeMonitor you need to enable some
optional settings in the LM configuration.

### Enable e-mail notifications

LifeMonitor can send you emails to notify you of important events related to
your workflows or the workflows you've [subscribed
to](https://www.lifemonitor.eu/lm_dashboard#subscribing-to-workflows).  We
recommend performing this step to -- among other things -- receive notifications
if your workflow tests start failing, or when a release of your workflow has
been registered with the LifeMonitor test monitoring service and with the
WorkflowHub registry.

[Click
here](https://www.lifemonitor.eu/lm_test_monitoring#enabling-e-mail-notifications)
to access the LifeMonitor documentation on how to perform this step.


### Configure GitHub integration

If you [installed the LM GitHub App](#installing-the-lm-github-app) on your
workflow repository your GitHub account should already be connected to
LifeMonitor. Verify this by accessing your account's [profile
page](https://app.lifemonitor.eu/account/profile) and checking that the
**Connected Accounts** table includes an entry for GitHub and it says
**CONNECTED**; if it says "CONNECT", then click on the button and follow the
procedure.  See the [LifeMonitor
documentation](https://www.lifemonitor.eu/faq#which-external-accounts-are-linked-with-my-lifemonitor-account)
for more details.

If you want, you can now check the LifeMonitor [GitHub integration
settings](https://app.lifemonitor.eu/account/profile?currentView=githubSettingsTab)
for your account. These are the default settings which will be applied unless
overridden with a repository-specific configuration file (we'll see how in a
later section).


### Connect your WorkflowHub account to LifeMonitor

Connecting your WorkflowHub identity to LifeMonitor allows LifeMonitor to
perform specific WorkflowHub actions on your behalf (using your identity).
These actions consist in registering and updating workflow entries as well as
querying the WorkflowHub for the workflows you have previously registered.

Check the *Account* tab on your [LifeMonitor account
profile](https://app.lifemonitor.eu/account/profile): if the buttons shown
in the image below say "**connected**", you're ok; else, click on the
appropriate "**connect** button and log into the WorkflowHub with the account
you'd like LifeMonitor to use.
![Connect WorkflowHub buttons](./images/account_connections_whub-dev-disconnected.png)
The LifeMonitor documentation has [a more detailed explanation](https://www.lifemonitor.eu/faq#which-external-accounts-are-linked-with-my-lifemonitor-account).


### Enable WorkflowHub Integration for your account

Enabling registry integration for WorkflowHub enables the LM automations related
to workflow registry activities.


Go to the *Registry Integration* tab on your [LifeMonitor account
profile](https://app.lifemonitor.eu/account/profile?currentView=registrySettingsTab).
Enable the switches for both "wfhub" and "wfhubdev".  When you flip the switch,
you may be taken to the WorkflowHub authentication page; in that case, log into
the WorkflowHub with the same account credentials that you connected in the
previous step.

![LM config registry integration](./images/lm_config_registry_integration.png)


## LifeMonitor Checks

Once the LM GitHub app is installed, it starts working for you by running its automated
checks. The checks generally report one problem at a time, by opening an issue
or opening a PR with a suggested fix.  In the following subsections we'll show
the sequence of PR's and issues that are opened in our Sort and Change Case
workflow repository.

### RO-Crate metadata

LM might notice that we don't have an RO-Crate:
![Missing RO-Crate PR](./images/missing_ro_crate_pr.png)

The LM app **automatically** executes
[repo2crate](https://github.com/crs4/repo2rocrate) as recommended by [best
practices]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#best-practice-repositories-and-ro-crate) and
opens a pull request to propose the resulting RO-Crate for integration in our
repository.  Review the RO-Crate and integrate additional metadata or correct
it as necessary. Finally, merge the changes into your repository.  LifeMonitor with
automatically delete the PR branch once merged, to keep your repository tidy.
![Merge PR](./images/merge_pr.png)


### LifeMonitor configuration file

Next, LM suggests that we add a [LifeMonitor configuration
file](https://www.lifemonitor.eu/lm_wft_best_practices_github_app#configuration-file)
to our repository.  Through this file in YAML format, we can customize some aspects of
LifeMonitor's behavior and provide it with additional metadata.
![LM config PR](./images/lm_config_pr.png)

This time, we're going to make some changes to the default configuration before
accepting it.  The PR's "Conversation" tab on GitHub tells us to push changes to
a specific branch to make changes to the file:
![LM config PR branch](./images/lm_config_pr_branch.png)

So we'll go to the local copy of our workflow repository, fetch the remote and
checkout this branch:

```console
$ cd sort-and-change-case-workflow
$ git fetch origin
$ git checkout lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349
Branch 'lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349' set up to track remote branch 'lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349' from 'origin'.
Switched to a new branch 'lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349'
```

Now we can edit the file locally with our favorite text editor:

```console
$ nvim .lifemonitor.yaml
```

We're going to set the workflow name and make it public (which means that anyone
will be able to see it on LifeMonitor):

```yaml
name: "Sort and change case"
public: true
```

We're also going to enable automatic registration of workflow releases in
WorkflowHub and LifeMonitor.  We're going to edit the configuration so that:

1. workflow tag names that follow the [Semantic Versioning](https://semver.org/)
   convention are recognized as releases;
2. the WorkflowHub record is updated for each new release.

```yaml
    push:
      tags:
        - name: "*.*.*"  #  pattern to identify a release tag
          update_registries: ["wfhub"]  # Registry to be updated. "wfhub" == WorkflowHub
```

If you're just testing things, avoid dirtying your WorkflowHub collection.
Instead, point the LM app to the development instance of WorkflowHub by
specifying:

```yaml
update_registries: ["wfhubdev"]  # Registry to be updated. "wfhubdev" == WorkflowHub dev
```

Commit your changes now and push them to the repository:

```console
$ git commit -a -m "Update LifeMonitor settings"
[lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349 0924ad7] Update LifeMonitor settings
1 file changed, 22 insertions(+), 52 deletions(-)
rewrite .lifemonitor.yaml (63%)
$ git push origin
```

Now you can go back to the Pull Request and merge it.
![LM config PR merged](./images/lm_config_pr_merged.png)

### New and changing checks

The set of automated checks provided by LifeMonitor is under active development.
This means that the exact sequence of issue and PR's you see may change over
time: this is normal.  Do feel free to [open an
issue](https://github.com/crs4/life_monitor/issues) if you notice something that
doesn't work as you expect or if you have a suggestion for a new automated
check.

## Automatic workflow registration to WorkflowHub


To have your workflow's WorkflowHub entry automatically updated, you
**must** [connect your WorkflowHub (or dev.WorkflowHub)
account](#connect-your-workflowhub-account-to-lifemonitor) to LifeMonitor
and [enable WorkflowHub (or dev.WorkflowHub)
integration](#enable-workflowhub-integration-for-your-account).

## Make a workflow release

Now you can make a release of your workflow. The LifeMonitor
[configuration](#lifemonitor-configuration-file) we specified for the workflow
repository specifies that we identify release by git tags matching the
expression.  You can tag the workflow as release "0.1.0" with the git CLI:

```console
$ git tag 0.1.0
$ git push --tags origin
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:crs4/sort-and-change-case-workflow.git
* [new tag]         0.1.0 -> 0.1.0
```

The LifeMonitor now start working and automatically register the workflow with
both the LifeMonitor test monitoring functionality and the WorkflowHub registry
you enabled and selected.

> <tip-title>E-mail notifications</tip-title>
> If you enabled e-mail notifications, you should receive an email telling
> you that the registration happened and providing you with a direct link to the
> workflow on LifeMonitor (from which you'll be able to access the corresponding
> WorkflowHub entry).
> ![LM new registration](./images/lm_new_registration.png)
{: .tip}

The workflow should also appear on your [LifeMonitor
dashboard](https://app.lifemonitor.eu/dashboard).  If the GitHub Action that
tests your workflow has not been executed yet, then LM will report that the test
status is "unavailable" -- but this is normal.  After you execute your action
then the system will report its passing or failing status.

The workflow entry in LifeMonitor also has a link to the corresponding
WorkflowHub entry.  Follow the link to reach a page with your workflow's
metadata:
![LM new registration workflowhub](./images/lm_new_registration-workflowhub.png)

## Profit

Now that you've completed the setup, you'll be notified if your workflow's tests
start to fail. {LM} will also periodically try to trigger the execution of your
testing GitHub Action should there be no commits on the repository that would
trigger the tests on their own.  Also, any new workflow releases that you create
will be automatically be propagated to both LifeMonitor and WorkflowHub, keeping
your registry up to date and your workflow a little more FAIR.


topic: fair
tutorial: ro-crate-in-python
title: RO-Crate in Python
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["ro-crate"]
content_hash: 05c3024bd0a16d5d3fd86a493d4f3098
last_modified: 2026-05-27T09:01:48.603123
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-in-python/tutorial.html

content:
This tutorial will show you how to manipulate [RO-Crates](https://w3id.org/ro/crate/) in Python using the [ro-crate-py](https://github.com/ResearchObject/ro-crate-py) package. It is based on the [ro-crate-py documentation](https://github.com/ResearchObject/ro-crate-py/blob/e1218fbca595f4c33059cfe15849ee2ae9e6896b/README.md).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Let's start by installing the library via [pip](https://docs.python.org/3/installing/). Note that the name of the package is `rocrate`.

```bash
pip install rocrate
```

# Creating an RO-Crate

In its simplest form, an RO-Crate is a directory tree with an `ro-crate-metadata.json` file at the top level. This file contains metadata about the other files and directories, represented by [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html). These metadata consist both of properties of the data entities themselves and of other, non-digital entities called [contextual entities](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html). A contextual entity can represent, for instance, a person, an organization or an event.

Suppose Alice and Bob worked on a research project together, and then wrote a paper about it; additionally, Alice prepared a spreadsheet containing experimental data, which Bob then used to generate a diagram. For the purpose of this tutorial, you can just create placeholder files for the documents:

```python
import os

data_dir = "exp"
os.mkdir(data_dir)

for filename in ["paper.pdf", "results.csv", "diagram.svg"]:
    with open(os.path.join(data_dir, filename), "w") as file:
        pass
```

Let's make an RO-Crate to represent this information:

```python
from rocrate.rocrate import ROCrate

crate = ROCrate()
paper = crate.add_file("exp/paper.pdf", properties={
    "name": "manuscript",
    "encodingFormat": "application/pdf"
})
table = crate.add_file("exp/results.csv", properties={
    "name": "experimental data",
    "encodingFormat": "text/csv"
})
diagram = crate.add_file("exp/diagram.svg", dest_path="images/figure.svg", properties={
    "name": "bar chart",
    "encodingFormat": "image/svg+xml"
})
```

We've started by adding the data entities. Now we add contextual entities representing Alice and Bob:

```python
from rocrate.model.person import Person

alice_id = "https://orcid.org/0000-0000-0000-0000"
bob_id = "https://orcid.org/0000-0000-0000-0001"
alice = crate.add(Person(crate, alice_id, properties={
    "name": "Alice Doe",
    "affiliation": "University of Flatland"
}))
bob = crate.add(Person(crate, bob_id, properties={
    "name": "Bob Doe",
    "affiliation": "University of Flatland"
}))
```

At this point, we have a representation of the various entities. Now we need to express the relationships between them. This is done by adding properties that reference other entities:

```python
paper["author"] = [alice, bob]
table["author"] = alice
diagram["author"] = bob
```

You can also add whole directories together with their contents. In an RO-Crate, a directory is represented by the `Dataset` entity:

```python
logs_dir = os.path.join(data_dir, "logs")
os.mkdir(logs_dir)

for filename in ["log1.txt", "log2.txt"]:
    with open(os.path.join(logs_dir, filename), "w") as file:
        pass

logs = crate.add_dataset("exp/logs")
```

Finally, we serialize the crate to disk:

```python
crate.write("exp_crate")
```

This should generate an `exp_crate` directory containing copies of all the files we added and an `ro-crate-metadata.json` file containing a [JSON-LD](https://json-ld.org) representation of the metadata. Note that we have chosen a different destination path for the diagram, while the paper and the spreadsheet have been placed at the top level with their names unchanged (the default).

Some applications and services support RO-Crates stored as archives. To save the crate in zip format, you can use `write_zip`:

```python
crate.write_zip("exp_crate.zip")
```

> <comment-title>How `rocrate` handles the contents of `exp/logs`</comment-title>
>
> Exploring the `exp_crate` directory, we see that all files and directories contained in `exp/logs` have been added recursively to the crate. However, in the `ro-crate-metadata.json` file, only the top level Dataset with `@id` `"exp/logs"` is listed. This is because we used `crate.add_dataset("exp/logs")` rather than adding every file individually. There is no requirement to represent every file and folder within the crate in the `ro-crate-metadata.json` file - in fact, if there were many files in the crate it would be impractical to do so.
>
> If you do want to add files and directories recursively to the metadata, use `crate.add_tree` instead of `crate.add_dataset` (but note that it only works on local directory trees).
{: .comment}

## Appending elements to property values

What ro-crate-py entities actually store is their JSON representation:

```python
paper.properties()
```

```json
{
  "@id": "paper.pdf",
  "@type": "File",
  "name": "manuscript",
  "encodingFormat": "application/pdf",
  "author": [
    {"@id": "https://orcid.org/0000-0000-0000-0000"},
    {"@id": "https://orcid.org/0000-0000-0000-0001"},
  ]
}
```

When `paper["author"]` is accessed, a new list containing the `alice` and `bob` entities is generated on the fly. For this reason, calling `append` on `paper["author"]` won't actually modify the `paper` entity in any way. To add an author, use the `append_to` method instead:

```python
donald = crate.add(Person(crate, "https://en.wikipedia.org/wiki/Donald_Duck", properties={
  "name": "Donald Duck"
}))
paper.append_to("author", donald)
```

Note that `append_to` also works if the property to be updated is missing or has only one value:

```python
for n in "Mickey_Mouse", "Scrooge_McDuck":
    p = crate.add(Person(crate, f"https://en.wikipedia.org/wiki/{n}"))
    donald.append_to("follows", p)
```

## Adding remote entities

Data entities can also be remote:

```python
input_data = crate.add_file("http://example.org/exp_data.zip")
```

By default the file won't be downloaded, and will be referenced by its URI in `ro-crate-metadata.json`:

```json
{
  "@id": "http://example.org/exp_data.zip",
  "@type": "File"
}
```

If you add `fetch_remote=True` to the `add_file` call, however, the library (when `crate.write` is called) will try to download the file and include it in the output crate.

Another option that influences the behavior when dealing with remote entities is `validate_url`, also `False` by default: if it's set to `True`, when the crate is serialized, the library will try to open the URL to add / update metadata such as the content's length and format.

## Adding entities with an arbitrary type

An entity can be of any type listed in the [RO-Crate context](https://www.researchobject.org/ro-crate/1.1/context.jsonld). However, only a few of them have a counterpart (e.g., `File`) in the library's class hierarchy, either because they are very common or because they are associated with specific functionality that can be conveniently embedded in the class implementation. In other cases, you can explicitly pass the type via the `properties` argument:

```python
from rocrate.model.contextentity import ContextEntity

hackathon = crate.add(ContextEntity(crate, "#bh2021", properties={
    "@type": "Hackathon",
    "name": "Biohackathon 2021",
    "location": "Barcelona, Spain",
    "startDate": "2021-11-08",
    "endDate": "2021-11-12"
}))
```

Note that entities can have multiple types, e.g.:

```
    "@type" = ["File", "SoftwareSourceCode"]
```

# Consuming an RO-Crate

An existing RO-Crate package can be loaded from a directory or zip file:

```python
crate = ROCrate('exp_crate')  # or ROCrate('exp_crate.zip')
for e in crate.get_entities():
    print(e.id, e.type)
```

```
./ Dataset
ro-crate-metadata.json CreativeWork
paper.pdf File
results.csv File
images/figure.svg File
https://orcid.org/0000-0000-0000-0000 Person
https://orcid.org/0000-0000-0000-0001 Person
...
```

The first two entities shown in the output are the [root data entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html) and the [metadata file descriptor](https://www.researchobject.org/ro-crate/1.1/metadata.html), respectively. The former represents the whole crate, while the latter represents the metadata file. These are special entities managed by the `ROCrate` object, and are always present. The other entities are the ones we added in the [section on RO-Crate creation](#creating-an-ro-crate). As shown above, `get_entities` allows to iterate over all entities in the crate. You can also access only data entities with `crate.data_entities` and only contextual entities with `crate.contextual_entities`. For instance:

```python
for e in crate.data_entities:
    author = e.get("author")
    if not author:
        continue
    elif isinstance(author, list):
        print(e.id, [p.get("name") for p in author])
    else:
        print(e.id, repr(author.get("name")))
```

```
paper.pdf ['Alice Doe', 'Bob Doe']
results.csv 'Alice Doe'
images/figure.svg 'Bob Doe'
```

You can fetch an entity by its `@id` as follows:

```python
article = crate.dereference("paper.pdf")  # or crate.get("paper.pdf")
```

# Command Line Interface

> <comment-title>Jupyter Notebook users: switch to a terminal</comment-title>
> The code cells in this section use Unix shell commands, which can't be run within a notebook. Open a Unix/Linux terminal to follow along.
{: .comment}

`ro-crate-py` includes a hierarchical command line interface: the `rocrate` tool. `rocrate` is the top-level command, while specific functionalities are provided via sub-commands. Currently, the tool allows to initialize a directory tree as an RO-Crate (`rocrate init`) and to modify the metadata of an existing RO-Crate (`rocrate add`).

```console
$ rocrate --help
Usage: rocrate [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  add
  init
  write-zip
```

## Crate initialization

The `rocrate init` command explores a directory tree and generates an RO-Crate metadata file (`ro-crate-metadata.json`) listing all files and directories as `File` and `Dataset` entities, respectively.

```console
$ rocrate init --help
Usage: rocrate init [OPTIONS]

Options:
  --gen-preview
  -e, --exclude CSV
  -c, --crate-dir PATH
  --help                Show this message and exit.
```

The command acts on the current directory, unless the `-c` option is specified. The metadata file is added (overwritten if present) to the directory at the top level, turning it into an RO-Crate.

## Adding items to the crate

The `rocrate add` command allows to add files, datasets (directories), workflows, and other entity types (currently [testing-related metadata](https://crs4.github.io/life_monitor/workflow_testing_ro_crate)) to an RO-Crate:

```console
$ rocrate add --help
Usage: rocrate add [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  dataset
  file
  test-definition
  test-instance
  test-suite
  workflow
```

Note that data entities (e.g., workflows) must already be present in the directory tree: the effect of the command is to register them in the metadata file.

## Example

To run the following commands, we need a copy of the ro-crate-py repository:

```bash
git clone https://github.com/ResearchObject/ro-crate-py
```

Navigate to the following directory in the repository we just cloned:

```bash
cd ro-crate-py/test/test-data/ro-crate-galaxy-sortchangecase
```

This directory is already an RO-Crate. Delete the metadata file to get a plain directory tree:

```bash
rm ro-crate-metadata.json
```

Now the directory tree contains several files and directories, including a Galaxy workflow and a Planemo test file, but it's not an RO-Crate anymore, since there is no metadata file. Initialize the crate:

```bash
rocrate init
```

This creates an `ro-crate-metadata.json` file that lists files and directories rooted at the current directory. Note that the Galaxy workflow is listed as a plain `File`:

```json
{
  "@id": "sort-and-change-case.ga",
  "@type": "File"
}
```

To register the workflow as a `ComputationalWorkflow`, run the following:

```bash
rocrate add workflow -l galaxy sort-and-change-case.ga
```

Now the workflow has a type of `["File", "SoftwareSourceCode", "ComputationalWorkflow"]` and points to a `ComputerLanguage` entity that represents the Galaxy workflow language. Also, the workflow is listed as the crate's `mainEntity` (this is required by the [Workflow RO-Crate profile](https://w3id.org/workflowhub/workflow-ro-crate/1.0), a subtype of RO-Crate which provides extra specifications for workflow metadata).

To add files or directories after crate initialization:

```bash
cp ../sample_file.txt .
rocrate add file sample_file.txt -P name=sample -P description="Sample file"
cp -r ../test_add_dir .
rocrate add dataset test_add_dir
```

The above example also shows how to set arbitrary properties for the entity with -P. This is supported by most `rocrate add` subcommands.


topic: fair
tutorial: fair-access
title: Access
difficulty: intermediate
time_estimation: 50M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 0aed678f65b1e64de2a3ee9b3fb68c29
last_modified: 2026-05-15T22:28:13.143559
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-access/tutorial.html

content:
Data access including levels of access are described.  Learners will be able to illustrate data access in terms of the FAIR Principles using companion terms including communications protocol and authentication.  Learners will also be able to interpret the data usage licence associated with different data sets

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data access and the FAIR Principles
Data access relates to the following 5 FAIR Principles (Table 4.1).  We will discuss and signpost these in this episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a globally unique and persistent identifier<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | **A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>A2. metadata are accessible, even when the data are no longer available** |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>**R1.1. (meta)data are released with a clear and accessible data usage license**<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards        


Table 4.1: The 15 FAIR Guiding Principles.  Principles relating to data access are highlighted in **black**.

# What is data access?
Making data accessible means it can be made available to use by both humans and computers, though FAIR data does not necessarily mandate that all data is openly accessible and we discuss this in a minute.

As outlined in the Accessibility Principles (meta)data must be retrievable by their identifier using a standardised communication protocol (FAIR Principle A1) which is open, free and universally implementable (FAIR Principle A1.1).

A standardised communication protocol is something like http(s) or FTP that allows data to be requested and downloaded, for example, by clicking on a link on a webpage.  Simply put, **a protocol** is a method that connects two computers and ensures secure data transfer.  Web browsers such as Firefox and Chrome can use the http(s) communication protocol since it is universally implementable, open and free.  

It is important to note that (meta)data access is not limited to humans clicking links on webpages. For a machine ‘user’, examples of accessing data include the use of an ‘application programming interface’ (API), and Unix command line, using  wget and [curl](https://curl.se/docs/manpage.html).

We mentioned above that FAIR data does not necessarily mandate open data.  Commonly, this relates to controlled access, which is discussed later in this Episode, but in the exercise below we investigate a dataset that has been deleted where it has been previously accessible.  FAIR states: _“metadata are accessible even when the data are no longer available”_ (FAIR Principle A2) which is one of the few principles that relate solely to metadata (and not data).  This means data can be deleted from its original online location at any time, but its original metadata must remain accessible.  PIDs (usually connected through URLs) to the original data must remain live, though the record may, for example, change to display metadata only. This is useful where originally referenced data changes over time, or becomes obsolete or deprecated; a record of the original metadata, and where appropriate reasons for its removal, or redirection to updated records, provide a provenance trail from the original data, which may have been referenced, for example, in publications, where it is not possible to update the PID.  


> <question-title></question-title>
>
> The [Protein Data Bank in Europe](https://www.ebi.ac.uk/pdbe/) (PDBe) is a searchable repository of biological macromolecular structures.  Please take a look at the following record that has been retired: [1ins](https://www.ebi.ac.uk/pdbe/entry/pdb/1ins).  You will see that the crystal structure is no longer available, but what metadata is available?
>
> > <solution-title></solution-title>
> >
> > Metadata includes the original citation, an explanation of why the record is no longer available and a redirection to the replacement  entry.
> >
> {: .solution}
{: .question}

# Types of data access

Where restricted access is required (for example, sensitive data or data subject to intellectual property) generally only the recommended parts are restricted, whereas associated metadata and non-sensitive data are openly accessible.  As part of the FAIR Principles, terms of access need to be stated, usually as part of a data licence (FAIR Principle R1.1).  Remember, FAIR data is **as open as possible, and as closed as necessary**.
Resources for working with sensitive data are given at the end of this Episode.

There are four types of data access as described in [RDMkit](https://rdmkit.elixir-europe.org/sharing#what-should-be-considered-for-data-sharing):

**Open access**: Anyone can access the data, and use it for any purpose.

**Registered access or authentication procedure**: researchers are required to register and authenticate to have the right to access the data (login and password)

**Controlled access or Data Access Committees (DACs)**: researchers will apply for access, and their application reviewed by a data access committee

**Access upon request (not recommended)**: a researcher provides their contact details for access. Contact details should be provided in the metadata which will be publicly available.

Any access requiring login and password makes use of _“[a protocol] allowing for an authentication and authorisation procedure, where necessary”_ (FAIR Principle A1.2).  Commonly for authentication, a researcher will be assigned a unique ID, or the system may support sign-in with an ORCID ID, which is the case of many data repositories including [Zenodo](https://zenodo.org/).  In some cases, there may be an option to use a Google account or institutional email to sign in, and many infrastructures also support ‘single sign in’.

# Data usage licence

A data usage licence describes the legal rights on how others use your data.  When you publish your data, you should describe clearly in what capacity your data can be used. 

There are many types of licences that can be used, including the  [MIT licence](https://opensource.org/license/mit/) (for software) or the commonly used [Creative Commons licences](https://creativecommons.org/licenses/) (a selectable collection of licences). These licences provide precise descriptions of the rights to data use, where the latter defines rights for sharing, adapting and commercialisation. Open access data usually carries the [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) or [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) licence permitting open sharing and adaptation, even for commercial purposes.  The licence is applied by adding the licence declaration to the data similar to this training page.  Take a look at the banner at the bottom of this page.  It states: Licensed under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) 2018–2023 by [The Carpentries](https://carpentries.org/). 

# Making sensitive data accessible

Controlled access is often afforded to sensitive data or commonly any data that could potentially harm, see [RDMkit](https://rdmkit.elixir-europe.org/sensitive_data#is-your-data-sensitive) for a full definition.

Where access is granted, sensitive data is often de-identified, meaning that identifying (meta) data is removed or reassigned, leaving the analytics-appropriate component.

For example, a participant’s name can be removed from a questionnaire, and their home address can be substituted for the name of the town they live in.  This anonymises the data since the participant can no longer be located.  Note though that other information in the questionnaire could compromise this.  If other data reveal that the participant won the town’s 10K road race in 2023, we could potentially identify the individual using the name of the town and an online search.  If more information in the questionnaire states that the participant has a rare disease, we are broaching disclosure of sensitive, personal data.

Though people refer to anonymisation when de-identifying (meta)data, often they mean pseudonymisation.  Data anonymisation and pseudonymisation are slightly different.

**Data anonymisation** is the process of irreversibly de-identifying personal data such that an individual cannot be identified by anyone, including the study team and the individual themselves.  If data are anonymised, no one can link data back to the subject. 

**Pseudonymisation** is more commonly used.  It is a process where identifying-fields are replaced by artificial identifiers called pseudonyms or pseudonymised IDs.
Commonly, a person’s name or medical ID will be replaced with a unique participant ID within the study.
Pseudonymisation ensures no one can link data back to the individual, apart from nominated members of the study team who will be able to link pseudonyms to identifying information, such as medical records.


> <question-title></question-title>
>
> Use RDMkit's guidelines on [sensitive data](https://rdmkit.elixir-europe.org/sensitive_data) to familiarise yourself further on de-identification of data. What further training can you identify?
>
> > <solution-title></solution-title>
> >
> > At the bottom of the page, under “Training”, useful resources are given.  The [TeSS training portal](https://tess.elixir-europe.org/) permits users to search for courses, events, videos and other learning material for data in the life sciences.
> >
> {: .solution}
{: .question}


# Useful resources
- More about Data Access and Sharing: [RDMkit](https://rdmkit.elixir-europe.org/sharing), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/accessibility.html)
- Definition and examples of Standardised Communication protocol using authentication: [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/)
- More about Data licensing: [RDMkit](https://rdmkit.elixir-europe.org/licensing), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/reusability/ATI-licensing.html)
- The [Creative Commons licences](https://creativecommons.org/choose/)
- Working with sensitive data: [EBI training](https://www.ebi.ac.uk/training/events/working-sensitive-data/)


topic: fair
tutorial: dataplant-arcs
title: DataPLANT ARCs
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["plants", "fair", "data stewardship"]
content_hash: 0e826ef4d89389dd0d3395175c4fc69c
last_modified: 2026-05-27T09:01:48.591068
zenodo_link: https://zenodo.org/records/13970369
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/dataplant-arcs/tutorial.html

content:
In this tutorial we will guide you through the process of creating your ARC (Annotated Research Context)
ARCs are a part of your Research Data Management (RDM) strategy, and are aimed specifically at plant research
data. ARCs are an [RO-crates](https://www.researchobject.org/ro-crate/) implementation developed by
[DataPLANT (NFDI4Plants)](https://www.nfdi4plants.org/). ARCs allow you to bundle and annotate
your research data in a standardized and [FAIR](https://www.go-fair.org/fair-principles/) (Findable, Accessible,
Interoperable, Reusable) manner.

![ARC concept showing bundling of experimental data, computation, and annotations](images/arcs-intro.png)


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Getting set up for the tutorial

For this tutorial you need three things:
 1. An account on [DataHUB](https://git.nfdi4plants.org)
 2. The [ARCitect](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/) tool installed
 3. Example data to fill your ARC with

Below we will walk you through each of these 3 setup steps.

## Create a DataHUB account

DataHUB is a GitLab instance designed to hosts ARCs. Here you can work collaboratively to create your ARC, and once you are ready to publish your ARC, you can do this from here as well.

> <hands-on-title> Create a DataHUB account </hands-on-title>
>
> 1. Already have an account? Please [log in](https://git.nfdi4plants.org/])
> 2. New to DataHUB? Create an account: [https://git.nfdi4plants.org/](https://git.nfdi4plants.org/)
>    - If you are not familiar with Git or GitLab that is ok, ARCitect will take care of syncing to DataHub behind the scenes
>
{: .hands_on}


## Download demo data

In this tutorial we will create an ARC for an example research data. In this example investigation we grew Talinum plants under
drought conditions, with watered plants as control. Under these conditions the plants switch their type of photosynthesis
(water: C3, drought: CAM). To study this change, we profiled gene expression (Transcriptomics) using RNASeq, and metabolites
(Metabolomics) via GC-MS (Gas-Chromatography Mass Spectrometry)


> <hands-on-title> Download example ARC data </hands-on-title>
>
> 1. Download the zip file from Zenodo (<2MB) to your machine
>
>    ```
>    https://zenodo.org/records/13970369/files/GTN_ARC_tutorial_data.zip
>    ```
>
> 2. Unzip the file
>    - You can put the files wherever is most convenient for you
>
> 3. Have a quick look at the files and folders
>    - This example data mimics how you might have your project data organised on your computer
>    - In the next part we will show how to organise this data in an ARC-compatible way
>
{: .hands_on}


## Install ARCitect

Now we will install the ARCitect tool. This tool will help you create and fill your ARC.


> <hands-on-title> Install the ARCitect tool on Linux  </hands-on-title>
>
> 1. Follow the installation instructions on the [DataPLANT knowledgebase](https://nfdi4plants.org/nfdi4plants.knowledgebase/) for your OS:
>    - [Windows Installation Instructions](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/installation/#tab-21)
>    - [macOS Installation Instructions](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/installation/#tab-22)
>    - [Linux Installation Instructions](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/installation/#tab-23)
>
>    > <comment-title> ARCitect version </comment-title>
>    >
>    > This tutorial uses version {{page.extra.arcitect_version}} of ARCitect. Since this tool is still
>    > under rapid development, a newer version of ARCitect may already be available. Feel free to install
>    > the latest version of ARCitect, but be aware that some screenshots and instructions may be outdated.
>    >
>    {: .comment}
>
>
>
> 2. Start ARCitect to verify that everything is installed correctly. You should see a screen like this:
>
>    ![screenshot of the ARCitect interface after startup](images/arcitect-home.png)
>
{: .hands_on}


# Initialize your ARC structure

Now that you have everything set up for the course, we can start creating our ARC. First we will organize our example data into the ARC structure, then we will add structured metadata describing our experiments and data.


## The ISA data model

ARCs build on the [ISA Abstract Model](https://isa-specs.readthedocs.io/en/latest/isamodel.html)
for metadata annotation.

The ISA model comes with a hierarchy (ISA: Investigation - Study - Assay)
that aligns well with most projects in (plant) biology labs. It allows grouping multiple assays (measurements) to one study,
and multiple studies to one investigation.

![Overview of the ISA model](images/isa-model.png "Image source left panel: [isa-tools.org](https://isa-tools.org/format/specification.html)")

Your ARC has one isa.investigation.xlsx workbook at its root, where metadata about the investigataion is captured. Similarly, each study or assay that you add to your ARC contains one isa.study.xlsx or isa.assay.xlsx, respectively.


> <comment-title> Before you start </comment-title>
>
> Before creating your own ARC, invest some time to think about the following questions.
>
> - What is my **investigation**?
> - What is my **study**?
> - Which **assay** did I perform?
> - What is my (raw) **dataset**?
> - What **protocols** did I use?
>
{: .comment}


## The ARCitect interface

The ARCitect window consists of 4 main parts; on the left is the menu panel, next to it is a panel displaying the ARC folder structure, then the main panel where we will configure our ARC, and on the righ is (optionally) a help panel.

![Screenshot of the ARCitect graphical user interface](images/ARCitect-GUI.png "Overview of ARCitect interface. A) menu panel, B) ARC folder structure, C) Main panel, D) Help panel (off by default).")

Let's have a closer look at the menu panel:

![screenshot of the ARCitect menu panel](images/arcitect-sidebar.png)

This menu bar allows you to:
1. **Login**: Log in to DataHUB (required to sync your ARC to DataHUB)
2. **New ARC**: Create a new ARC
3. **Open ARC**: Open an existing ARC from your machine
4. {% icon download-cloud %} **Download ARC**: From DataHUB to your machine (e.g. from a colleague or published)
5. {% icon save %} **Save ARC**: Save your changes to your machine
6. **Explorer**: Open the local ARC folder on your computer
7. **Commit**: prepare your ARC changes for sync to DataHUB
8. **DataHUB sync**: push your comitted changes to DataHUB
9. **History**: Show previous versions of your ARC
10. **Validation**: Configure validation (checks) for our ARC
11. **Services**: Check on the status of ARC services (e.g. is DataHUB down for maintenance?)
12. **Settings**: For example turn on tool tips or the help panel



## Creating a new ARC

Let's start by creating a new ARC.

> <hands-on-title> Create a new ARC in ARCitect </hands-on-title>
>
> 1. In the left-hand panel of ARCitect, select **New ARC**
>    - Navigate to a location on your machine where you would like to create the ARC folder
>    - Choose a descriptive name for your ARC, e.g. `TalinumPhotosynthesis`
>
>    > <comment-title>What's in a name?</comment-title>
>    >
>    > By default, the name give your ARC here will be used for:
>    >  - the ARC folder on your machine
>    >  - the ARC repository on DataHUB at `https://git.nfdi4plants.org/<YourUserName>/<YourARC>` (see next steps)
>    >  - the identifier for your investigation
>    >
>    > **Note:** Make sure that no ARC exists at `https://git.nfdi4plants.org/<YourUserName>/<YourARC>`. Otherwise you will sync to that ARC.
>    {: .comment}
>
> 2. If all went well, you should see the folder structure that was created for your ARC:
>    ![screenshot of the ARC folder panel after initializing our ARC](images/new-arc.png)
>
> 3. Click on the **Explorer** button in the menu to see the files created on your machine
>
{: .hands_on}

The basic ARC structure is initialized for you, and looks something like this:

```bash
my-arc
├── studies
│   ├── study1              # subfolder for each study
│   │   ├── protocols       # folder for protocol documents
│   │   ├── resources       # folder for any other supporting information and files
│   │   isa.study.xlsx      # study-level metadata (e.g. sample sheet, culturing/harvesting conditions)
│   │   README.md           # Free-text description of your study
│   ├── assay2
│   │   ├── ...
├── assays
│   ├── assay1              # subfolder for each assay
│   │   ├── dataset         # assay result files (e.g. sequencing files)
│   │   ├── protocols       # assay protocol files
│   │   isa.assay.xlsx      # assay-level metadata (e.g. instrument)
│   │   README.md
│   ├── assay2
│   │   ├── ...
├── runs                    # analysis results
│   ├── ...
├── workflows               # analysis workflows (CWL, or any other scripts/code used)
│   ├── ...
├── isa.investigation.xlsx  # investigation-level metadata
└── README.md
```


## Add investigation-level metadata

We will start by adding some basic data about our ARC. We do this in the investigation-level metadata.

> <hands-on-title>Adding investigation information</hands-on-title>
>
> 1. In the ARC folder panel (second panel from the left), click on the investigation
>    - This is the root of the ARC folder structure (e.g. *TalinumPhotosynthesis*)
>    - This will bring up the investigation information in the main panel
>    - Notice that the Identifier has already been filled in for you (but you can still change it)
>    - Behind the scenes this information will be saved as an Excel sheet `isa.investigation.xlsx`
>
>    ![screenshot of the investigation metadata screen](images/investigation-empty.png)
>
> 2. **Add a title** for your investigation
>    - This should be a concise but descriptive title
>    - Tip: think of what you would name your publication based on this investigation
>    - For this tutorial, you could set your title to `Effect of Drought on Talinum Photosynthesis`
>
> 3. **Add a description** for your investigation
>   - Here you can describe your main research questions and methods in a few sentences or paragraphs.
>   - Tip: think of this as a short abstract like you would provide in a publication
>   - For this tutorial you can add any text here
>
> 4. **Add an ARC Contact** (you)
>    - Click on the **{% icon plus %} (plus) icon** under Contacts
>    - Click on the **{% icon dropdown %} (dropdown) icon** on the right of the contact to expand it
>    - Fill in at least your First name, Last name, and email.
>    - **Tip:** if you have an [ORCiD](https://orcid.org) ID, fill that in and hit "Search" to autofill these fields
>
> 5. Your investigation tab should look something like this
>
>    ![screenshot of investigation metadata filled in](images/investigation-filled.png)
>
{: .hands_on}


## Saving your ARC

It is a good idea to frequently save your work. There are 2 levels of saving your ARC; you can:
1. Save it locally (to your computer)
2. Save it online to DataHUB (syncing)

By default your ARC on DataHUB is private to you, but you have the option of sharing it with your collaborators.

> <comment-title>Why and how often to sync to DataHUB?</comment-title>
>
> Syncing your ARC to DataHUB frequently has several advantages:
>  - It makes it easy to collaborate with colleagues on your ARC
>  - It is a backup of your ARC in case anything happens to your computer
>  - It is version controlled. This means you can always go back to any previous version your ARC if needed
>
> We advise you to save your ARC to your local machine as often as you can, and sync to DataHUB every time you have
> completed a unit of work (e.g. "added data files", "added culturing metadata", "added protocol documents" etc), this
> makes it easy for others (and future you) to see what you have changed.
>
{: .comment}

Since we have just completed our first unit of work (initialized our ARC and added basic investigation metadata), let's save and sync our ARC to DataHUB now.

> <hands-on-title>Save and Sync our ARC</hands-on-title>
>
> 1. Save locally simply by hitting the **Save ARC** button on the left
>    - That's it! Too easy!
>    - Do this as often as possible to avoid losing work
>
> 2. Click **Login** at the top of the menu panel
>    - Select `git.nfdi4plant.org` as your host
>    - Click the **Login** button
>    - Since you already logged into DataHub via your browser earlier, ARCitect should now automatically log you in
>
> 3. To check that you are successfully logged in, the Login button should now be replace with your DataHub username
>    ![screenshot of the menu panel as it looks when you are logged in](images/arcitect-loggedin.png)
>
> 4. Click on **Commit** in the menu
>    - This screen will show you all files that have chanced since your last sync
>    - You can choose whether to sync all files, or only some
>    - Choosing which files to sync is called "committing" those file.
>    - In our case only one file has changed, `isa.investigation.xlsx`. By default it is already selected to be committed
>
> 5. Add a **commit message**
>    - in a few words, describe your changes
>    - this will help you and your colleagues
>    - For example, for this changes you can write `added investigation metadata`
>    - You can leave the rest of the fields to the default values
>
>    ![screenshot of the commit menu](images/arcitect-commit.png)
>
> 6. Click the **Commit** button
>    - There will be a confirmation screen like the one below
>    - Click the **Ok** button
>
>    ![Screenshot of the commit confirmation dialogue window](images/arcitect-commit-confirmation.png)
>
> 7. Click on **DataHUB Sync** in the menu
>
>    ![Screenshot of the DataHUB sync menu](images/arcitect-sync.png)
>
> 8. Click on **+ (plus)** to add a 'remote' (DataHUB location to save your ARC)
>    - you only have to do this once
>    - you can keep all the default settings
>    - Click **Add**
>
>    ![screenshot of the remote adding dialogue window](images/arcitect-add-remote.png)
>
> 9. Click **Push** to sync from your computer to DataHUB
>    - **Pull** will do the reverse, it will 'pull' in changes from DataHUB into your local copy of the ARC,
>      for example if a collaborator made changes, or you worked on it from a different machine.
>
> 10. Check that the push was successfull
>     - Navigate to `https://git.nfdi4plants.org/YourUsername/TalinumPhotosynthesis`
>     - Replace 'YourUsername' with your DataHUB username in the URL
>     - If you used a different Identifier for your ARC, replace 'TalinumPhotosynthesis' with your ARC Identifier in the URL
>     - You should see your ARC on DataHUB:
>
>     ![screenshot of your new ARC on DataHUB](images/datahub-firstpush.png)
>
{: .hands_on}

Congrats! You have initialized, saved, and synced your ARC! Now let's continue with the good part, actually filling your ARC.


# Structure your data in the ARC

## Think about your data

Next we will divide our research into studies and assays. As a rule of thumb, create a study for every research question, and an assay for every measurement you performed (e.g. sequencing, mass-spec, imaging). An investigation can have one study, or several. A study can have one or more assays, and each assay can be associated with one or more studies.


> <hands-on-title> Evaluate the example data </hands-on-title>
>
> 1. Recall our example research:
>
>    *In this example investigation we grew Talinum plants under drought conditions, with watered plants as control.
>    Under these conditions the plants switch their type of photosynthesis (water: C3, drought: CAM). To study this
>    change, we profiled gene expression (Transciptomics) using RNASeq, and metabolites (Metabolomics) via GC-MS
>    (Gas-Chromatography Mass Spectrometry)*
>
> 2. Have a look at the demo data we downloaded
>
> > <question-title> Which studies and assays do we have? </question-title>
> >
> > 1. Which studies do we have?
> > 2. Which assays did we perform?
> >
> > > <solution-title></solution-title>
> > > 1. We had one research question, the effect of drought on photosynthesis, so we will create 1 study
> > > 2. We performed 2 measurements, RNA sequencing and GC-MS, so we will create 2 assays in our ARC
> > {: .solution}
> {: .question}
{: .hands_on}

## Create a study

Let's create our study now.

> <hands-on-title>Create an ARC study</hands-on-title>
>
> 1. Click on the **{% icon plus %} (plus) icon** next to the studies folder
> 2. Provide a study identifier, e.g. `talinum_drought`
>    ![screenshot of the study creation dialogue window](images/arcitect-new-study.png)
> 3. Click **{% icon plus %} New Study** button to confirm
> 4. **Expand** the new study folder that was created in your ARC
>    - you should see the protocols and resources folders created
>      ![screenshot of the ARC structure after adding the new study](images/arcitect-new-study-files.png)
{: .hands_on}

As we did for the investigation, we can also add some basic study information

> <hands-on-title>Add basic study metadata</hands-on-title>
>
> 1. Click on your study `talinum_drought`
> 2. The main panel will now show study metadata, here you can add basic information about your study
>    such as description, contacts, associated publications.
>
>    ![screenshot of the study metadata screen](images/arcitect-study-metadata.png)
>
> 3. **Add a description** for the study
> 4. **Add yourself** as a contact
>    - Remember that you can use your ORCiD ID for a fast way to fill this data
> 5. **Add a publication**
>    - Normally you would add the paper you published based on the data in this ARC
>    - For this tutorial, add any publication (e.g. one of yours, or one you find interesting)
>    - The easiest way to do this, is to add the **DOI link** or **Pubmed ID**, then the information will be autofilled
>    - If you don't have a publication in mind, try one of the following papers:
>      - Pubmed ID: `7825135`
>      - DOI: `https://doi.org/10.1136/bmj.331.7531.1498`
>
>    ![Example of study metadata filled](images/arcitect-study-metadata-filled.png)
{: .hands_on}

Since we have once again completed a unit of work, let's save and sync our ARC again!

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
>    - **Save ARC** button in menu
> 2. Sync your ARC to DataHUB
>    - **DataHub Sync** button in menu
>    - Commit all changed files
>    - Provide a good commit message (e.g. `added drought study`)
{: .hands_on}


## Create an assay

Next, we will create our 2 assays. Recall that we performed two measurements/assays, RNA sequencing, and metabolomics.

> <hands-on-title>Create the RNA-seq assay</hands-on-title>
>
> 1. Click on the **{% icon plus %} (plus) icon** next to the assays folder
> 2. Give your assay a name, e.g. `rnaseq`
> 3. In **measurement type** field, choose `mRNA Sequencing`
>    - this field is **ontology based**
>    - as soon as you start typing, suggetions for known terms will pop up
>    - Notice the TSR and TAN fields are automatically filled in once you select a term. These fields help make your ARC machine readable.
>
> 4. Similarly, set the  **Technology Type** to `Next-generation Sequencing`
>
>    > <question-title></question-title>
>    > Look through our demo data, can you find out which sequencing machine was used?
>    > > <solution-title></solution-title>
>    > > In the example data, we've got a **methods** folder where we've stored notes on our assays,
>    > > In the file called **illumina_libraries** we see that the sequencer we used was a `Illumina HiSeq 2000`
>    > {: .solution}
>    {: .question}
>
> 5. Under **Technology platform**, fill in `Illumina HiSeq 2000`
>
>    ![screenshot of assay metadata filled](images/arcitect-assay.png)
>
{: .hands_on}


Awesome, now let's repeat the process for our metabolomics assay.

> <hands-on-title>Create the metabolomics assay</hands-on-title>
>
> 1. Create a new assay called `metabolomics`
> 2. Under **measurement type** choose `Gas Chromatography Mass Spectrometry`
> 3. We can leave the Technology type/platform fields empty for now
>
>   ![screenshot of metabolomics assay](images/arcitect-assay2.png)
{: .hands_on}


And once again, let's save our ARC and sync it to DataHUB.

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
>    - **Save ARC** button in menu
> 2. Sync your ARC to DataHUB
>    - **DataHub Sync** button in menu
>    - Commit all changed files
>    - Provide a good commit message (e.g. `added rnaseq and metabolomics assays`)
{: .hands_on}



# Adding files to your ARC

As our next step, let's add the files we have in our demo folder to our ARC.

## Add protocol documents

> <hands-on-title>Adding protocols</hands-on-title>
>
> 1. Open the folder with demo data on your computer
> 2. Look through the **methods** folder
>    - This contains files with notes about methods we used in our research
>    - In our ARC, these usually go in **protocols** folders
>    - Studies and assays both have protocols folders
>
>    > <question-title></question-title>
>    >
>    > Where should each of the 4 files in the methods folder go in our ARC?
>    >
>    > > <solution-title></solution-title>
>    > > - `plant_material.txt`: contains a general sample sheet, and should go in the study-level protocol folder
>    > > - `RNA_extraction.txt` and `Illumina_libraries.txt` are specific to the RNA sequencing assay, so should go in the protocols folder for the rnaseq assay
>    > > - `metabolite_extraction.txt` is specific to our metabolomics assay, so should go in the metabolomics protocol folder
>    > {: .solution}
>    {: .question}
>
> 3. There are two ways to add files to your ARC, we will do both now, in the future you can choose whichever option you prefer
>
> 4. **Adding files via ARCitect**
>    - Expand the `studies` folder
>    - Expand the `talinum_drought` folder
>    - Right-click on the `protocols` folder
>    - Choose `Import files` option
>    - Browse to the demo data and choose the `methods/plant_material.txt` file
>
>    ![screenshot of menu to add documents](images/arcitect-import-files.png)
>
> 5. **Adding files via file explorer**
>    - Click on `Explorer` in the left-hand menu
>    - Open the `assays/metabolomics/protocols` folder of your ARC
>    - In another window, open the demo data folder
>    - Copy or drag the `metabolite_extraction.txt` file from the demo data folder to your ARC folder
>
> 6. **Add the two remaining protocol files** to the ARC in your preferred way
>    - Files `illumina_libraries.txt` and `RNA_extraction.txt`
>    - Both should go to `assays/rnaseq/protocols/` folder in your ARC
>
{: .hands_on}

Note that adding files via ARCitect import option will create a copy of the file, rather than moving it, so if you have large data files you might want to move the files via the explorer option.

## Add assay files

Next, let's add the RNA-seq data we have to the ARC structure.

> <hands-on-title>Add RNA-seq data</hands-on-title>
>
> 1. Look in the `rnaseq_data` folder of the demo data
>    - Files with the `.fastq.gz` suffix are raw sequencing files
>    - Raw data goes in the `assays/rnaseq/dataset/` folder
>    - `NGS_samplesheet.xlsx` can go into the assay's protocol folder
>
> 2. Move the files to the ARC in your preferred way
>
>    ![screenshot of arc explorer panel after adding files](images/arcitect-assay-files-rnaseq.png)
>
{: .hands_on}

Finally, let's do the same for our metabolomics data

> <hands-on-title>Add Metabolomics data</hands-on-title>
>
> 1. Look in the `metabolomics_data` folder of the demo data
>    - Folders with the `.D` suffix are raw data files
>    - The rest of the files can go to the protocols folder
>
> 2. Move all the files to the ARC metabolomics assay folder
>    - Note that if you move them via ARCitect, you should use the **Import Directories** option for the raw data folders
>
>    ![screenshot of arcitect files structure after adding metabolomics files](images/arcitect-assay-files-metabolomics.png)
>
{: .hands_on}



Congrats! you have added data to your ARC! As always, let's make sure to save & sync our work before we continue.

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
> 2. Sync your ARC to DataHUB
>    - Provide a good commit message (e.g. `added protocols and raw data files`)
{: .hands_on}


# Add your experimental metadata (SWATE)

Now that we have all our data in the ARC, we need to add metadata about our research in a structured way.
We already put a lot of information about our samples and experiments in the protocols folders,
but these are just free-text notes, like we would put in a methods section of a paper, or in our lab notebook.
The information is there but it is not **structured** yet.

In this section, we will show you how you can add structured metadata to your ARC. We do this by using a
standard format ([isa-tab](https://isa-specs.readthedocs.io/en/latest/isatab.html)), combined with
ontology-based annotations. We call these **annotation tables**, and these tables with metadata can be created
directly within the ARCitect tool.

> <comment-title>SWATE</comment-title>
>
> If you have previously already looked into ARCs, you may remember a tool called SWATE.
> SWATE helps you create these annotation tables. SWATE has now been integreted into ARCitect
>
{: .comment}

## Add sample information to the study metadata

First, we will add information about our samples (plants) to our study.

We took some notes about our samples in `plant_material.txt` file (now in our protocols folder)

```
Plant Material and Growth Conditions

Plants:
  - Species: Talinum fruticosum
  - Age: 28-d-old

Growth Chamber
  - 12 h light (30°C, 37% relative humidity)/12 h dark (22°C)
  - 425 µEinstein

Drought treatment:
  - no water for 12 days
  - re-water for 2 days

Sampling:
  - Leaves: topmost mature unshaded leaves, ca. 3–4.5 cm (see pictures)
  - shock-frozen in liquid N2

drought samples (CAM)
  - DB_097
  - DB_099
  - DB_103

re-watered samples (C3)
DB_161
DB_163
DB_165
```

> <tip-title> Viewing and editing text documents in ARCitect</tip-title>
>
> Text files that we added to our ARC, can also be viewed and edited from within ARCitect
>
> 1. Expand the talinum_drought study
> 2. Expand the protocols folder
> 3. Click on the `plant_material.txt` file
>
>    ![screenshot of markdown editor in ARCitect](images/arcitect-markdown-editor.png)
> 4. From this screen you can edit the text as needed.
>    - text files support [Markdown syntax](https://www.markdownguide.org/cheat-sheet/)
{: .tip}

### The ISA-Tab format

We will now take this free-text description of our samples, and turn them into a structured annotation table
using ARCitect.

At it's core, an annotation table often looks like:

Source Name | .. | .. | Sample Name
------------|----|----|-----------
plant 1     | .. | .. | sample1
plant 1     |    |    | sample2
plant 2     |    |    | sample3
...         |    |    | ...

The **Source Name** column and lists the sources (plants) you used in your experiments.
The final column, **Sample Name** describes the samples derived from the sources (plants).
Then there can be any number of columns in between describing characteristics of the plants (e.g. species, location, conditions, etc).
Which columns appear here depends on your experimental setup, and can be chosen by you.

These sample names you provide here can be referenced in other annotation tables in the study, or in assay-level annotation tables.
ARCitect uses these sample identifiers to put information from different annotation tables together, so make sure you use
consistent spelling and capitalization.

### Create an annotation table from scratch

There are 2 ways to add annotation tables, you can either:
1. build them up by hand, choosing the columns ("Building Blocks") yourself.
2. use a pre-defined template. These templates are either provided by the DataPLANT core team, or by other communities that use ARCs (or you!).

We will show both ways of adding annotation tables in this section of the tutorial.

First, let's add our sample information as an annotation table the manual way. Since this information pertains to the study as a whole,
not just a single assay, we will add our sample sheet information at the study level.

> <hands-on-title>Initialize smple information annotation table </hands-on-title>
>
> 1. Click on your study name in the ARC structure
>    - This should bring up the study metadata we filled in previously
> 2. At the bottom of the main panel, click on the **{% icon plus %} (plus) icon**
>    - the icon is located next to the current tab named *Metadata*
>
>    ![screenshot showing the button for adding an annotation table](images/arcitect-annotation-table-add.png)
>
>    - You should then see a new tab named **NewTable0** appear
>
>    ![screenshot after adding the new annotation table](images/arcitect-annotation-table-new.png)
>
> 3. **Rename** the table to **Sample Information**
>    - by right-clicking on the tab name **NewTable0**
>    ![screenshot of renaming option](images/arcitect-annotation-table-rename.png)
>
> 4. In the center panel, choose **Or start from Scratch!**
>
> 5. We fill first add our **Source Name** column
>    - In the dropdown, select **Input**
>    - Next select **Source Name**
>    - Click **Add Column**
> 6. Next, add an output column **Sample Name**
>    - Click on **Add building block** icon at the top of the table
>    - In the dropdown, select **Output**
>    - Next select **Sample Name**
>    - Click **Add Column**
>
>    ![screenshot of the two added columns](images/arcitect-annotation-table-start.png)
>
> 7. Let's add a column for the species as well
>    - Click on the **Add building block** button
>    - Change the dropdown to **Characteristic**
>    - In the search bar next to it, type **Species**
>    - Select the **NCIT** ontology term
>
> 8. The column probably appear on the right
>    - It is customary to keep the output column on the right
>    - Right-click on the species column
>    - Select **Move column**
>    ![screenshot of the move column option](images/arcitect-annotation-table-move-column.png)
>    - Change the column to `1`
>    - Click **Apply** to get a preview of the new order
>    - Click **Submit** to apply the change
>    ![screenshot of final move dialogue window](images/arcitect-annotation-table-move-column2.png)
>    - You should now have 3 columns, with species column in the middle
>
> 9. To add data, we have to **add rows**
>    - We had 6 samples, so we will add 6 rows
>    - At the bottom, change the 1 to a 6, and click on the {% icon plus %} (plus) button to add the rows
>    ![screenshot of the add rows button](images/arcitect-annotation-table-add-rows.png)
>    - You should now have 6 rows in your table
>    ![annotation table with rows added](images/arcitect-annotation-table-withrows.png)
>
{: .hands_on}

Now that we have our basic table initialized, let's start adding some data. For our example study, we grew 6 plants
(we will label them `DB001` to `DB006`). We grew 3 plants in drought conditions, and 3 controls. From our `sample_information.txt`
file we see that the samples we derived from these plants had IDs DB_097, DB_099, DB_103 (drought/CAM), and DB_161, DB_163, DB_165
(controls/re-watered/C3). All plants are species *Talinum fruticosum*.

Let's fill all this information in our annotation table.

> <hands-on-title>Fill the annotation table</hands-on-title>
>
> 1. Fill the first column (**Source Name**) with our 6 plant names
>    ```
>    DB001
>    DB002
>    DB003
>    DB004
>    DB005
>    DB006
>    ```
>    > <tip-title>Copying a whole column at once </tip-title>
>    > You can copy the values one by one, but you can also copy the whole column at once.
>    > - Copy the IDs above
>    > - In your ARC table, Select the first cell of the *Source Name* column
>    > - Hold down the shift key
>    > - Select the last (6th) cell of your column
>    > - Paste the IDs (`CTRL+V`)
>    {: .tip}
>
> 2. Fill the last column (**Sample Name**) with our 6 sample names
>    ```
>    DB_097
>    DB_099
>    DB_103
>    DB_161
>    DB_163
>    DB_165
>    ```
> 3. Now we will fill in the Species column
>    - Select the first cell in the Species column
>    - Start typing the species name `Talinum fruticosum`
>    - Since this is an ontology column, you will get suggestions as you type.
>    - In our case, we see NCBI ontology term for our species, please select this value.
>    ![screenshot of ncbi term in search](images/arcitect-species-search.png)
>
> 4. Copy the species value to the enitre column
>    - Right-click on the species term you just added in the firs row
>    - Select **Fill column** to apply this term to all 6 rows.
>    ![screenshot of our table after these steps](images/arcitect-annotation-sample.png)
{: .hands_on}

This is the basic process of building our annotation table. We can now add more building blocks (columns) as needed,
for all the information we have about our plants that we wish to describe. In the next section, we will add more columns.

But first, let's have a closer look at our annotation table.

> <comment-title> Why do we call them Building Blocks?</comment-title>
>
> 1. Click on the (expand) icon next to the species column name
>    - You will now see additional hidden columns
>    - These columns provide information about the ontology term you chose (TSR and TAN)
>    - There is also a hidden unit column. (For species this column is unused)
>
>
> ![screenshot of the hidden columns](images/arcitect-hidden-columns.png)
>
> This is why we use the term **Building Block** instead of **column**; behind the scenes there are actually more columns.
> We hide these columns in ARCitect, because most of the time these columns are not so interesting to humans, but they are
> what makes your ARC machine-readable.
>
> This machine readability is essential for example for ARC searching tools so that you can perform advanced
> searches such as *"Find all ARCs with data of Talinum Fruticosum that performed an RNA-seq assay"* and facilitate re-use
> of data. This will help people find your research, and will help you find data for your next study.
>
{: .comment}


Let's add some more building blocks for other information we have about our plants.
Based on our methods document, we know a few more characteristics of our plants:

- Light intensity of 425 µEinstein
- Plant age of 28 days
- Watering exposure of 12 days drought (for 3 of the samples) or 12 days drought followed by 2 days rewatering (for the other 3 samples)

Notice that the first two of these characteristics include a value (425, 28) and a unit (µEinstein, days).
In the next step we will show you how to deal with units in ARCitect

> <hands-on-title>Add building blocks with units</hands-on-title>
>
> 1. **Add a new building block** for plant age
>    - Select **Characteristic** from the dropdown
>    - Type **Plant Age**
>    - Select the DPBO term *Plant age*
>    - In the results, you can always click on the {% icon dropdown %} (dropdown) icon to get more information about the term
>    ![screenshot of ontology term information](images/arcitect-plant-age.png)
>
> 2. Before you add the column to the table, you can also **set the unit** for the column
>    - Click the button labelled **Unit**
>    - Search for **Day**
>    - Find an ontology term with a good description
>    ![screenshot of window adding building block with unit selected](images/arcitect-building-block-add-with-unit.png)
>    - Click **Add Column**
>    - Notice that the column is now already filled with the unit
>
> 3. The column will appear to the right of the column you were last on
>    - If you want your column in a different place, move it now in the same way we did before (right click the column)
>
> 4. Add the value **28** to the age column for all 6 samples
>    - You can see that your chosen unit is automatically added to the value you provide in the cell
>    - remember, you can fill in one cell, then use the "Fill Column" option from the right-click menu to apply the same value to all rows.
>
>    ![screenshot of our annotation table after adding plant age column](images/arcitect-annotation-table-age-added.png)
{: .hands_on}


> <comment-title>Characteristic vs Factor vs Parameter vs Component</comment-title>
>
> When adding building blocks, you may have noticed that a distinction is made between *characteristic*, *parameter*,
> *factor* and *component*. But when to use which?
>
> Have a look at this [Annotation Principles Guide](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/docs/guides/isa_AnnotationPrinciples.html)
> For a more precise description, but in short:
>
> - **Characteristics columns** describe inherent properties of the source material, e.g., a certain strain or ecotype,
>   but also the temperature an organism was exposed to.
> - **Factor columns** describe independent variables that determine the specific output of your experiment when process and analysis were identical.
>   Most of the time, Factors are the most important building blocks for downstream computational analysis.
> - **Parameter columns** describe steps in your experimental workflow, e.g., the temperature or extraction buffer used for your assay.
>   Multiple Parameter columns form a protocol.
> - **Component columns** are used to list anything physical of a protocol, e.g. instrument names, software names, or reagents names.
>
{: .comment}

Keeping this in mind, let's add more building blocks

> <hands-on-title>Add more buiding blocks</hands-on-title>
>
> 1. Add a building block for **light intensity exposure** (PECO:0007224)
>    - Add it as a **Parameter column**
>    - Set the unit to *microeinstein per square meter per second* (UO:0000160)
>    - Fill the column with a value of 425 for all samples
>
> 2. Add a building block for **Watering exposure** (PECO:0007383)
>    - Add it as **Factor column**
>    - set the value to "12 days drought" for the drought samples (`DB_097,DB_099,DB_103`)
>    - set the value to "12 days drought + 2 days rewatered"* for the other 3 samples
>
>    ![screenshot of the annotation table at this point](images/arcitect-annotation-table-complete.png)
>
{: .hands-on}

As a final step, we can associate our protocol document (`plant_material.txt`) with our annotation table.

> <hands-on-title>Add more buiding blocks</hands-on-title>
>
> 1. Add a new building block
>    - Add it as a **ProtocolREF** column (under *More*)
>    - Click **Add column**
> 2. For the value, set the file name `plant_material.txt` for all columns
>    - ARCitect will know to look in the protocols folder for this file
>    ![screenshot of the ProtocolREF column](images/arcitect-annotation-table-protocolref.png)
{: .hands_on}

Well done! You have added your first annotation table! You can add as many columns as you need in your annotation tables. The aim is always to capture as much information as you can in this way.

While we succeeded in creating our annotation table, it was quite a bit of work adding in our building blocks one by one. Luckily, there are a lot of templates available that can make your life easier! In the next section, we will show you how to do this.

## Add assay annotation

Building annotation tables from scratch can be a bit time consuming, especially if we have to repeat it for every ARC we create
Therefore, DataPLANT has also a way to share templates for annotation tables. For example, the metadata for RNA sequencing will be very similar for
everybody. So instead of creating this annoation table from scratch, we can load a pre-existing template. This is just a starting point, we can still
add and remove building blocks as needed, but it will save us a lot of time.

### Create RNA-Seq assay annotation tables from template

> <hands-on-title></hands-on-title>
>
> 1. Click on the RNA-seq assay in your ARC
> 2. Add a new annotation table
>    - {% icon plus %} (plus) icon at the bottom
> 3. **Rename** the table to a better name
>    - e.g. "RNA extraction"
> 4. Choose **Start from an existing template!** in the center of the screen
> 5. Search for **RNA**
>    - The results will show a few options from the official DataPLANT templates
>    - You can also search other community templates by changing the **Select community** dropdown to **All** (or a specific community)
>    ![template search results for term RNA](images/arcitect-template-rnaseq.png)
>
> 6. Choose the **RNASeq extraction** template
>    - Click **Select** to confirm
>    - This will give you a preview of the building blocks in the template
>    - Click **Add template**
> 7. Add 6 rows to the table
>    ![empty template file after importing](images/arcitect-template-rna-extraction.png)
> 8. Try to add RNA extraction information to this table
>    - Open the `RNA_extraction.txt` file in your protocols folder to view the information to add.
>
> > <question-title></question-title>
> >
> > 1. How does your final annotation table look?
> >
> > > <solution-title>Some Hints</solution-title>
> > > 1. The **biosource amount** column does not have a unit set yet. To do so now, please:
> > >    - Search for *milligram* in the column
> > >    - Right-click on the term, and select `as unit cell`.
> > >    - Choose "Fill column" from the right-click menu to apply this to all rows
> > > 2. Set the unit for **buffer volume** column to **microliter** in the same way
> > {: .solution}
> >
> > > <solution-title> Full solution </solution-title>
> > > Below are screenshots of a filled annotation table. Note that there are often multiple ways to fill an ARC.
> > >
> > > ![first half of the final annotation table](images/arcitect-annotation-table-rna-extraction-solution-part1.png)
> > > ![second half of the final annotation table](images/arcitect-annotation-table-rna-extraction-solution-part2.png)
> > >
> > {: .solution}
> {: .question}
{: .hands_on}


As always, let's save and sync our ARC

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
> 2. Sync your ARC to DataHUB
>    - Provide a good commit message
{: .hands_on}

# Adding computational workflows to your ARC

We have now added raw data, protocols, and annotation tables to our ARC. We can publish this *Data ARC* as-is,
but ARCs can also support addition of analysis workflows and results.

ARCs offer integrations for [CWL](https://www.commonwl.org/) (Common Workflow Language) workflows, but this is out of scope for this tutorial.
You can read more about adding CWL workflows to your ARC in the [DataPLANT knowledgebase](https://nfdi4plants.org/nfdi4plants.knowledgebase/docs/guides/ComputationalWorkflows/cwl_examples.html)

If you do not have CWL workflows, you can put any other scripts or code you used in the `workflows` folder, and
any analysis results in the `runs` folder. This will not be structured or statndardized, but better than nothing.

# Publishing your ARC

Once you have finished your ARC, you can choose to publish it. It will then become public for anybody to find and re-use.
Before you can do this, your ARC must pass some basic validation. This process checks if your ARC is in valid ARC format,
and if any files you reference are really included, etc.


## ARC validation

By default, the basic ARC validation is enabled in DataHUB

> <hands-on-title></hands-on-title>
>
> 1. Open your ARC repository on DataHUB. Tip: a quick way to get there is
>    - click on **DataHUB sync** in the ARCitect menu
>    - click on the **globe icon** to open the DataHUB repository in the browser
> 2. On the right of your DataHUB ARC repository, you should see a badge labelled **pipeline**
>    with either a green *passed* label or a red *failed* label
>
>    ![example of ARC pipeline badge](images/datahub-pipeline-passed.png)
>
> 3. You can click on this badge to get more information about the tests
>    - you can also click on the green checkmark or red cross next to your latest commit see validation pipeline details
>    - if your tests are failing, this can give you more information about what is wrong
>    - if it is green, you usually don't need to look at the test details.
>    - We will cover testing in more detail in a different tutorial, for now, just know that it exists.
{: .hands_on}

This test performs some basic checks, but in order to publish our ARC, we need additional validation. This is not
enabled by default.

## Enabling Invenio testing

Publishing of ARCs uses [Invenio](https://inveniosoftware.org/). Before you can publish your ARC,
we must enable the *Invenio validation package*.

> <hands-on-title> Invenio Validation </hands-on-title>
>
> 1. Click on **Validation** in the ARCitect menu panel
> 2. Under **Custom Validation Packages**, select **Invenio**
>    ![Screenshot of menu adding invenio](images/arcitect-validation-invenio-add.png)
> 3. Click **Save**
> 4. Sync the change to DataHUB
>    - Remember to **commit** first
> 5. Open your ARC repository on DataHUB again
>    - You should see additional badges
>    - Note that it may take a few minutes for these badges to show up
>    ![screenshot of invenio badges on DataHUB](images/datahub-validation-invenio-badges.png)
{: .hands_on}


## Publishing Process

Once all of the test pass, you can start the submission process for publishing your ARC.

> <hands-on-title></hands-on-title>
>
> 1. Open your ARC repository on DataHUB
> 2. Click on the invenio testing badge
>    - This can only be done if the badge is green
> 3. This will open a screen where you can preview the basic ARC information
>    ![Screenshot of invenio publishing page](images/datahub-invenio-publish.png)
> 4. We will not publish this training ARC, but for your real ARC you would now check the boxes to agree to the terms, and then hit the **Publish** button.
{: .hands_on}


## Where is your ARC published to?

After publishing, your ARC will be public, and automatically included in several locations.

### ARChive

The DataPLANT [ARChive](https://archive.nfdi4plants.org/) will list all published ARCs

For example [this ARC](https://archive.nfdi4plants.org/records/9e5jx-75d83)

![Example of a published ARC on ARChive](images/archive-example.png)

You see this looks a bit like a publication. Notice the DOI on the right. Every ARC will receive a DOI.
You can use this DOI in your publication to point readers to your ARC.

> <comment-title> Updating a published ARC </comment-title>
> Note that you can still update your ARC after publishing, and you can publish a newer version. You will
> receive a new DOI for the updated ARC, and the original version of your ARC will remain accessible
> with the original DOI.
{: .comment}

### ARC Registry

The [ARC registry](https://arcregistry.nfdi4plants.org/isasearch) is another place where your ARC will be listed.
The ARC registry can search across different institution's DataHUBs, and supports advanced searching of ARC metadata.

![screenshot of ARC registry homepage](images/arcregistry.png)

Many of these ARC searching and aggregation portals are still under active development, and may change/improve a lot in
the coming years.

# Next steps on your ARCification journey

You should now know the basics needed to create an ARC for your own data using the ARCitect tool.

For support during the process of creating an ARC for your own project(s), have a look at:
- The [DataPLANT knowledgebase](https://nfdi4plants.org/nfdi4plants.knowledgebase/).
- The [DataPLANT helpdesk](https://helpdesk.nfdi4plants.org/?topic=Documentation_KnowledgeBase)



# Conclusions

Congrats! You have now created your first ARC using ARCitect, you synced it to DataHUB, and saw the procedure for publishing
your ARC.


topic: fair
tutorial: fair-ena
title: Sequence data submission to ENA
url: https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-ena/tutorial.html
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: none", "type: none", "type: none"]
tags_json: ["FAIR", "ENA"]
content_hash: d5ba807f471553a0ee94eac5a0d47712
last_modified: 2026-05-15T22:28:13.145340
workflows_json: []
content:
DNA sequencing has become one of the key technologies in molecular biology, with applications in diagnostics, evolutionary biology, drug discovery, forensics and much more. Drop in sequencing costs and breakthroughs in sequencing technologies has seen increasing utilization of sequencing as a research tool, featuring in thousands of life-science publications every year. 

Prior to publication many journals and funders require authors to submit their raw sequence data to one of the three INSDC member databases – **ENA, NCBI or DDBJ** – between which data is synchronised on a daily basis. INSDC is the core infrastructure for sharing nucleotide sequence data and metadata in the public domain. Data in INSDC member databases is available permanently, for free and with unrestricted access. For each submitted sequence a unique accession number is issued which can be reported in the publication. 

The three databases have different methods for making submissions. If your database of choice is ENA and you 
need to submit data stored on a remote server, you are in the right place. This tutorial will cover how to find your way around the ENA Webin portal for uploading raw sequencing read data as well as accompanying metadata, and use [cURL](https://en.wikipedia.org/wiki/CURL) to copy read files over to ENA's FTP server. 

If you would like to use Galaxy tools for submission to ENA you may find [Submitting sequence data to ENA]({% link topics/galaxy-interface/tutorials/upload-data-to-ena/tutorial.md %}) tutorial helpful. 


> <tip-title>What is cURL?</tip-title>
> cURL is a command-line tool and library for transferring data over the internet. It allows you to send and receive data from various protocols like HTTP, FTP, and more. In simple terms, it's a tool that helps your computer talk to other computers on the internet and fetch or send information, like downloading files from a website or making API requests. 
{: .tip}

> <agenda-title></agenda-title>
>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}


## ENA Submission Routes

There are three routes via which one can submit data to ENA. 
1.	The Webin Interactive portal allows you to submit data by filling out forms on your browser. This is what we will cover in this tutorial,
2.	Webin-command-line interface (CLI), which allows submissions through a terminal, and
3.	Programmatic submission, which requires the submissions to be prepared as XML documents and then uploaded using cURL or Webin portal. 

**Table 1.** ENA submission routes
> |       | Interactive | Webin-CLI | Programmatic |
> | ----- |--------------------|
> | Study | Y    | N    | Y    |
> | Sample | Y    | N    | Y    |
> | Read data | Y    | Y    | Y    |
> | Genome Assembly | N    | Y    | N    |
> | Transcriptome Assembly | N    | Y    | N    |
> | Template Sequence | N    | Y    | N    |
> | Other Analyses | N    | N    | Y    |
{: .matrix}

As you can see from the Table 1, you can submit some data types only through a certain route. However, read data can be submitted via all three routes. 

To begin with the submission process you will need to have a Webin submission account. You can register an account free of cost through the Webin portal (https://www.ebi.ac.uk/ena/submit/webin/login).

For data submission, ENA provides a detailed guide (https://ena-docs.readthedocs.io/en/latest/submit/general-guide.html), which I recommend you consult for questions specific to your user case. Remember that ENA frequently changes its submission protocols, so you should check the ENA guide for updates.

## ENA Metadata Model

![ENA metadata model](./images/metadata_model.png "ENA metadata model")

The ENA metadata model (Figure 1) represents how metadata for a submission is structured and linked. Different information pertaining to a submission is organised into objects within the metadata model. Therefore, understanding the metadata model will enable you to determine which objects you need to submit. 

### 1. Study
All submissions require the study object. This object defines study accession, ownership, affiliation and release date. It also binds together all related objects into one cohesive project.

Study object submission generates a BioProject accession (PRJEB*) which can be used for citing data submitted to ENA, and a study accession (ERP*).

### 2. Sample
Sample object defines metadata about the sequenced biomaterial, e.g. bacteria, virus, etc. One sample should represent one sequencing library. 

To ensure that each sample is registered with sufficient metadata so as to provide enough context for the data to be easily interpreted, a mandatory, recommended and optional set of attributes should be provided for a sample.  Sample checklists have been developed in collaboration with different research communities to ensure that they are relevant and realistic for their context. They differ with the nature of the sample.

A registered sample will receive a BioSample accession (SAMEA*) and ENA sample accession (ERS*).

### 3. Raw reads
Metadata for raw reads is represented in 'experiment' and 'run' objects, where 'experiment' holds metadata on the sequencing method and 'run' holds metadata about the read files and their location on an FTP server. 

Each read file submission will receive a run accession (ERR*) and an Experiment accession (ERX*). 

# 02. Submitting data
## 1. Webin login
[Log in to ENA Webin portal](https://www.ebi.ac.uk/ena/submit/webin/). If you don't have a login you will have to register for an account. Go ahead and do that - it is free. Click on the 'Register' button and fill in the information requested.  

## 2. Study metadata
###  Metadata preparation
To register a study you will need the following metadata:
1. Study Name
2. Short descriptive study title
3. Detailed study abstract
4. Release date (if before two years from the date of submission)

1-3 will be displayed on the Project entry page. Make it as detailed as possible so as to make your study searchable. To see an example see the submission for [PRJEB55803](https://www.ebi.ac.uk/ena/browser/view/PRJEB55803).

> <hands-on-title>Register study</hands-on-title>
> Once you have the above information at hand register your study.
> 1. From the Webin dashboard click on 'Register Study' to bring up the study submission form. 
> 2. Complete the form to your best knowledge - ensure attributes with '*' are completed. 
> 3. Click 'Submit' to create a new study. 
> 4. If successful, you will receive a confirmational pop-up message with two accession numbers - PRJEB* (BioProject accession) and ERP*. Save them as you will need them for read submission step.
> 5. Check the processing status of your submissions via the Studies Report.
>
> > <tip-title>Filling in study metadata</tip-title>
> >You may want to set the submission date to the maximum allowed - 2 years from today. You can change it when you are ready to release the data. 
> >Study Fields are editable after submission should you need to update any information.
> {: .tip}
>
> ><question-title></question-title>
> >How can you confirm that your study has been successfully registered?
> > > <solution-title></solution-title>
> > > Click on the Dashboard icon, select 'Studies Report'. The accessions for your study should appear next to the Study title and other attributes. 
> > {: .solution}
> {: .question}
>
{: .hands_on}


## 3. Sample metadata
Next, you should register the samples from which your sequence data is derived.  
###  Metadata preparation
Minimum metadata requirement for different checklists varies. We list only what is required for the ENA default checklist (ERC000011).
1. Taxonomy ID of the organism sequenced
2. Scientific name of the organism sequenced
3. Unique name of the sample (auto-generated unless provided)
4. Sample title
5. Sample description
6. Collection date
7. Geographic location

Other checklists include:
+ Pathogen checklist
+ Environmental checklist
+ Marine checklist 

Find the complete list of [checklists](https://www.ebi.ac.uk/ena/browser/checklists).

> <hands-on-title>Download sample checklist</hands-on-title>
> 1. From the Dashboard, inside the 'Samples' box, click on 'Register Samples'.
> 2. Expand the 'Download spreadsheet to register samples' box and select 'Other Checklists'.
> 3. Click on 'ENA default sample checklist' (or the most appropriate checklist for your study). 
> 4. The mandatory attributes will be auto-selected. Expand the 'Optional Fields' box and select attributes you would like to include. 
> 
> Your checklist will contain columns shown in Table 2.
>    
> **Table 2.** Sample metadata checklist
>
> | Checklist	| ERC000011	| ENA default sample checklist | | | | |
> | tax_id	| scientific_name |	sample_alias	| sample_title	| sample_description	| collection date	| geographic location (country and/or sea) |
> | #units | | | | | | |
> 
> > <tip-title>Optional Fields</tip-title>
> > When registering a sample, it is recommended that you provide as much metadata as possible so as to make your study more searchable and useable.
> {: .tip}
> > <question-title></question-title>
> > What format should the collection date be in? Read descriptions for other attributes to get a clear understanding of the attributes and permitted values. 
> > > <solution-title></solution-title>
> > > Date/time in any of these formats is acceptable:
> > > 2008-01-23T19:23:10+00:00; 2008-01-23T19:23:10; 2008-01-23; 2008-01; 2008
> > {: .solution}
> {: .question}
> 
> {:start="5"}   
> 5. Click on 'Next' to get to the 'Download TSV Template' button.
{: .hands_on}

> <hands-on-title>Upload sample checklist</hands-on-title>
> 1. Open the Checklist_*.tsv file in a text editor of your choice e.g. MS Excel.  
> 2. Fill one sample per row, starting from row 4. Make sure you fill in all the columns. 
> 3. In the '#units' row specify the unit of measure, if required.
> 4. Make sure you save the checklist.tsv file in tab-delimited format with .tsv or .tab extension and not .xlsx or .csv.
> 5. In the Webin Portal, select 'Register Samples' and then 'Upload filled spreadsheet to register samples'. Upload your checklist.tsv file via 'Choose File' button. 
> 6. Click on 'Submit' to submit the checklist.
> 7. If the structure of your checklist is correct and the upload is successful a pop-up message will appear with Sample accession numbers -  SAMEA* (BioSample) and ERS*.
> 8. Check the processing status of your submissions via Samples Report. 
>
> > <tip-title>Sample Alias</tip-title>
> > Remember to enter sample aliases that correspond to what you use in related publications. This will enable readers to find sample-specific metadata and read files, even if you only state a Project accession number in your paper. Sample aliases can, furthermore, be optionally displayed as Unique name in the Sample report. 
> {: .tip}
{: .hands_on}

## 4. Read metadata
###  Metadata preparation
Having registered the sample, we can now add information about sequence reads generated from them. Reads can be submitted in several formats, such as fastq, BAM, CRAM, fast5, etc. we will look at submitting paired-end fastq files.
Metadata required for this are:
1. Sample alias or accession (from Sample submission step)
2. Study alias or accession (from Study submission step)
3. Sequencing instrument model (e.g. Illumina MiSeq, Illumina NextSeq 500, MinION, etc.)
4. Library name (if any)
5. Type of source material sequenced (E.g. genomic, transcriptomic, etc.)
6. Library selection method used (e.g. PCR, cDNA, etc.)
7. Library sequence strategy (e.g. RNA-Seq, WGS, etc.)
8. Library layout i.e. single or paired-end
9. Forward fastq file name (must be compressed, and name to include subdirectory)
10. MD5 checksum of forward fastq file (if not using Webin file uploader)
11. Reverse fastq file name (must be compressed, and name to include subdirectory)
12. MD5 checksum of reverse fastq file (if not using Webin file uploader)

### Fastq file preparation
First, we need to confirm that your read files are in the correct format. Refer to [ENAs file format guidelines](https://ena-docs.readthedocs.io/en/latest/submit/fileprep/reads.html) if you are not sure.

> <hands-on-title>Linux or OSX</hands-on-title>
> #### On a Linux-based operating system
> 1. Compress the fastq files for the upload using gzip. 
>
>    Open the terminal on your machine then type the commands below. First move to the directory where fastq files are located, then compress the fastq files using gzip command.
>
>    ```
>    # In the command below replace '/path/to/fastq/directory' with the correct path
>    cd /path/to/fastq/directory
>
>    gzip *.fastq
>    ```
>
> 2. To enable verification of the integrity of the uploaded fastq file, ENA requires md5 checksum for each file.
>
>    Type the command below to calculate and print md5 sums to tab-separated file (for easy cut-and-paste later).
>    ```
>    for f in *.gz; do md5 $f | awk '{ gsub(/\(|\)/,""); print $2"\t" $4 }'; done > md5sums.tsv
>    ```
>    md5sums.tsv will contain a tab-separated table of fastq.gz filenames and their md5sum. 
>
{: .hands_on}


> <hands-on-title>Read checklist submission</hands-on-title>
> 1. From the 'Dashboard' select 'Submit Reads' from Raw Reads box. 
> 2. Similar to how you downloaded the sample checklist, download the Read submission template for 'paired reads using two fastq files'.
> 3. Select the relevant optional fields to be included in the spreadsheet.
> 
> > <tip-title></tip-title>
> > + Tick the box next to 'Show Descriptions' to reveal text describing the attributes
> > + To find the sample alias and accession, look in 'Sample Reports'
> > + To find the study accession, look in 'Studies Reports'  
> {: .tip}
> 
> {:start="4"}
> 4. Click 'Next' and download the spreadsheet .tsv file.
> 5. Fill in the spreadsheet, paying particular attention to the following:
> sample: same as you used in sample submission - alias or accession
> study: use study accession
> forward/reverse_file_name: ensure this is the same as what will be uploaded. It should be the compressed .gz file you created.
> forward/reverse_file_md5: copy this from md5sums.tsv you generated earlier.
> 6. Uploaded the completed template spreadsheet and submitted it.
> 7. If metadata validation is successful a pop-up message bearing a run accession (ERR*) and experiment accession (ERX*) will appear for each read submission. 
> 8. Confirm the processing status of your submission via Runs Report.
> 9. Any errors generated in the process will be communicated to you on the registered email.
>
> <tip-title></tip-title>
>  The run submission holds information about the raw read files generated in a run of sequencing as well as their location on an FTP server. The experiment submission holds metadata that describes the methods used to sequence the sample.
> {: .tip}
>
{: .hands_on}

## 5. Read data upload
ENA offers [several methods](https://ena-docs.readthedocs.io/en/latest/submit/fileprep/upload.html) to achieve this. Please take a look at all the options should any suit your needs better than the one provided below.

If you are using a Linux-based system you can use the below commands without needing to install any software.

```
cd /path/to/fastq.gz
curl -T <<your_file>>.gz -u Webin-XXX:password ftp://webin2.ebi.ac.uk/
```

Check the processing status of your upload via the Run Processing Report.  

> <warning-title></warning-title>
> The Webin upload area is a temporary transit location which is not backed up. Always ensure you retain a local copy of the data till the files have been successfully submitted and archived. 
{: .warning}

## 6. Post-submission editing
Metadata submitted for each object can be edited via the Reports (Studies, Samples, Runs) by clicking on the box-arrow icon in the Action column next to the table presenting the reports. The underlying xml of the submitted item can be inspected and edited. 

However, for complex edits you might have to contact [ENA helpdesk](https://www.ebi.ac.uk/ena/browser/support).


topic: fair
tutorial: ro-crate-in-galaxy
title: Exporting Workflow Run RO-Crates from Galaxy
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ro-crate", "workflows"]
content_hash: 1d0ee0e3249aeeda7fd3f3ea27683ba6
last_modified: 2026-05-15T22:28:13.173136
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-in-galaxy/tutorial.html

content:
Workflows are a powerful Galaxy feature that allows you to scale up your analysis by performing an end-to-end analysis with a single click of a button. In order to keep provenance of the workflow invocation (an invocation of a workflow means one run or execution of the workflow) it can be exported from Galaxy in the form of a [Workflow Run Crate](https://w3id.org/ro/wfrun/workflow) RO-Crate profile.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}


Additionally, the exported Workflow Run Crate allows for sharing workflow run provenance with those unfamiliar with Galaxy and its standard export format.

This tutorial will guide you through the steps of defining such a report for your workflow, .

This tutorial will show you how to generate Galaxy-based [Workflow Run RO-Crate](https://w3id.org/ro/crate/) after running the workflow.

{% include _includes/cyoa-choices.html option1="Locally" option2="Not_locally" default="Not_locally"
       text="Are you running Galaxy locally ?" %}
       
<div class="Locally" markdown="1">

## Enable RO-Crate on your local instance
> <hands-on-title>Update your galaxy configuration</hands-on-title>
> - Go to where your Galaxy folder is in your computer
> - In your root Galaxy folder navigate to the `config` folder where a the `galaxy.yml` should be located. Please open it.
> - (In case you only find a `galaxy.yml.sample` file, copy this one and name it `galaxy.yml`) 
> - make sure the option `enable_celery_tasks` is set to `true`:
> ```
> galaxy:
>       enable_celery_tasks: true
> ```
> That's it ! Now you can launch your local instance as usual.
{: .hands_on}
</div>

<div class="Not_locally" markdown="1">
</div>

## Import an example workflow

For this tutorial, we will use the workflow from the [Galaxy 101 for everyone tutorial]({% link topics/introduction/tutorials/galaxy-intro-101-everyone/tutorial.md %}). If you have not done this tutorial yet, the only thing you need to know is that this is a workflow that takes as input a table of data about different species of iris plants, this table is subsequently sorted and filtered, and some plots are made. The specifics of the workflow are not important for this tutorial, only that it outputs a number of different kinds of outputs (images, tables, etc).

We will start by importing this workflow into your Galaxy account:

> <hands-on-title>Import the workflow</hands-on-title>
>
> 1. **Import the workflow** into Galaxy
>
>    {% snippet faqs/galaxy/workflows_run_trs.md path="topics/galaxy-interface/tutorials/workflow-reports/workflows/galaxy-101-everyone.ga" title="Galaxy 101 for Everyone" %}
>
{: .hands_on}


## Run the workflow

Galaxy will produce several export options for any workflow. The default export gives us a serialization of the invocation data model while the RO-Crate export gives an Workflow Run Crate which includes the default export as well.

Let’s run the workflow and export the RO-Crate.

> <hands-on-title>Run the workflow</hands-on-title>
>
> 1. Import the file `iris.csv` via link
>
>    ```
>    https://zenodo.org/record/1319069/files/iris.csv
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
> 2. Run **GTN Training: Galaxy 101 For Everyone** workflow using the following parameters:
>    - *"Send results to a new history"*: `No`
>    - *"1: Iris Dataset""*: the `iris.csv` file we just uploaded
>
>    {% snippet faqs/galaxy/workflows_run.md  %}
>
> 3. **View the workflow outputs** once the workflow has completed
>    - The workflow produces several text and tabular outputs, and two plot (image) outputs
>
{: .hands_on}

## Export the Workflow Run Crate

After the workflow has completed, we can export the RO-Crate. The crate does not appear in your history, but can be accessed from the {% icon galaxy-history-options %} **-> Show Invocations** menu on the top right of your history OR on the left pannel from the {% icon galaxy-panelview %} Workflow Invocations .

> <hands-on-title>Export the Workflow Run Crate</hands-on-title>
>
> 1. In the top right of your history, go to {% icon galaxy-history-options %} **-> Show Invocations**
>
>    ![screenshot of the history options to get the Show invocations button](./images/show_invocation.png)
>
> 2. Our latest workflow run should be listed at the top.
>    - Click on it to expand it:
>
>    ![screenshot of the workflow invocations menu, with our latest invocation at the top](./images/workflow-invocation-summary.png)
>
> 3. Click on the **Export** tab in the expanded view of the workflow invocation.
>
> 4. Click on the Export tab in the expanded view of the workflow invocation.
>        You should see a page that contains three download options:
> 	     - Research Object Crate (RO-Crate) 
> 	     - BioCompute Object
> 	     - File
> 5. Click on the **Generate** {% icon galaxy-download %} option of the RO-Crate box (1st box)
>
>    ![screenshot of the beginning of the workflow run export options](./images/workflow-invocation-export.png)
>
{: .hands_on}

Great work! You have created a Workflow Run Crate. This makes it easy to track the provenance of the executed workflow.


topic: fair
tutorial: fair-intro
title: FAIR in a nutshell
url: https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-intro/tutorial.html
difficulty: intermediate
time_estimation: 10M
tools_json: []
requirements_json: []
tags_json: ["fair", "open", "data stewardship"]
content_hash: 9bab8e613897f8f0ea70875eb8d11698
last_modified: 2026-05-15T22:28:13.151340
workflows_json: []
content:
The FAIR (Findable, Accessible, Interoperable, Reusable) principles emphasize machine-actionability. The main objective of FAIR is to increase data reuse by researchers. The core concepts of the FAIR principles are based on good scientific practice and intuitively grounded. 

This tutorial is a short introduction to the FAIR principles and their origin. You can find out more at the **FAIR Pointers module** at the [FAIR Data Management]({% link learning-pathways/fair-training.md %}) learning path.


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# FAIR and its origins

The FAIR Guiding Principles aid in designing data publishing platforms for easier manual and automated deposition, exploration, sharing, and reuse {% cite wilkinson %}. FAIR stands for specific improvements in data management and archival practices as it outlines clear, high-level, domain-independent principles that can be used to create a variety of scholarly outputs:


| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a globally unique and persistent identifier<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. (meta)data are released with a clear and accessible data usage license<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        |

Table 1: The FAIR guiding principles as described in Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship {% cite wilkinson %}.

A report from the European Commission Expert Group on [FAIR data](https://zenodo.org/record/1285272#.ZGc58exByha) describes the origins of FAIR and its development in 2014-2015 by a [FORCE11](https://force11.org/groups/) Working Group. The following exercise dips into this report and asks you to investigate some of FAIR’s history and foundation.

## Open data and FAIR
The level of accessibility and usability criteria distinguish open from FAIR data. Open data is accessible without limitations, whereas FAIR data specifies certain requirements for access and use.

![text reading fair does not equal open](../../images/fair_open.png)

Open data can be modified, and distributed for any reason. Although extensively used and accessible, FAIR data additionally includes the following usability standards that go beyond permission alone: 

- In order to be found and cited, FAIR data must be identified and deposited into online public records. 

- It is necessary to make FAIR data available so that it may be accessed, read, and processed. 

- FAIR data needs to be captured and presented in a form that can be used.

The published FAIR Guiding Principles: Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship {% cite wilkinson %}.

## FAIRification and FAIRness of data
Making your data FAIR compatible involves adopting the 15 guiding principles listed in Table 1, which is the process known as "FAIRification." The degree to which you adhere to these criteria determines how FAIR your data are. In other words, FAIRness refers to how FAIR your data is to a certain level and indicates a tacit method of evaluating its compliance.

Documentation and frameworks for data FAIRification. Each of the 15 FAIR principles is put into context with real data examples: [GO FAIR](https://www.go-fair.org/fair-principles/), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html).

# Conclusion
The FAIR Principles place a strong focus on encouraging individual data reuse while also supporting the capacity of machines to automatically discover and utilise the data. This short introduction aims to develop and disseminate guidance and processes needed to make and keep data FAIR.


topic: fair
tutorial: bioimage-REMBI
title: REMBI - Recommended Metadata for Biological Images – metadata guidelines for bioimaging data
difficulty: introductory
time_estimation: 15m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["Data management", "Metadata", "Bioimaging"]
content_hash: 8dd32d4ee0716b58520d8a12f87a8773
last_modified: 2026-05-27T09:01:48.578137
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/bioimage-REMBI/tutorial.html

content:
# Metadata guidelines for bioimaging data

REMBI (Recommended Metadata for Biological Images) was proposed as a draft metadata guidelines to begin addressing the needs of diverse communities within light and electron microscopy. Currently, these guidelines are in draft form to encourage discussion within the community, but they provide a useful guide as to what metadata should be gathered to make your image data FAIR. They divide the metadata requirements into eight modules which further split into attributes - that seems to be a daunting task, doesn't it? But at the same time it's exciting news for the community! To find out more, have a look at the [REMBI article](https://www.nature.com/articles/s41592-021-01166-8).


> <question-title></question-title>
>
> In the [REMBI paper](https://www.nature.com/articles/s41592-021-01166-8), the authors consider three potential user groups who require different metadata. Find out what are these three groups and their metadata requirements.
>
> > <solution-title></solution-title>
> > The identified three user groups are: Biologists, Imaging scientists, Computer-vision researchers.
> > - A research biologist may be interested in the biological sample that has been imaged to compare it to similar samples that they are working with.
> > - An imaging scientist may be interested in how the image was acquired so they can improve upon current image acquisition techniques.
> > - A computer vision researcher may be interested in annotated ground-truth segmentations, that can be obtained from the image, so they can develop faster and more accurate  algorithms.
> {: .solution}
>
{: .question}

> <tip-title>Instructor Note</tip-title>
>
> If you're an instructor leading this training, you might ask people to work in small groups for this exercise and encourage the discussion. Ask group members to share which of the user groups they identify as and what metadata they would want.
>
{: .tip}

# Categories of metadata
REMBI covers different categories of metadata, such as:
- study
- study component
- biosample
- specimen
- image acquisition
- image data
- image correlation
- analyzed data

Within each module, there are attributes that should be included to make the published data FAIR. We will explore all the modules and attributes suggested by REMBI and we'll show some examples as well.

## Study
The first module of REMBI metadata describes the Study and should include:
- Study type
- Study description
- General dataset information

### Study type

Ideally, the study type will be part of an ontology. You can look up the main subject of your study using a tool like [OLS](https://www.ebi.ac.uk/ols/index) to find a suitable ontology. This will help others to see where your study sits within the wider research area.

> <comment-title>Example</comment-title>
>
> <table>
>   <tr>
>     <th>Study type</th>
>     <td>Regulation of mitotic cell division</td>
>   </tr>
> </table>
>
{: .comment}


### Study description

A brief description of the project. The Study Description should include the title of the study, a brief description and any related publication details such as authors, title and  DOI. If you are gathering metadata prepublication, you can fill in the publication details later or enter a draft title or the journal name you plan to submit to. It’s still a good idea to include the category, so you don’t forget.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th colspan="2">Study description</th>
>   </tr>
>   <tr>
>     <th>Title</th>
>     <td>Imaging mitotic cells</td>
>   </tr>
>   <tr>
>     <th>Description</th>
>     <td>Visualising HeLa cells using confocal microscopy</td>
>   </tr>
>   <tr>
>     <th>Publication details</th>
>     <td>TBC</td>
>   </tr>
> </table>
{: .comment}


### General dataset information

This should include all the information that relates to all the data in the project. This can include the names of contributors and the repository where the data is or will be stored. State the licence under which you intend to make the data available, the repository you intend to submit to and if you are using a schema for structuring your metadata. This helps to keep all collaborators on the same page. Any other general information with respect to the study can be included here, but try to keep this broad as more detailed information should be included in other sections of the metadata.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th colspan="2">General Dataset Information</th>
>   </tr>
>   <tr>
>     <th>Contributors</th>
>     <td>Alica and Bob</td>
>   </tr>
>   <tr>
>     <th>Repository</th>
>     <td>Bioimage Archive</td>
>   </tr>
>   <tr>
>     <th>Licenses</th>
>     <td>CC-BY</td>
>   </tr>
>   <tr>
>     <th>Schemas</th>
>     <td>Datacite Metadata</td>
>   </tr>
> </table>
{: .comment}

## Study component

A study component can be thought of as an experiment, both the physical experiment and subsequent data analysis, or a series of experiments that have been conducted with the same aim in mind.

The associated metadata should describe the imaging method used and include a description of the image dataset. The REMBI guidelines store high-level metadata in the study component and then divide the more detailed metadata into other modules.

Within the Study component we include the Imaging Method which should describe the techniques used to acquire the raw data. This could be one or multiple methods, which should be part of a relevant ontology. For Confocal Microscopy data, we can use the Biological Imaging Methods Ontology, although it is also present in a number of other ontologies.

The description of the study component should include an overview of what was imaged as well as any processed data that is created during analysis.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Imaging Method</th>
>     <td>Confocal Microscopy</td>
>   </tr>
>   <tr>
>     <th>Study Component Description</th>
>     <td>Images of cells and segmented binary masks</td>
>   </tr>
> </table>
{: .comment}


> <tip-title>Storing metadata</tip-title>
>
> You could either choose to store the metadata in the same file as your study data or have a new file for each study component. This could be stored in the same place as your study metadata, or you could create a subdirectory structure.
{: .tip}

## Biosample

The first thing you need for the biosample metadata is an Identity. This is a code that you assign to each sample you are describing, which will link this metadata to the physical sample. Then, state what the biological entity is, which should come from a relevant ontology. Use a taxonomy to name the organism. Next, describe the variables in your experiment. The REMBI guidelines split the variables into three types:
- intrinsic - describe an innate trait of the biosample, such as a genetic alteration
- extrinsic - describe something you added to the sample, for example, a reagent
- experimental - things that you intentionally vary, like time

You can leave out some of the variables if they are not part of your experiment.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Identity</th>
>     <td>CM001</td>
>   </tr>
>   <tr>
>     <th>Biological entity</th>
>     <td>JURKAT E-6.1 cell</td>
>   </tr>
>   <tr>
>     <th>Organism</th>
>     <td>Homo sapiens</td>
>   </tr>
>   <tr>
>     <th>Intrinsic variable</th>
>     <td>Jurkat E6.1 transfected with emerald-VAMP7</td>
>   </tr>
>   <tr>
>     <th>Extrinsic variable</th>
>     <td>Aspirin</td>
>   </tr>
>   <tr>
>     <th>Experimental variables</th>
>     <td>Dose response of aspirin</td>
>   </tr>
> </table>
{: .comment}

## Specimen

 The specimen metadata should include:
- the experimental status (control or test)
- the location within the biosample, such as a coordinate or a particular well in a plate
- how the sample was prepared
- how the signal is being generated
- the content and biological entities of different channels.

 Include enough information so that someone with experience in the field could reproduce a sample by following the information you provided. Assume they would know typical techniques and name them using terms from an ontology if possible. Only include lots of detail if you are describing a novel technique.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Experimental status</th>
>     <td>Control</td>
>   </tr>
>   <tr>
>     <th>Location within biosample</th>
>     <td>Plasma membrane within 100 nm of coverslip (TIRF)</td>
>   </tr>
>   <tr>
>     <th>Preparation method</th>
>     <td>Cos-7 cells cultured in DMEM medium, and then plated on #1 coverslips and imaged live in L-15 medium</td>
>   </tr>
>   <tr>
>     <th>Signal/contrast mechanism</th>
>     <td>fluorescent proteins</td>
>   </tr>
>   <tr>
>     <th>Channel – content</th>
>     <td>Green: eGFP, Red: mCherry</td>
>   </tr>
>   <tr>
>     <th>Channel – biological entity</th>
>     <td>Green: EGFR, Red: Src</td>
>   </tr>
> </table>
{: .comment}

## Image acquisition
Here you should include all the information about the instrument you used and how it was set up. Like with the specimen metadata, describe this information as though you are speaking to someone who already knows how to use a similar instrument. What would they need to know to produce the same image data?

Check with your facility manager if they have any guidelines for what details need to be recorded for your particular instrument. Make sure that the parameters you record can actually be used by someone else if they don’t have exactly the same instrument or setup. For example, don’t say that you used a certain percentage of laser power, as this doesn’t tell you how much power was used unless you also provide the total power of the laser. If the instrument software has automatically generated a metadata file, remember to save this. Depending on its content, this may be sufficient.

Start with the details of the equipment for the Instrument Attributes. If this is commercial equipment, include the make and model, a short description of what type of instrument it is and details about its configuration. If the instrument is bespoke, you will need to include more details. Next, you should include image acquisition parameters. These relate to how the instrument was set up for the particular experiment. Some of these may be captured automatically by the instrument’s software, so make things easy for yourself and check if a file is generated and what’s in it. If a file is generated, then you only need to manually record anything that is missing from the file.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Instrument attributes</th>
>     <td>Olympus FV3000, laser point scanning confocal, 500-550 nm filter, 37-degree chamber.</td>
>   </tr>
>   <tr>
>     <th>Image acquisition parameters</th>
>     <td> </td>
>   </tr>
>   <tr>
>     <th>Objective</th>
>     <td>20x</td>
>   </tr>
>   <tr>
>     <th>Excitation Wavelength</th>
>     <td>488 nm</td>
>   </tr>
>   <tr>
>     <th>PMT gain</th>
>     <td>500 V</td>
>   </tr>
>   <tr>
>     <th>Pixel dwell time</th>
>     <td>2 𝜇s</td>
>   </tr>
>   <tr>
>     <th>Confocal aperture</th>
>     <td>200 𝜇m</td>
>   </tr>
> </table>
{: .comment}

> <tip-title>Helpful resources</tip-title>
>
> To help you collect the information for your own data, you might have a look at the local resources from your institution or universities. For example, at Warwick University, there are [webpages](https://warwick.ac.uk/fac/sci/med/research/biomedical/facilities/camdu/methodsreporting/) describing the metadata that needs to be collected for some of the microscopes.
>
{: .tip}

## Image data

In this section, you record all the information related to all the images you have. Not only the primary or raw images, but also any processed images,  perhaps such as binary files showing the resulting segmentation.

You need to say what format the images are in and if they have undergone any compression, the dimensions of the images, and what the physical size of the pixel or voxel is, including the units. Most of this information you should be able to get from the metadata or header of the image files.

Next, you need to state the physical size of the image or magnification, calculated from the pixel or voxel size and the dimension extents. Give any information related to how the channels are represented. For processed images, you need to provide the methods used for processing.

Finally, say you have used contrast inversion, do the bright features in the image correspond to areas of high signal, or is it the other way around?

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Type</th>
>     <td>Primary Image, Segmentation</td>
>   </tr>
>   <tr>
>     <th>Format and compression</th>
>     <td>Primary: .oir (Olympus), Segmentation: .tiff</td>
>   </tr>
>   <tr>
>     <th>Dimension extents</th>
>     <td>x: 512, y: 512, z: 25</td>
>   </tr>
>   <tr>
>     <th>Size description</th>
>     <td>153.6 x 153.6 x 25 𝜇m</td>
>   </tr>
>   <tr>
>     <th>Pixel/Voxel size description</th>
>     <td>0.3 x 0.3 x 0.1 𝜇m</td>
>   </tr>
>   <tr>
>     <th>Image processing method</th>
>     <td>Fiji: Median filter (3 pixel kernel), Otsu threshold</td>
>   </tr>
>   <tr>
>     <th>Contrast inversion</th>
>     <td>No</td>
>   </tr>
> </table>
{: .comment}


## Image correlation

If you have used different imaging modalities with the same sample, this part of the metadata should describe how the images relate to one another. You could use this section to describe generally the relationship between images. In the example below, images from different modalities have been aligned.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Spatial and temporal alignment</th>
>     <td>Manual</td>
>   </tr>
>   <tr>
>     <th>Fiducials used</th>
>     <td>Soil grains</td>
>   </tr>
>   <tr>
>     <th>Transformation matrix</th>
>     <td>See file: Transforms.csv</td>
>   </tr>
>   <tr>
>     <th>Size description</th>
>     <td>153.6 x 153.6 x 25 𝜇m</td>
>   </tr>
>   <tr>
>     <th>Related images and relationship</th>
>     <td>Primary XCT: Data/XCT
>       Primary XRF: Data/XRF
>       Processed XRF: Data/Transformed_XRF</td>
>   </tr>
> </table>
{: .comment}

## Analysed data

This section should not include metadata for any image data, including processed images, as that should have been covered in the Image Data section. Instead, it should describe the analysis results you have, such as measurements. Have you done some numerical analysis or some phenotyping or something else? There is no need to describe the methods in great detail if they are already described in the relevant publication.

> <comment-title>Example</comment-title>
> <table>
>   <tr>
>     <th>Analysis results type</th>
>     <td>Speed of cell division</td>
>   </tr>
>   <tr>
>     <th>Data used for analysis</th>
>     <td>Preprocessed images, Cell tracks</td>
>   </tr>
>   <tr>
>     <th>Analysis method</th>
>     <td>Track cell lineage: BayesianTracker (btrack) with configuration track_config.json
>       Measure speed: Numerical analysis in Python</td>
>   </tr>
> </table>
{: .comment}

# Final notes

For more examples, check out REMBI Supplementary Information - either in [pdf](https://static-content.springer.com/esm/art%3A10.1038%2Fs41592-021-01166-8/MediaObjects/41592_2021_1166_MOESM1_ESM.pdf) or [spreadsheet](https://docs.google.com/spreadsheets/d/1Ck1NeLp-ZN4eMGdNYo2nV6KLEdSfN6oQBKnnWU6Npeo/edit#gid=1023506919).


At first glance, it might seem to be quite a stretch to collect all that metadata! But don’t get discouraged - following those guidelines will ensure better communication between the scientists and will make your research FAIR: Findable, Accessible, Interoperable, Reusable. During big data era when we are surrounded by so much resources, it’s crucial to get good data management habits, share them with others and hence contribute to the development of Science toghether.


topic: evolution
tutorial: mtb_phylogeny
title: Tree thinking for tuberculosis evolution and epidemiology
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["prokaryote", "one-health", "phylogenetics", "microgalaxy"]
content_hash: 1cce7ca67097f771a2254b0aa327ff6a
last_modified: 2026-05-27T09:01:48.562125
zenodo_link: https://zenodo.org/record/6010176
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/evolution/tutorials/mtb_phylogeny/tutorial.html

content:
> "Nothing in biology makes sense except in the light of evolution."
{: .quote author="Theodosius Dobzhansky" year="1973"}

Phylogenetic trees are a tool for organizing biological diversity. Just as maps provide a spatial framework to the geographer, phylogenies provide an evolutionary context to the biologist: they capture the relationship among "things" (species, individuals, genes), represented as tips in the tree, based on common ancestry.

In evolutionary and epidemiological studies of *Mycobacterium tuberculosis*, it is now common to encounter large phylogenetic trees. Being able to read and critically examine them is extremely useful. Phylogenies can be used to understand the origin of a disease or the onset of an epidemic, to distinguish ongoing transmission from imported cases, to investigate how specific traits like antibiotic resistance evolve *et cetera et cetera*. Many research ideas originate from looking at and discussing patterns present in phylogenies.

This tutorial provides an introduction to phylogenetic trees in the context of whole genome sequencing of *Mycobacterium tuberculosis* strains. Phylogenetics is a vast topic, and we can only scratch its surface here. For those motivated to delve deeper into the topic, the [Resources section](#resources) contains links and reading suggestions.


## Basic concepts: How to read a phylogeny

Below are two phylogenies of the *Mycobacterium tuberculosis* complex (MTBC) to illustrate some basic vocabulary and concepts.

> <details-title>Need a more thorough introduction to the topic?</details-title>
>
> If the following discussion sounds a bit too esoteric, you might be interested in a more thorough introduction to the topic of phylogeny like provided, for example, by the [introduction to phylogenetics](https://www.ebi.ac.uk/training/online/courses/introduction-to-phylogenetics/) from the [EMBL-EBI collection of online tutorials](https://www.ebi.ac.uk/training/on-demand?facets=type:Online%20tutorial).
>
{: .details}

### Rooted trees and tree topology

![Phylogeny](./images/mtbc_1strainPerLineage.nwk.COMB.svg "Rooted [A] and unrooted [B] phylogeny of the MTBC. Some basic tree vocabulary is shown in red.")

The trees in figure 1 look rather different at a first glance, but they are identical except for one key aspect: tree A is **rooted** while tree B is not.

What is the difference between the two? First, let's state what is the same in the two trees: the tree **topology**, that is, the relative branching order. The same groupings are present in the two trees: they both contain the same information about the **relatedness** of strains. An example: TB isolated from Peruvian mummies is most similar to *M. pinnipedii* known from marine mammals; they share a most recent common ancestor. This can be seen in the rooted as well as in the unrooted tree.

The key difference between the rooted and the unrooted tree is that only the rooted tree shows the **direction** and sequence of branching events. The unrooted tree does not tell us, for example, whether *M. bovis* diverged early or late in the history of the MTBC. It is thus compatible with the old hypothesis that human TB evolved from animal TB. The rooted tree shows that this hypothesis is most likely wrong: animal-associated strains are not ancestral to human-associated strains.

The best way to root a tree is by including an **outgroup**: a species or lineage which we know *a priori* to lay outside the phylogeny we're interested in. *M. canettii* usually serves this purpose for studying the MTBC, but you can also root, for example, a phylogeny of lineage 2 by including a lineage 4 strain.

### Branch lengths

Besides relatedness and direction, a third important piece of information contained in a phylogeny is the branch length. When a phylogeny has been estimated from DNA or protein sequences, branch lengths usually reflect the evolutionary distance between nodes in the tree. This information can be used to translate distance in terms of expected nucleotide changes into years, and thus to connect evolutionary change to historical events.

![Phylogeny](./images/eldholm2016_tree.png "Phylogeny of the central Asian clade, including the Oslo outbreak. Modified Eldholm et al. 2016."){:width="100"}

As branch lengths reflect evolutionary distances, they can also be used to identify transmission clusters and outbreaks. Figure 2 shows a (rooted) tree of the Central Asian Clade (CAC), which is part of lineage 2 ({% cite Eldholm2016 %}). The orange color highlights the Afghan strain family within the CAC. At the bottom of the tree, note the clade with short branch lengths. This is how one would expect an outbreak to look in a phylogenetic tree: a set of strains clustering together and separated by extremely short branches, reflecting their almost identical genomes.

> <comment-title>Phylogenetics with <i>Mycobacterium tuberculosis</i></comment-title>
>
> Phylogenetics with MTB has some particularities rarely encountered with other organisms.
>
> 1. There seems to be **no horizontal gene transfer** (HGT) in the MTBC. HGT complicates phylogenetic inference in many bacteria because a piece of DNA introduced by HGT has a different history and thus phylogeny than genes not affected by HGT.
>
> 2. There is **little genetic diversity** in the MTBC: any two strains differ by only around 2,500 SNPs over the whole genome. To achieve a good resolution of recent evolution (e.g. during an outbreak), whole genome sequences are thus extremely useful.
>
> 3. A large proportion of DNA polymorphisms in the MTBC are **singletons**, that is, variants present only in a single strain. This adds to the problem of low diversity, since singletons are not informative about tree topology.
>
> 4. In this workshop, and indeed in many studies of the MTBC, SNPs are called not against the reference strain H37Rv, but against a reconstructed ancestral genome. This means that the number of SNPs identified does not reflect the evolutionary distance from some random strain like H37Rv, but from the most recent common ancestor of the MTBC. Take a look at Figure 3 in {% cite Goig2018 %} to see how this affects the number of SNPs identified in a genome. Below we will see that this has implications for the interpretation of a tree.
>
{: .comment}


## Infering phylogeny from SNP alignments

Aligned DNA or protein sequences are the starting material for phylogenetic inference with molecular data.

In this tutorial we will explore the phylogenetic relationship between 19 MTBC strains and 1 strain of *M. canettii*, where the latter is included as outgroup, allowing us to root the phylogeny. Our starting point will be an alignment of all SNPs identified from whole-genome sequencing data of these 20 strains. This *SNP alignment* was generated by

1. mapping short sequenced reads from each sample to the infered sequence of the most recent common ancestor of the MTBC as the reference
2. calling variants against that reference from the mapped reads of each sample
3. incorporating the SNP variants of each sample into the reference to obtain a consensus genome of that sample
4. combining the consensus genomes into a single multi-sequence fasta dataset
5. discarding bases found to be identical in all samples from all sequences

> <comment-title>Recommended tutorials</comment-title>
>
> Steps 1 and 2 are explained in detail in the tutorial [MTB variant analysis]({% link topics/variant-analysis/tutorials/tb-variant-analysis/tutorial.md %}).
> Steps 3 - 5 are conducted for the exact same 20 samples used here in the [tutorial on transmission clusters]({% link topics/evolution/tutorials/mtb_transmission/tutorial.md %}).
>
> If you want to gain a complete understanding of the whole process of sequencing-based analysis of MTBC strains, we recommend you, if you have not done so yet, to work through these two tutorials in the order above before continuing with this one here.
>
{: .comment}

Figure 3 shows a snapshot of an illustrative SNP alignment.
Each row in the alignment represents a different strain, each column a position in the reference genome.

![Alignment](./images/MEGA_alignment.png "Illustrative example of a SNP alignment. Note the absence of conserved positions.")

Because this alignment is based on SNPs, it contains *only variable positions*. This is important to keep in mind! Genetic distances between strains will be hugely overestimated if we leave out all the positions which show no variation!

Near the end of this tutorial, we will show and discuss ways to correct for this.

> <comment-title>The alternative approach: genome assembly</comment-title>
>
> A frequently used alternative approach to obtain a phylogeny from short read data is to a) assemble the genomes (see the numerous [Galaxy tutorials]({% link topics/assembly/index.md %}) on this topic), b) annotate genes, c) extract genes present in all strains (the "core" genes), d) align the core genes. This approach underlies core genome multilocus sequence typing (cgMLST), which is often used to genotype bacterial pathogens (e.g. {% cite Zhou2021 %}).
>
{: .comment}

With this bit of introduction you should, hopefully, be prepared to follow the flow and purpose of the analysis presented here.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Phylogenetic analysis of MTBC strains with Galaxy and Rstudio

## Get the data

If you have worked at least through the first part of the [tutorial on transmission clusters]({% link topics/evolution/tutorials/mtb_transmission/tutorial.md %}) before coming here, you are all set.

You can continue working in the same history you created for the other tutorial, or copy the *SNP alignment* (the output of **Finds SNP sites** {% icon tool %}) over to a new one, and can safely skip the following first hands-on part.

If you are not interested in that other tutorial, here are the instructions for setting up a history for the analysis here, and for obtaining a copy of the SNP alignment from [Zenodo](https://zenodo.org/record/6010176).

> <hands-on-title>Create a new history and prepare the input data</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Give the history a suitable name
>
>    {% snippet faqs/galaxy/histories_rename.md name="MTB tree thinking tutorial" %}
>
> 3. {% tool [Upload](upload1) %} a version of the SNP alignment via this URL
>
>    ```
>    https://zenodo.org/record/6010176/files/SNP_alignment.fasta
>    ```
>
>    and make sure the dataset format is set to `fasta`.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>
> 4. {% tool [Replace Text in entire line](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) in entire line %} to clean up the aligned sequence names
>
>    - {% icon param-file %} *"File to process"*: the uploaded SNP alignment
>    - In {% icon param-repeat %} *"1. Replacement"*:
>      - *"Find pattern"*: `^(>.+)\.fastq.*`
>      - *"Replace with"*: `\1`
>
>    From just a brief inspection of the downloaded SNP alignment, you should see that the sequence names in that file carry a `.fastq.vcf` or `.fastq.gz.vcf` suffix on the actual sample names.
>    The above replacement operation, drops these unnecessary endings, which we would otherwise carry over to all outputs generated in the analysis.
>
> 5. When the Replace Text tool run is finished, **rename** the output dataset to, for example, **SNP alignment**
>
>    {% snippet faqs/galaxy/datasets_rename.md name="SNP alignment" %}
>
> 6. Take a look at the alignment
>
>    Click on the dataset that you just renamed to expand it in the history, then click on {% icon galaxy-barchart %} **Visualize** and try both, the "Editor" and the "Multiple Sequence Alignment".
>
>    Can you see one of the MTBC particularities mentioned above, the predominance of singletons?
>
>    How many sites are there in the alignment?
>
>    Also take a look at the M. canettii sequence: being the outgroup, it has a large number of SNPs.
>
{: .hands_on}

## Set up the coding environment

We will use R to estimate a tree from a nucleotide alignment and to plot and manipulate the tree. We will provide you with the exact code to produce the figures we need, and, conveniently, you can run this code from an RStudio session that you can start as an interactive tool inside Galaxy.

> <details-title>Can I use my own coding environment?</details-title>
>
> If you have worked with R or RStudio previously, feel free to use any kind of coding environment for exploring and manipulating the tree - just download it from your history and import it into any R session you have access to.
>
{: .details}

> <hands-on-title>Starting an RStudio session and setting things up</hands-on-title>
>
> 1. Note the dataset number (the number in front of and separated with a **:** from the dataset name) of the  file called *"SNP_alignment.fasta"* 
>
> 2. Run {% tool [RStudio](interactive_tool_rstudio) %}
>
>    {% snippet faqs/galaxy/interactive_tools_rstudio_launch.md %}
>
> 3. Install and use the [ape R package](https://cran.r-project.org/web/packages/ape/index.html)
>    1. Click on the **Terminal** tab (at the top of the Rstudio window)
>    2. Execute the command: `conda install r-ape`
>
>       This will install the ape package into the environment of the running session.
>    3. When the previous step has finished, switch back to the **Console** tab and run the command
>
>       ```r
>       library("ape")
>       ```
>
>       You might get a warning about R versions, which you can ignore.
> 4. Import the SNP alignment into the Rstudio session
>
>    ```r
>    alignment_file <- gx_get(###)
>    ```
>
>    where you have to replace `###` with the dataset number of the alignment file in your history.
>
{: .hands_on}


## Estimate the tree 

We have now set up our R session and are ready to start. As a first step we would like to load the alignment into R and infer a tree from it. 

In this tutorial, we'll use the [neighbor-joining algorithm](https://academic.oup.com/mbe/article/4/4/406/1029664) to estimate a phylogenetic tree for the 20 strains. The details of how these methods construct trees from an alignment are beyond the scope of this introductory course. To be able to read trees, it is not necessary to know the statistical and computational details of how the trees are estimated. The books listed in the [Resources](#resources) section provide in-depth introductions into the different principles of phylogenetic inference, in particular Baum & Smith 2013 and Yang 2014.

> <comment-title></comment-title>
>
> If you are experienced with R, you are encouraged also to start playing with all the code from here on, to modify it, and to explore the numerous phylogenetics packages and functions available in R.
>
{: .comment}


> <hands-on-title>Estimate a phylogeny and plot it</hands-on-title>
>
> ```r
> # Load the alignment
> aln <- read.dna(alignment_file, format="fasta") 
>
> # Remove the ugly file extensions from the sample names
> rownames(aln) <- gsub('.fastq.vcf', '', rownames(aln))
> rownames(aln) <- gsub('.fastq.gz.vcf', '', rownames(aln))
>
> # Estimate a distance matrix, showing how many nucleotide differences there are between any two samples
> dmat <- dist.dna(aln, model='N')
>
> # Infer the tree from the matrix using neighbor-joining
> tree <- nj(dmat)
>
> # Plot the tree
> plot(tree)
> ```
>
> > <question-title></question-title>
> >
> > 1. Take a look at the tree. Is it rooted or unrooted? What is the strain far apart from all other strains?
> >
> > > <solution-title>1</solution-title>
> > >
> > > ![Unrooted tree](./images/tree_unrooted.svg)
> > >
> > > 1. The tree is unrooted, and the outlier strain is *M. canettii*, our outgroup. The much longer branch leading to this strain shows that many SNPs separate *M. canettii* from the common ancestor of the MTBC.
> > >
> >{: .solution}
>{: .question}
{: .hands-on}

## Root the tree

Phylogenetic trees are great tools because they are at the same time quantitative (we can do calculations on branch lengths, estimate uncertainty of a tree topology etc.) and visually appealing, allowing to actually "see" biologically interesting patterns. Often this requires some tweaking of the tree, for example by coloring parts of the tree according to some background information we have about the samples.

To make the phylogeny more interpretable, we will now root the tree using the *M. canettii* strain as the outgroup, then exclude that strain, such that patterns *within* the MTBC become clearer.

> <hands-on-title>Root the tree and drop its outgroup</hands-on-title>
>
> ```r
> # Root the tree
> tree_rooted <- root(tree, "ERR313115")
>
> # Remove the outgroup to make distances within MTB clearer
> tree_rooted <- drop.tip(tree_rooted, "ERR313115")
> tree_rooted$root.edge <- 0.005
> plot(tree_rooted, root.edge = T, cex=0.6)
> ```
{: .hands_on}

![Rooted tree](./images/tree_rooted.svg){:width="600"}

This already looks better, the tree topology stands out more clearly now, and we can identify groups of closely related strains.

## Create tree with lineage as tip label instead of strain name

A first piece of information we now want to add to the phylogeny is to which lineage the strains belong. This will allow us to assess whether our tree is consistent with the known phylogeny of the MTBC, shown in Figure 1A, and to visualize which lineages are present in our sample. The information to which lineage a strain belongs can be found in the output of TB-profiler, as explained in the [tutorial on TB variant analysis]({% link topics/variant-analysis/tutorials/tb-variant-analysis/tutorial.md %}).

> <hands-on-title>Generate a tree with lineage labels</hands-on-title>
>
> ```r
> # Assign lineages to samples, as identified by TB-profiler
>
> mtbc_lineages <- c(
>   "ERR181435" = "L7",
>   "ERR313115" = "canettii",
>   "ERR551620" = "L5",
>   "ERR1203059" = "L5",
>   "ERR2659153" = "orygis",
>   "ERR2704678" = "L3",
>   "ERR2704679" = "L1",
>   "ERR2704687" = "L6",
>   "ERR5987300" = "L2",
>   "ERR5987352" = "L4",
>   "ERR6362078" = "L2",
>   "ERR6362138" = "L2",
>   "ERR6362139" = "L4",
>   "ERR6362156" = "L2",
>   "ERR6362253" = "L2",
>   "ERR6362333" = "L2",
>   "ERR6362484" = "L4",
>   "ERR6362653" = "L2",
>   "SRR998584" = "L5",
>   "SRR13046689" = "bovis"
> )
>
> # Replace the tree labels
> tree_lineages <- tree_rooted
> tree_lineages$tip.label <- as.character(mtbc_lineages[tree_rooted$tip.label])
>
> # Define some colors for the lineages
>
> color_code_lineages = c(
>   L1 = "#ff00ff",
>   L2 = "#0000ff",
>   L3 = "#a000cc",
>   L4 = "#ff0000",
>   L5 = "#663200",
>   L6 = "#00cc33",
>   L7 = "#ede72e",
>   bovis="black",
>   orygis="black")
>
> pal_lineages <- as.character(color_code_lineages[tree_lineages$tip.label])
>
> # Plot the old and new tree version next to each other
> par(mfrow = c(1, 2))
> plot(tree_rooted,cex = 0.7, root.edge = TRUE)
> plot(tree_lineages,cex = 0.8, tip.color = pal_lineages, root.edge = TRUE)
> ```
>
> > <question-title></question-title>
> >
> > Looking at the different lineages present in the tree, does our phylogeny make sense? Or asked differently: does our phylogeny show the same branching patterns between lineages as the established phylogeny in Fig. 1A?
> >
> > > <solution-title>2</solution-title>
> > >
> > > ![Colored lineages](./images/tree_rooted_lineages.svg){:width="600"}
> > >
> > > There is indeed a problem with our phylogeny: one L2 and one L5 strain do not cluster with the other strains of these lineages. Instead, they appear near the root of the tree, with very short (ERR5987300) to non-existent (ERR1203059) branches. Other parts of the tree are consistent with Fig. 1A, suggesting that we can focus our first round of troubleshooting on these two strains.
> > >
> >{: .solution}
>{: .question}
{: .hands-on}

## First round of exercises

> <question-title>Exercise 1</question-title>
>
> An important part of bioinformatics consists in trying to find out whether a surprising observation has biological significance --- or reflects a mistake somewhere in the numerous steps leading to the result. Have we just discovered two new lineages of MTB, or did we commit a stupid mistake? To find out, take a look a the TB-profiler and the VCF files for the two strange strains, which should be present in your Galaxy history if you've followed the previous tutorial. Compare them with "normal" strains. Do you notice something?
>
> > <solution-title></solution-title>
> >
> > The VCF files hold an important hint to explain our puzzling observation: ERR1203059.vcf contains not a single SNP, ERR5987300.vcf only 81 SNPs. By contrast, the other strains have between 750 and 1250 SNPs. What happened here? To find out, we would have to take a closer look at the steps leading from BAM to VCF files. One possibility is that the sequencing depth for these samples was so low that most SNPs were filtered out because they did not pass the quality filtering.
> >
> {: .solution}
>
{: .question}

> <question-title>Exercise 2</question-title>
>
> In the [tutorial on transmission clusters]({% link topics/evolution/tutorials/mtb_transmission/tutorial.md %}) two clusters of samples got identified and are reproduced below. How do these clusters show up in the phylogenetic tree? What additional information does the tree contain?
>
> | Sample       | Cluster_id | DR profile | Clustering  |
> |--------------|------------|------------|-------------|
> | ERR5987352   | 10         | Pre-MDR    | Clustered   |
> | ERR6362484   | 10         | Pre-MDR    | Clustered   |
> | ERR6362138   | 12         | MDR        | Clustered   |
> | ERR6362156   | 12         | Pre-XDR    | Clustered   |
> | ERR6362253   | 12         | MDR        | Clustered   |
>
> > <solution-title></solution-title>
> >
> > Clusters 10 and 12 appear as clades of closely related strains in the phylogeny: cluster 12 being part of lineage 2, cluster 10 of lineage 4. The phylogeny additionally reveals that cluster 12 is part of a larger clade of rather closely related lineage 2 strains. While clustering with a fixed SNP threshold produces a binary outcome (clustered/unclustered), the phylogeny reveals the gradual nature of relatedness. With a more permissive SNP threshold for clustering, or a different pipeline to call SNPs, we might well identify a larger cluster of L2 strains.
> >
> {: .solution}
>
{: .question}

## Map a trait onto the tree

Phylogenies are particularly useful when combined with additional information. For our 19 MTB strains, for example, we might know such things as the country of origin, the sampling date, or various phenotypes determined in the lab, for example the virulence of the strains in an animal model. By mapping this additional information onto the phylogeny, we can gain insights into how, where and when these traits evolved.

For our 20 samples, a trait you previously identified (if you have been doing the [tutorial on transmission clusters]({% link topics/evolution/tutorials/mtb_transmission/tutorial.md %}), is the DR profile. Let us map this trait onto the tree and see if we can learn something from the observed patterns.

> <hands-on-title>Generate a tree with DR profiles as labels</hands-on-title>
>
> ```r
> # Same as before, but with DR profiles instead of lineages
>
> mtbc_dr <- c(
>   "ERR181435" = "Sensitive",
>   "ERR313115" = "Sensitive",
>   "ERR551620" = "MDR",
>   "ERR1203059" = "Sensitive",
>   "ERR2659153" = "Sensitive",
>   "ERR2704678" = "Sensitive",
>   "ERR2704679" = "Sensitive",
>   "ERR2704687" = "Sensitive",
>   "ERR5987300" = "PreXDR",
>   "ERR5987352" = "PreMDR",
>   "ERR6362078" = "MDR",
>   "ERR6362138" = "MDR",
>   "ERR6362139" = "PreMDR",
>   "ERR6362156" = "PreXDR",
>   "ERR6362253" = "MDR",
>   "ERR6362333" = "PreXDR",
>   "ERR6362484" = "PreMDR",
>   "ERR6362653" = "MDR",
>   "SRR998584" = "Sensitive",
>   "SRR13046689" = "Other"
> )
>
> tree_dr <- tree_rooted
> tree_dr$tip.label <- as.character(mtbc_dr[tree_rooted$tip.label])
>
> color_code_dr = c(
>   Sensitive = "#ff00ff",
>   PreXDR = "#0000ff",
>   PreMDR = "#a000cc",
>   MDR = "#ff0000",
>   Other = "#663200"
> )
>
> pal_dr <- as.character(color_code_dr[tree_dr$tip.label])
>
> par(mfrow = c(1, 2))
> plot(tree_rooted,cex = 0.7, root.edge = TRUE)
> plot(tree_dr,cex = 0.8, tip.color = pal_dr, root.edge = TRUE)
> ```
>
{: .hands_on}

## Exercises continued

> <question-title>Exercise 3</question-title>
>
> In the previous tutorial on clustering, you have come across the hypothesis that unclustered cases of DR represent *de novo* evolution of DR, while clustered cases of DR represent instances of DR transmission. Looking at lineage 2 in the phylogeny above, does this hypothesis hold? How many times would MDR have evolved independently in lineage 2? Is there an alternative explanation for the prevalence of MDR in lineage 2?
>
> > <solution-title></solution-title>
> >
> > ![Colored DR profiles](./images/tree_rooted_dr.svg){:width="600"}
> >
> > MDR would have evolved three times according to the clustering perspective mentioned above: once in cluster 12, once in ERR6362078, and once in ERR6362653. The phylogeny suggest a simpler alternative: MDR could have been already present in the common ancestor of the six L2 strains in our sample; it could have evolved only once, along the long branch leading from the split from lineage 3 to the most recent common ancestor of the six samples. This picture, however, might change with a more extensive sampling of lineage 2. Six samples are hardly sufficient to make claims about the prevalence and evolution of MDR in lineage 2. As for the interpretation of clustering, sampling design is crucial for the interpretation of phylogenies and should always be kept in mind in order to avoid overinterpretation.
> >
> {: .solution}
>
{: .question}

## Bootstrapping 

Before drawing big conclusions from our tree, let us get an idea of how robust it actually is. 

Bootstrapping is a resampling approach that allows us to do that. The basic idea is that we draw sites randomly from our original alignment, and from this "artifical alignment" we estimate a tree. We repeat this procedure 100 times and then ask how many times the same nodes occur in the bootstrap trees. All these steps are conveniently implemented in ape's boot.phylo() function. 

> <hands-on-title>Get bootstrap support</hands-on-title>
>
> ```r
> # Reset graphical device
> dev.off()
>
> # Define nr of bootstrap replicates
> n_replicates = 100
>
> # Remove the ougroup from the alignment 
> root = 'ERR313115'
> aln_rooted <- aln[-which(rownames(aln) == root),]
>
> # Pack the tree estimation into a single function  
> f <- function(x) nj(dist.dna(x, model='N'))
>
> # Do the bootstrapping and add the values to the tree
> tree_rooted$bootstrap  <- boot.phylo(tree_rooted, aln_rooted, FUN = f, B = n_replicates, jumble=TRUE)
>
> # Plot tree with bootstrap values
> plot(tree_rooted, root.edge = T, cex=2)
> add.scale.bar(x=0.5, y=0.5, lwd=2, cex=1.5)
> nodelabels(tree_rooted$bootstrap, cex=2)
> ```
>
{: .hands_on}

## Final exercise

> <question-title>Exercise 4</question-title>
>
> 1. Is our tree robust?  
>
> 2. In what context would you expect phylogenetic trees with poor bootstrap support?
>
> > <solution-title></solution-title>
> >
> > ![Bootstrap support](./images/tree_bootstrap.svg){:width="600"}
> >
> > 1. Yes, bootstrap values are generally high: a value of 99, for example, means that in 99 out of 100 bootstrap replicates the same split was observed. Only among the highly similar L2 strains there is some uncertainty. 
> >
> > 2. When analyzing a set of highly similar strains, for example in the context of a TB outbreak, bootstrap values can be low and patterns should be interpreted with care. Imagine, for example, that our dataset consisted of 20 L2 strains that differ from each other only by few or single SNPs. 
> >
> {: .solution}
>
{: .question}

# Conclusion

Any phylogenetic analysis requires careful exploration of results and an understanding of its limitations to avoid overinterpretaion. The combination of Galaxy and R accessed through an interactive tool is very powerful because the Galaxy platform provides the compute resources and tools necessary to calculate the phylogenetic tree, and offers straightforward access to RStudio. The flexibility of R and its packages then lets you take a deep dive into the results.

# Resources
To develop a deeper understanding of phylogenetic trees, there is no better way than estimating phylogenies yourself --- and work through a book on the topic in your own mind's pace.

## Books
- *Phylogenetics in the genomics era*, 2020. An [open access book](https://hal.inria.fr/PGE) covering a variety of contemporary topics.
- *Tree Thinking*, 2013, by David A. Baum & Stacey D. Smith
- *Molecular Evolution*, 2014, by Ziheng Yang

## Useful links
- [MEGA Software](https://megasoftware.net/)
- [Tutorial on how to read a tree, with a virus example](https://artic.network/how-to-read-a-tree.html)
- [Tree Of Life web project](http://tolweb.org)
- [Phylogenetic Inference in the Stanford Encyclopedia](https://plato.stanford.edu/entries/phylogenetic-inference/)

tools_json:
["Upload", "upload1", "Replace Text in entire line", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "RStudio", "interactive_tool_rstudio"]


topic: evolution
tutorial: bacterial-comparative-genomics
title: Phylogenetic analysis for bacterial comparative genomics
difficulty: introductory
time_estimation: 1H
requirements_json: []
tags_json: ["microgalaxy"]
content_hash: c2c4c4a1e3adfe80aa8bf4e2090a06d1
last_modified: 2026-05-27T09:01:48.560048
zenodo_link: https://zenodo.org/records/1
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/evolution/tutorials/bacterial-comparative-genomics/tutorial.html

content:
General introduction about the topic and then an introduction of the
tutorial (the questions and the objectives). It is nice also to have a
scheme to sum up the pipeline used during the tutorial. The idea is to
give to trainees insight into the content of the tutorial and the (theoretical
and technical) key concepts they will learn.

You may want to cite some publications; this can be done by adding citations to the
bibliography file (`tutorial.bib` file next to your `tutorial.md` file). These citations
must be in bibtex format. If you have the DOI for the paper you wish to cite, you can
get the corresponding bibtex entry using [doi2bib.org](https://doi2bib.org).

With the example you will find in the `tutorial.bib` file, you can add a citation to
this article here in your tutorial like this:
{% raw %} `{% cite Batut2018 %}`{% endraw %}.
This will be rendered like this: {% cite Batut2018 %}, and links to a
bibliography section which will automatically be created at the end of the
tutorial.


**Please follow our
[tutorial to learn how to fill the Markdown]({{ site.baseurl }}/topics/contributing/tutorials/create-new-tutorial-content/tutorial.html)**

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Galaxy and data preparation

> <hands-on-title> Data Upload </hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Rename the history
>
>    {% snippet faqs/galaxy/histories_rename.md %}
>
> 3. {% tool [Import](upload1) %} the contig file from [Zenodo]({{ page.zenodo_link }}) or from Galaxy shared data libraries:
>
>    ```
>    {{ page.zenodo_link }}/files/DRR187559_contigs.fasta
>    ```
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md %}
>
>    {% snippet faqs/galaxy/datasets_import_from_data_library.md %}
>
>
{: .hands_on}

# Tree building

> <hands-on-title> Task description </hands-on-title>
>
> 1. {% tool [FASTTREE](toolshed.g2.bx.psu.edu/repos/iuc/fasttree/fasttree/2.1.10+galaxy1) %} with the following parameters:
>    - *"Aligned sequences file (FASTA or Phylip format)"*: `fasta`
>        - {% icon param-file %} *"FASTA file"*: `output` (Input dataset)
>        - *"Set starting tree(s)"*: `No starting trees`
>    - *"Show advanced options"*: `No`
>
>    ***TODO***: *Check parameter descriptions*
>
>    ***TODO***: *Consider adding a comment or tip box*
>
>    > <comment-title> short description </comment-title>
>    >
>    > A comment about the tool or something else. This box can also be in the main text
>    {: .comment}
>
{: .hands_on}

***TODO***: *Consider adding a question to test the learners understanding of the previous exercise*

> <question-title></question-title>
>
> 1. Question1?
> 2. Question2?
>
> > <solution-title></solution-title>
> >
> > 1. Answer for question1
> > 2. Answer for question2
> >
> {: .solution}
>
{: .question}


# Conclusion

Sum up the tutorial and the key takeaways here. We encourage adding an overview image of the
pipeline used.

tools_json:
["Import", "upload1", "FASTTREE", "toolshed.g2.bx.psu.edu/repos/iuc/fasttree/fasttree/2.1.10+galaxy1"]


topic: evolution
tutorial: mtb_transmission
title: Identifying tuberculosis transmission links: from SNPs to transmission clusters
difficulty: intermediate
time_estimation: 2H
requirements_json: []
tags_json: ["prokaryote", "one-health", "microgalaxy"]
content_hash: 7ca2fd02cef3645703565eb18d347501
last_modified: 2026-05-27T09:01:48.562678
zenodo_link: https://zenodo.org/record/6010176
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/evolution/tutorials/mtb_transmission/tutorial.html

content:
In a disease outbreak situation, to understand the dynamics and the size of the outbreak, it is essential to detect transmission clusters to distinguish likely outbreak cases from unrelated background cases. Such detection is nowadays often based on actual sequencing data that enables quantitative conclusions about differences between pathogen isolates.

This tutorial guides you through transmission cluster identification from preprocessed whole-genome sequencing data of MTB strains. It consists of two parts:

1. Part one starts from per-sample lists of mutations/variants (in *variant call format*, VCF), as derived from sequencing data through mapping of sequenced reads to a reference genome followed by variant calling. From the lists of variants, more specifically from the single nucleotide variants found for every sample, you will construct a sample distance matrix and identify likely transmission clusters based on overall sample similarity.

2. Part two starts with drug-resistance profiles of the same samples and combines these profiles with the cluster information obtained in part one to enable reasoning about the validity of identified clusters and about evolution of drug resistance in the samples.

Both the lists of variants and the drug-resistance reports for all samples have been pre-generated for you, and can simply be imported into Galaxy from public sources.

> <comment-title>Origin of the input data</comment-title>
>
> This tutorial does *not* discuss the analysis steps that lead from sequencing data to lists of variants, nor the generation of drug-resistance profiles with the tool TB-Profiler. These topics are covered in the separate tutorial [M. tuberculosis Variant Analysis]({% link topics/variant-analysis/tutorials/tb-variant-analysis/tutorial.md %}) instead.
>
> If, after working through this and the other tutorial, you would like to combine the two to perform the complete analysis from sequenced reads to transmission clusters for the 20 samples used here, you can find their raw sequencing data in this [Zenodo record](https://zenodo.org/record/5911437), and two Galaxy workflows to process the data into VCFs and to generate drug-resistance profiles as [supplementary material](./workflows) to this tutorial.
>
{: .comment}

> <comment-title>Recommended background information</comment-title>
>
> A series of webinars has been produced alongside this tutorial to provide some theoretical background for the topics touched here. Specifically, these are:
>
> - [{% icon video %} Drug resistance prediction](https://youtu.be/Ddwt-_gQR2M)
> - [{% icon video %} Phylogenetic mutations](https://youtu.be/1ps_o5rpnmw)
> - [{% icon video %} The concept of clustering](https://youtu.be/l4cPUECJ7VU)
> - [{% icon video %} Genetic distance thresholds](https://youtu.be/kKNgmpy1N94)
>
> Watching at least some of them before or after doing the tutorial is highly recommended.
>
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Analysis Part 1: Identification of transmission clusters from per-sample variants

## Get the data

Any analysis should get its own Galaxy history. So let's start by creating a new one:

> <hands-on-title>Prepare the Galaxy history</hands-on-title>
>
> 1. Create a new history for this analysis
>
>    {% snippet faqs/galaxy/histories_create_new.md %}
>
> 2. Give the history a suitable name
>
>    {% snippet faqs/galaxy/histories_rename.md name="MTB transmission clusters tutorial" %}
>
{: .hands_on}

As mentioned in the introduction, mapping and variant calling have already be performed for the 20 samples
that we need to analyze. The result of this preprocessing are 20 VCF files that describe the mutations found
for each of the samples, and which we now need to import into Galaxy.

The fastest way to do so, which lets us build a collection from the 20 datasets with correctly named elements (corresponding to the sample identifiers) directly during data upload is through the so-called *Rule-based uploader*. The Galaxy training material has dedicated tutorials, [Rule Based Uploader]({% link topics/galaxy-interface/tutorials/upload-rules/tutorial.md %}) and [Rule Based Uploader Advanced]({% link topics/galaxy-interface/tutorials/upload-rules-advanced/tutorial.md %}), that introduce this powerful feature in detail.

For the purpose of this tutorial, it is enough if you simply follow the step-by-step instructions provided here exactly.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the following files from [Zenodo]({{ page.zenodo_link }})
>
>    ```
>    https://zenodo.org/record/6010176/files/ERR1203059.vcf
>    https://zenodo.org/record/6010176/files/ERR181435.vcf
>    https://zenodo.org/record/6010176/files/ERR2659153.vcf
>    https://zenodo.org/record/6010176/files/ERR2704678.vcf
>    https://zenodo.org/record/6010176/files/ERR2704679.vcf
>    https://zenodo.org/record/6010176/files/ERR2704687.vcf
>    https://zenodo.org/record/6010176/files/ERR313115.vcf
>    https://zenodo.org/record/6010176/files/ERR551620.vcf
>    https://zenodo.org/record/6010176/files/ERR5987300.vcf
>    https://zenodo.org/record/6010176/files/ERR5987352.vcf
>    https://zenodo.org/record/6010176/files/ERR6362078.vcf
>    https://zenodo.org/record/6010176/files/ERR6362138.vcf
>    https://zenodo.org/record/6010176/files/ERR6362139.vcf
>    https://zenodo.org/record/6010176/files/ERR6362156.vcf
>    https://zenodo.org/record/6010176/files/ERR6362253.vcf
>    https://zenodo.org/record/6010176/files/ERR6362333.vcf
>    https://zenodo.org/record/6010176/files/ERR6362484.vcf
>    https://zenodo.org/record/6010176/files/ERR6362653.vcf
>    https://zenodo.org/record/6010176/files/SRR13046689.vcf
>    https://zenodo.org/record/6010176/files/SRR998584.vcf
>    ```
>
>    1. Copy the links above
>    2. Open the {% tool [Upload](upload1) %} Manager
>    3. In the top row of tabs select **Rule-based**
>    4. Set **Upload data as** to `Collection(s)`
>    5. Paste the copied links into the text field on the right
>    6. Click **Build** to bring up the *Rule Builder* dialog
>    7. Click on the wrench icon {% icon tool %} next to **Rules** in the left panel of the window
>    8. In the text field, erase the prefilled content and copy/paste the following rule definition into it instead:
>
>       ```
>       {
>         "rules": [
>           {
>             "type": "add_column_basename",
>             "target_column": 0
>           },
>           {
>             "type": "add_column_substr",
>             "target_column": 1,
>             "length": 4,
>             "substr_type": "drop_suffix"
>           }
>         ],
>         "mapping": [
>           {
>             "type": "url",
>             "columns": [
>               0
>             ]
>           },
>           {
>             "type": "list_identifiers",
>             "columns": [
>               2
>             ],
>             "editing": false
>           }
>         ]
>       }
>       ```
>    9. Click on **Apply**
>    10. In the bottom row of options, change **Type** from "Auto-detect" to `vcf`
>    11. Set the name of the new collection to `MTB sample variants`
>    12. Click on **Upload**
>
>    It is ok to continue with the next step even while the upload of this data is still ongoing.
>    Just click on the **Close** button of the information pop-up window if it is still open when you are ready to proceed.
> 2. {% tool [Upload](upload1) %} the reference genome that was used for SNP calling from:
>
>    ```
>    https://zenodo.org/record/3497110/files/MTB_ancestor_reference.fasta
>    ```
>
>    and make sure the dataset format is set to `fasta`.
>
>    {% snippet faqs/galaxy/datasets_import_via_link.md format="fasta" %}
>
{: .hands_on}


## Generate a SNP alignment

In this tutorial we aim to calculate the genetic differences (in SNPs) between pairs of MTB genomes.
To do so, we need to compare, between each pair of genomes (thus pairwise), the nucleotides that are
observed at each position. Each time we find a different nucleotide at a given position, we will sum
1 SNP of genetic distance. For example, if two strains have a genetic distance of 5 SNPs, that would
mean that their respective genomes are almost identical, except for 5 positions along all the genome
in which they have different nucleotides.

To do such calculation we need to first build an alignment of all the genomes (a multiple-sequence alignment, or MSA).
Afterwards, we will use specific software to analyze this MSA, count SNPs, and thus calculate the genetic distance between each pair of samples.

![MSA of 5 MTB genomes](./images/MSA.png "A SNP is highlighted in a MSA of five MTB genomes")

### Generate complete genomes

The first step to generate the genomes MSA will be... to get the complete genomes of our samples!
In the [MTB Variant Analysis tutorial]({% link topics/variant-analysis/tutorials/tb-variant-analysis/tutorial.md %}) we have analyzed short-read high-throughput sequencing data (Illumina) to
obtain the respective VCF files that describe the mutations found in each of our samples, as compared
to the reference genome. We can now use these VCF files to build the complete genome of each of our
samples.

#### Filter VCF files for epidemiological/phylogenetic investigation
Interpreting **mixed calls** or **indels** in phylogenetic/epidemiological applications can be very
complicated. That is the reason why we typically use alignments that only contain **fixed SNPs**.

Thus, the first step in this tutorial will be to filter the VCFs so we are sure that
they only contain **fixed SNPs**, which we (somewhat arbitrarily) define as those single-nucleotide variants that are observed at an allele frequency of 90% or greater. We will be using the tool **TB Variant Filter** for this task.

*Note: TB variant Filter refers to SNPs as SNVs. These two short forms are interchangeable, meaning **S**ingle **N**ucleotide **P**olymorphism and **S**ingle **N**ucleotide **V**ariant, respectively.*

> <hands-on-title>Filter VCF files for epidemiological investigation</hands-on-title>
>
> 1. {% tool [TB Variant Filter](toolshed.g2.bx.psu.edu/repos/iuc/tb_variant_filter/tb_variant_filter/0.3.6+galaxy0) %} with the following parameters:
>    - {% icon param-collection %} *"VCF file to be filter"*: `MTB variants per sample`; the uploaded collection of VCF datasets
>    - *"Filters to apply"*: `Only accept SNVs`, `Filter variants by percentage alt allele`
>    - *"Show options for the filters"*: `Yes`
>    - *"Minimum alternate allele percentage to accept"*: `90.0`
>
> > <question-title></question-title>
> >
> > **TB Variant Filter** reads the VCF and output only SNPs that have, at least, 90% frequency.
> > How can the software extract such information from the VCF datasets?
> >
> > > <solution-title></solution-title>
> > >
> > > That information is contained, for each mutation, in the VCF:
> > > - The `TYPE` field within the INFO string will tell us if the mutation is a SNP (TYPE=snp)
> > > - You can look for other types of mutations like insertions (TYPE=ins)
> > > - The `AF` field within the INFO string describes the estimated **A**llele **F**requency
> > > - In the absence of the `AF` field, the allele frequency can be calculated as: `AO` (alternate allele observations) / `DP` (depth of reads at the variant site)
> > {: .solution}
> {: .question}
{: .hands_on}

#### Reconstruct the complete genome of each sample
Our new collection of VCF datasets now only contains **fixed SNPs** that were found in the genome of the respective strains.
Genomic positions not in the VCF mean that, at that particular position, the strain has the
same nucleotide than the reference genome. Knowing this information, one could reconstruct the
complete genome of each strain pretty easily. From the first to the last position in the genome,
one would put the same nucleotide than in the reference if that position is not in the VCF, or the
SNP described in the VCF otherwise. This is exactly what the tool **bcftools consensus** will do for us,
given the reference genome and the VCF of the strain we want to reconstruct the genome for.

> <hands-on-title>Reconstruct the complete genome of each sample</hands-on-title>
>
> 1. {% tool [bcftools consensus](toolshed.g2.bx.psu.edu/repos/iuc/bcftools_consensus/bcftools_consensus/1.15.1+galaxy3) %} with the following parameters:
>    - {% icon param-collection %} *"VCF/BCF Data"*: the collection of filtered variants per sample; output of **TB Variant Filter**
>    - *"Choose the source for the reference genome"*: `Use a genome from the history`
>        - {% icon param-file %} *"Reference genome"*: the uploaded MTB ancestor reference fasta dataset (the reference genome that was used for SNP-calling)
>        - *"Set output FASTA ID from name of VCF"*: `Yes`
>
{: .hands_on}

> <question-title></question-title>
>
> Imagine that we forgot to filter the VCFs to contain only fixed variants, and there are also
> SNPs with frequencies, of 15%, 30%, or 56.78%. Which allele do you think bcftools consensus would
> insert in the genome?
>
> > <solution-title></solution-title>
> > The behaviour of bcftools consensus in this case can be specified with the option `--haplotype`
> > For example, we can set `haplotype=2` so the second allele will be used... wait... what?
> {: .solution}
{: .question}

> <question-title>Second allelle!?</question-title>
>
> What do you think that things like "second allele" or "*The* alterntive allele" mean here?
>
> > <solution-title></solution-title>
> > Many of the bioinformatic programs are developed to analyze eukaryotic genomes, particularly
> > human genomes. That means that these programs have in mind that the genomes
> > are diploid and thus each posible position in the genome has two possible alleles. In
> > bacterial genomics, in contrast, we are **always sequencing a population** of cells with
> > potential genetic diversity (with the exception of single-cell sequencing).
> > That does not mean that we cannot use this type of software, we can (and we do!) but it is
> > good to know what they are meant for, and their possible limitations.
> {: .solution}
{: .question}

### Multiple-sequence alignment (MSA) of all genomes

Multiple sequence alignment is a process in which multiple DNA, RNA or protein sequences are arranged
to find regions of similarity that are supposed to reflect the consequences of different evolutionary
processes (Figure 1). MSAs are used to test hypotheses about these evolutionary processes and to infer phylogenetic
relationships, and for these reasons we build MSA for sequences for which we already assume some sort
of evolutionary relationship. You will learn more on MSAs and phylogenetic inference in the next
tutorial.

Building MSAs of several complete genomes can be a complicated process and computationally demanding. To perform
such a task there are many software packages available like **Muscle**, **MAFFT** or **Clustal**, just to mention some.
Which one are we going to use in this tutorial? Well, we are going to use a trick. We are going to
just stack one genome on top of each other within a text file. (More on why we can do this below).

Our aim is to generate a **multifasta** dataset in which the genomes of our samples are aligned.
Something that looks like this:

```
>Sample 1
TTGACCGATGACCCCGGTTCAGGCTTCACCACAGTGTGGAACGC...
>Sample 2
ATGACCGATGACCCCGGTTCAGGCTTCACCACAGTGTGGAACGC...
>Sample 3
TTCGTTCGATGACCCCGGTTCAGGCTTCACCACAGTGTGGAACG...
```

This could be done manually, by copy-pasting all genomes in a single text file.
However we can do the same with a specific command that *concatenates* datasets.

> <hands-on-title>Concatenate genomes to build a MSA</hands-on-title>
>
> 1. {% tool [Concatenate datasets tail-to-head (cat)](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1) %} with the following parameters:
>    - {% icon param-collection %} *"Datasets to concatenate"*: the collection of consensus genomes; output of **bcftools consensus**
>
{: .hands_on}

Now we have a multifasta file, where **each position of each genome corresponds to the same position
of the rest of genomes in the file**. This can be seen and used as a multiple-sequence alignment of
all of our genomes! However, it is important that you understand the following question...

> <question-title></question-title>
>
> Generating multiple-sequence alignments can be complicated and computationally demanding, and there are
> many software packages to perform such task. How is it possible that we were able to build a MSA by just
> stacking genomes one on top of each other? Can you think about what makes our case special, so we can
> just use this "trick"?
>
> > <solution-title></solution-title>
> >
> > We have generated the complete genome of each sample by substituting in the reference genome
> > those **SNPs** that we found in that said sample (described in the VCF). Remember that **we
> > removed indels from the VCF when filtering!** Because the complete genomes we generated do not
> > contain insertions or deletions, ALL the genomes have the same length (the length of the
> > reference genome) and each nucleotide corresponds to the same genomic coordinate (the one
> > also in the reference genome). So we are not aligning genomes *per se* and, knowing this, we
> > can build a MSA by just stacking genomes that **are the same length >>AND<< have the same coordinates**.
> {: .solution}
>
{: .question}

### Remove invariant positions

We have generated a MSA that is the basis for the transmission (clustering) and phylogenetic
analysis. Although we could already use this MSA for such analysis, it is common practice to remove
the invariant sites from the alignment. Think that our file now contains 20 genomes of 4.4 Mb each.
MTB genomes have a very low genetic diversity, meaning that in reality, there are only some hundreds or
thousands of SNPs *in total*, because the genomes are >99% identical between them. Identical positions
in a MSA provide no information, so we can remove those and
generate a **SNP alignment** that only contain variant positions with phylogenetic information. By
doing this, we will generate a much smaller file, that will be easier to handle by downstream applications.

We can exemplify this with a couple of pictures:

In the following picture, a polymorphic position in the alignment (SNP)
is highlighted in blue and green. Invariant positions are not highlighted and have an asterisc `*`
on the upper part of the alignment. Most of the MSA is composed of these invariant positions, making
our file larger than necessary.

![MSA of 5 MTB genomes](./images/MSA.png "A SNP is highlighted in a MSA of five MTB genomes")

After removing invariant positions, we end up with a SNP alignment like the following.

![MSA of 5 MTB genomes](./images/SNP_MSA.png "A SNP alignment where all positions are polymorphic")

> <hands-on-title>Removing invariant sites from a MSA</hands-on-title>
>
> 1. {% tool [Finds SNP sites](toolshed.g2.bx.psu.edu/repos/iuc/snp_sites/snp_sites/2.5.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"FASTA file"*: the concatenated consensus genomes; output of **Concatenate datasets**
>    - *"Output"*: `Sequence alignment / VCF`
>        - *"Output formats"*: `Multi-FASTA alignment file`
>
{: .hands_on}

In SNP alignments you have to **bear in mind that positions do not longer correspond to genomic
coordinates**, meaning that two contiguous nucleotides may correspond to coordinates thousands of
positions apart.

## Identify transmission clusters

### Calculate pairwise SNP distances

Now we are all set to calculate pairwise SNP distances between samples and decide whether two
patients are within the same transmission cluster or not. Having a SNP alignment, this is fairly
easy. We will use **SNP distance matrix**, that will generate a matrix with pairwise SNP distances.

> <hands-on-title>Distance matrix from SNP alignment.</hands-on-title>
>
> 1. {% tool [SNP distance matrix](toolshed.g2.bx.psu.edu/repos/iuc/snp_dists/snp_dists/0.8.2+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"FASTA multiple sequence alignment"*: the concatenated SNP sites-only sequences; output of **Finds SNP sites**
>
> > <comment-title></comment-title>
> >
> > Have a look at the distance matrix to make sure you understand the whole process. Given that
> > we only have 20 samples, you could already spot some samples that are involved in the same
> > transmission chain (samples with a small number of SNPs between them as explained below).
> {: .comment}
>
{: .hands_on}

### Determine transmission clusters based on a SNP threshold

Now that we have a distance matrix that describes the SNP distance between each pair of samples, we
could already describe the transmission clusters based on a **SNP threshold**, as [explained in the
respective webinar](https://youtu.be/kKNgmpy1N94) . If two samples are at a distance below that threshold, we will say that they
belong to the same transmission cluster, because they are close enough genetically speaking. Again,
we could do this manually, but we are doing bioinformatics, and we want to be able to do the same
analysis regardless of whether we are analyzing two or two million samples. Also, **note that two samples
that are dozens of SNPs apart may belong to the same transmission cluster** if there are other samples
linking them in between as exemplified in the picture below.

![MSA of 5 MTB genomes](./images/Cluster.png "Example of transmission cluster using 10 SNPs threshold")

> <question-title>Very Important Question</question-title>
>
> 1. In the image above exemplifying a transmission cluster, the distance between samples A and E is
> 17 SNPs. Being the other pairwise distances in the figure the same,
> would it be possible that the distance between A and E is different?
>
> > <solution-title></solution-title>
> >
> > 1. The figure used above as an example is a **flagrant oversimplification**. In the figure not all pairwise
> > distances are represented (for example between sample A and C).
> > Most importantly you have to remember that **transmission clusters do not reflect transmission
> > events**.
> > In fact it may happen that transmission does not happen within the cluster we are analyzing!
> > For example, when a patient that is not sampled is the source of infection of all the cases in
> > the cluster (may be a superspreading event). You have to consider that, taking into account the
> > same SNP distances, another possible (yet still oversimplified) scenario could be...
> > ![MSA of 5 MTB genomes](./images/Cluster2.png "Example of transmission cluster using 10 SNPs threshold")
> {: .solution}
{: .question}

### Determine transmission clusters 
The Galaxy team has implemented a tool that reads a SNP distance matrix like the one we just calculated and 
assigns clusters based on a distance threshold. We will use this tool to perform a clustering with a threshold
of 12 SNPs.

> <hands-on-title>Distance matrix from SNP alignment.</hands-on-title>
>
> 1. {% tool [Distance matrix-based hierarchical clustering tool metadata](toolshed.g2.bx.psu.edu/repos/iuc/clustering_from_distmat/clustering_from_distmat/1.1.1) %} with the following parameters:
>    - {% icon param-file %} *"Distance matrix"*: SNP distance matrix; output of **SNP distance matrix**
>    - *"Clustering method"*: UPGMA (scipy 'average' method)
>    - *"Generate cluster assignments?"*: Yes, and use distance threshold to divide into clusters
>        - *"Distance threshold for sample clustering"*: **12**
>
{: .hands_on}

> <question-title></question-title>
>
> How many transmission clusters did we find? How many samples are linked to recent transmission in our dataset?
>
> > <solution-title>Answer</solution-title>
> > We have found two transmission clusters with respective IDs 1 and 2. Transmission cluster 1
> > is composed of two samples,*ERR5987352* and *ERR6362484* , linked by recent transmission and transmission cluster 2 of three samples, *ERR6362138*, *ERR6362156* and *ERR6362253*.
> {: .solution}
{: .question}

> <question-title></question-title>
>
> Let's assume that we have the isolation dates of samples ERR6362484 and ERR5987352, which
> belong to the same transmission cluster. Sample ERR6362484 was isolated on January 2021, while sample
>  ERR5987352 was isolated on September 2021. Would you be able to determine who was the infector and who the infectee?
>
> > <solution-title></solution-title>
> > **NO**
> >
> >  Isolation dates have been used traditionally to define **index cases** within transmission clusters
> > under the assumption that the most likely scenario is the first isolated sample to be the
> > source of transmission. Today we know that this assumption often leads
> > to misidentification of index cases. **Remember:** we cannot rule out the possibility
> > that patients within the cluster were infected by an index case that was not sampled.
> >
> >  Read {% cite Xu_2019 %} for more information on this topic.
> {: .solution}
{: .question}


# Analysis Part 2: Combining cluster information with drug-resistance reports

Although we have stressed the fact that **clustering cannot be used to delineate
transmission events**, clustering is very useful to investigate outbreaks and determine which cases are
involved in the same transmission chain. We can leverage this information to investigate the relationship
between tuberculosis transmission and particular biological or clinical traits.

In this part of the tutorial, we will investigate the emergence and spread of drug resistance based
on our clustering analysis.

## Get the data

The [MTB Variant Analysis tutorial]({% link topics/variant-analysis/tutorials/tb-variant-analysis/tutorial.md %})
demonstrates the use of **TB-profiler** to generate a report with determinants of drug resistance of a
particular MTB strain, and predict its genotypic drug susceptibility. We have done **exactly the same**
for the 20 samples that we used in the clustering analysis, so we have now the TB-profiler reports for
all of them.

Like in part 1, we can use Galaxy's rule-based uploader functionality to upload the data, build a collection from it and name its elements correctly, all in one step.

> <hands-on-title>Data upload</hands-on-title>
>
> 1. Import the TB-profiler reports for the samples
>
>    ```
>    https://zenodo.org/record/6010176/files/ERR1203059.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR181435.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR2659153.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR2704678.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR2704679.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR2704687.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR313115.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR551620.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR5987300.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR5987352.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362078.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362138.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362139.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362156.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362253.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362333.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362484.TBprof.txt
>    https://zenodo.org/record/6010176/files/ERR6362653.TBprof.txt
>    https://zenodo.org/record/6010176/files/SRR13046689.TBprof.txt
>    https://zenodo.org/record/6010176/files/SRR998584.TBprof.txt
>    ```
>
>    1. Copy the links above
>    2. Open the {% tool [Upload](upload1) %} Manager
>    3. In the top row of tabs select **Rule-based**
>    4. Set **Upload data as** to `Collection(s)`
>    5. Paste the copied links into the text field on the right
>    6. Click **Build** to bring up the *Rule Builder* dialog
>    7. Click on the wrench icon {% icon tool %} next to **Rules** in the left panel of the window
>    8. In the text field, erase the prefilled content and copy/paste the following rule definition into it instead:
>
>       ```
>       {
>         "rules": [
>           {
>             "type": "add_column_basename",
>             "target_column": 0
>           },
>           {
>             "type": "add_column_substr",
>             "target_column": 1,
>             "length": 11,
>             "substr_type": "drop_suffix"
>           }
>         ],
>         "mapping": [
>           {
>             "type": "url",
>             "columns": [
>               0
>             ]
>           },
>           {
>             "type": "list_identifiers",
>             "columns": [
>               2
>             ],
>             "editing": false
>           }
>         ]
>       }
>       ```
>    9. Click on **Apply**
>    10. In the bottom row of options, change **Type** from "Auto-detect" to `txt`
>    11. Set the name of the new collection to `TB profiler reports`
>    12. Click on **Upload**
>
{: .hands_on}

## Summarize the data

TB-profiler reports are very useful and comprehensive, and we will use them to better investigate
drug resistance in our dataset. However, it is always useful to summarize the data on a per-sample basis,
on a table, so we can quickly check which strains are, for example, MDR, an which are susceptible to antibiotics.
We would like to generate a table like the following:

| Sample   | DR profile |
|----------|------------|
| Sample A | Sensitive  |
| Sample B | Sensitive  |
| Sample C | MDR        |
| Sample Z | XDR        |

and merge this with the tabular per-sample cluster report that we have generated in the first part of the tutorial.

If we take a look at a TB profiler report we can see that there is one line describing the genotypic drug susceptibility.
Let's have a look at the first part of the TB-profiler report for sample ERR6362653:

```

TBProfiler report
=================

The following report has been generated by TBProfiler.

Summary
-------
ID: tbprofiler
Date: Fri Jan 28 13:14:47 2022
Strain: lineage2.2.1
Drug-resistance: MDR

Lineage report
--------------
Lineage	Estimated Fraction	Family	Spoligotype	Rd
lineage2	1.000	East-Asian	Beijing	RD105
lineage2.2	0.996	East-Asian (Beijing)	Beijing-RD207	RD105;RD207
lineage2.2.1	0.999	East-Asian (Beijing)	Beijing-RD181	RD105;RD207;RD181
```

As you can see, this strain is multi-drug resistant as indicated by (`Drug-resistance: MDR`)
We could then look for this information in each TB-profiler report and generate this table manually,
for example in a spreadsheet. However this is not feasible when analyzing hundreds or thousands of samples
(and very **error-prone!**).

A far more scalable approach is to combine Galaxy tools to automate the necessary data processing steps for us.

### Select the line containing the drug resistance profile in the report of every sample

**grep** is used to search patterns of text within text files. Each time **grep** finds that
pattern, it will print as a result **the complete line** containing such pattern.

If we use **grep** to search for the pattern "`Drug-resistance:`", in a TB-profiler file, we will get
as output the complete line, for example `Drug-resistance: MDR`

> <hands-on-title>Keep only `Drug-resistance:` lines from TB-profiler reports</hands-on-title>
>
> 1. {% tool [Select lines that match an expression](Grep1) %} with the following parameters:
>    - {% icon param-collection %} *"Select lines from"*: the uploaded collection of TB-profiler reports
>    - *"that"*: `Matching`
>    - *"the pattern"*: `Drug-resistance:`
>
{: .hands_on}

### Collapse collection into a table with each line prepended with a sample name

Next, we want to turn the collection of single-line datasets we just created into a single two-column table with
one column of sample IDs and another one with the corresponding drug resistance status.
> <hands-on-title>Prepend the sample name to the DR profile</hands-on-title>
>
> 1. {% tool [Collapse Collection](toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0) %} with the following parameters:
>    - {% icon param-collection %} *"Collection of files to collapse into single dataset"*: the collection of single-line drug-resistance reports; output of **Select lines**
>    - *"Prepend File name"*: `Yes`
>      - *"Where to add dataset name"*: `Same line and each line in dataset`
>
{: .hands_on}

### Clean up the table

In this step we will use a simple tool that searches and replaces text to remove the redundant string "Drug-resistance: " from our table by replacing it with *nothing*.

> <hands-on-title>Removing redundant content</hands-on-title>
>
> 1. {% tool [Replace Text](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2) %} with the following parameters:
>    - {% icon param-file %} *"File to process"*: `Single file` (output of **Collapse Collection** {% icon tool %})
>    - In *"Replacement"*:
>        - {% icon param-repeat %} *"Insert Replacement"*
>          - *"Find pattern"*: `Drug-resistance: `
>
>              Note the space at the end of the pattern!
>          - *"Replace with"*: leave this field empty
>
> 2. Once the tool has finished running, change the format of the output to **tabular**
>
>    Though by now the content of the dataset looks like a nice table, Galaxy still thinks it is in general **txt** format because the original TB-profiler reports were of that format.
>
>    {% snippet faqs/galaxy/datasets_change_datatype.md datatype="tabular" %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How many MDR strains did we find in the dataset?
> 2. What does it mean to be Pre-MDR?
>
> > <solution-title></solution-title>
> > 1. Eight MDR strains, three of which are pre-XDR because they have additional resistance to fluoroquinolones. (You can look into details by looking into the TB profiler reports).
> > 2. As MDR means to be resistant to INH and RIF, pre-MDR means to be either INH-monoresistant or RIF-monoresistant.
> > If we have a look at the respective TB-profiler reports, we can see that these three strains are RIF-monoresistant.
> {: .solution}
{: .question}

## Combine the drug-resistance and the cluster tables

The two tables we have generated in the two parts of the tutorial share a sample column, which we can use now to join the two tables.

> <hands-on-title>Joining tables</hands-on-title>
>
> 1. {% tool [Join two files](toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2) %} with the following parameters:
>    - {% icon param-file %} *"1st file"*: transmission clusters report; output of *Distance matrix-based hierarchial clustering*
>    - *"Column to use from 1st file"*: `Column: 1`
>    - {% icon param-file %} *"2nd File"*: final drug-resistance report; output of **Replace Text** above
>    - *"Column to use from 1st file"*: `Column: 1`
>    - *"Output lines appearing in"*: `Both 1st & 2nd file`
>
>
{: .hands_on}


# Interpretation of results

Now that we have performed a clustering analysis and know which DR mutations each strain carries,
we can try to answer a series of questions about how DR may have been emerging and spreading in our study
population.

The final combined table you generated and that we are going to base some questions on, should look similar to this one (header added and sample order rearranged for clarity here):

| Sample       | Cluster_id | DR profile |
|--------------|------------|------------|
| ERR1203059   | -          | Sensitive  |
| ERR181435    | -          | Sensitive  |
| ERR2659153   | -          | Sensitive  |
| ERR2704678   | -          | Sensitive  |
| ERR2704679   | -          | Sensitive  |
| ERR2704687   | -          | Sensitive  |
| ERR313115    | -          | Sensitive  |
| ERR551620    | -          | MDR        |
| ERR5987300   | -          | Pre-XDR    |
| ERR6362078   | -          | MDR        |
| ERR6362139   | -          | Pre-MDR    |
| ERR6362333   | -          | Pre-XDR    |
| ERR6362653   | -          | MDR        |
| SRR13046689  | -          | Other      |
| SRR998584    | -          | Sensitive  |
| ERR5987352   | 1          | Pre-MDR    |
| ERR6362484   | 1          | Pre-MDR    |
| ERR6362138   | 2          | MDR        |
| ERR6362156   | 2          | Pre-XDR    |
| ERR6362253   | 2          | MDR        |

> <question-title></question-title>
>
> Assuming that we have a very good sampling of the outbreak. Which strains **may** represent instances
> of *de novo* evolution of drug resistance and which ones instances of *transmitted* (primary) resistance?
> Remember that you can look at the TB-profiler reports of independent samples for detailed information.
>
> > <solution-title></solution-title>
> > In a simplistic scenario, we could consider clustered strains as instances of transmission and
> > unclustered strains as instances of de novo evolution of DR. Thus, we see that for example there
> > are three MDR strains (ERR551620, ERR6362078, ERR6362653) that are unclustered and therefore may
> > represent cases in which drug resistance evolved independently as a consequence of treatment within the respective
> > patients. However you need to always bear in mind that, although **within our population** those MDR
> > strains do not seem to be linked to transmission, this does NOT rule out the possibility that
> > some of these patients were infected with an MDR strain somewhere else, or that other cases from the cluster where simply not sampled.
> >
> > When looking at clustered strains, distinguishing between transmitted and *de-novo* may be tricky. Note
> > that, for example, for the two RIF-monoresistant strains linked within the same transmission
> > cluster there are, at least, a couple of possible scenarios: one in which a RIF-monoresistant strain
> > evolved in one patient and was transmitted to the other patient afterwards, and one in which both
> > were infected with the same RIF-monoresistant strain from a third patient that we have not sampled.
> > We need to note that, in the first scenario, drug resistance evolved *de novo* in one patient,
> > and was *transmitted* to the other patient, whereas in the second scenario drug resistance was
> > *transmitted* in both cases.
> {: .solution}
{: .question}

> <question-title></question-title>
>
> 1. The same principles as explained above apply to the three MDR strains that are
>    within the same transmission cluster. In this case, however, there is one strain that shows clear
>    evidence of *de-novo* evolution of DR. Do you know which strain and why?
>
> > <solution-title></solution-title>
> > 1. Within this cluster of MDR strains, there is one tagged as Pre-XDR by TB-profiler. If we have
> >    a look at the TB profiler report, we can see that this strain carries an additional mutation in
> >    *gyrA* that confers resistance to fluoroquinolones. This is compatible with an scenario in which
> >    fluoroquinolone resistance evolved independently within this patient after being infected with
> >    the MDR strain.
> {: .solution}
{: .question}

> <question-title></question-title>
>
> There is one strain with a DR profile "other", because it is only resistant to pyrazinamide. This
> strain is not within a transmission cluster. Therefore, we might conclude that pyrazinamide resistance
> most likely evolved *de-novo* in this patient due to antibiotic treatment, but we would be wrong. Do you know why?
>
> > <solution-title></solution-title>
> > The strain is indeed PZA-resistant. And indeed this is strain is NOT linked to transmission
> > within our population. However, if we have a look at the TB-profiler report, we observe that this
> > is a *M. bovis* strain, many of which are known to be intrinsically resistant to PZA.
> {: .solution}
{: .question}

> <question-title></question-title>
>
> 1. Is it possible to find, in the same transmission cluster, two RIF-monoresistant strains that
>    carry different rpoB mutations?
> 2. Is it possible to find, in the same transmission cluster, strains of different MTB sublineages?
>
> > <solution-title></solution-title>
> > 1. Yes, it is **possible**. In that scenario, the **same susceptible strain** would **recently**
> >    have been transmitted to two patients, and RIF resistance would have evolved **independently** in both.
> > 2. No, by definition. Remember that clustering is based on a threshold that we set of genetic
> >    distance measured in SNPs. We want to cluster samples that are genetically so similar that we
> >    can consider them as the same genotype, that is to say, as the same strain. Two different
> >    sublineages, by definition, do not belong to the same genotype and will have a distance in SNPs
> >    between them well beyond any SNP threshold we could use.
> {: .solution}
{: .question}


# Conclusion

You have learned how to perform a clustering analysis to identify patients that are linked by events of
**recent transmission**. Clustering analysis is very useful in outbreak investigation and
can also be used to describe the emergence and spread of drug-resistance within a population. You have
also learned, however, that interpreting clustering results requires careful considerations, given the
limitations of the methodology. Clustering analysis is better complemented with phylogenetic analysis,
which may help overcome some of these limitations.

In the following tutorial you will perform a phylogenetic analysis of these same 20 strains.


# Bonus

You might have noticed that one of the strains analyzed presents thousands of differences (SNPs) to
the reference genome, standing out from the rest of strains. This strain is a *M. canettii* strain,
that was actually not part of the outbreak investigated. However we decided to include it here. Why? Let's find out
in the follow-up tutorial [Tree thinking for tuberculosis evolution and epidemiology]({% link topics/evolution/tutorials/mtb_phylogeny/tutorial.md %}).

tools_json:
["Upload", "upload1", "TB Variant Filter", "toolshed.g2.bx.psu.edu/repos/iuc/tb_variant_filter/tb_variant_filter/0.3.6+galaxy0", "bcftools consensus", "toolshed.g2.bx.psu.edu/repos/iuc/bcftools_consensus/bcftools_consensus/1.15.1+galaxy3", "Concatenate datasets tail-to-head (cat)", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_cat/0.1.1", "Finds SNP sites", "toolshed.g2.bx.psu.edu/repos/iuc/snp_sites/snp_sites/2.5.1+galaxy0", "SNP distance matrix", "toolshed.g2.bx.psu.edu/repos/iuc/snp_dists/snp_dists/0.8.2+galaxy0", "Distance matrix-based hierarchical clustering tool metadata", "toolshed.g2.bx.psu.edu/repos/iuc/clustering_from_distmat/clustering_from_distmat/1.1.1", "Select lines that match an expression", "Grep1", "Collapse Collection", "toolshed.g2.bx.psu.edu/repos/nml/collapse_collections/collapse_dataset/5.1.0", "Replace Text", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_replace_in_line/1.1.2", "Join two files", "toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/tp_easyjoin_tool/1.1.2"]


