# tutorials

topic: teaching
tutorial: hybrid-training
title: Hybrid training
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: 0a39b7d245905ebf39464c7b98b99b14
last_modified: 2026-02-26T21:12:18.334349
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/hybrid-training/tutorial.html

content:
**hybrid training events**, i.e. pairing-up on-site helpers with remote instructors across multiple sites simultaneously:

![Hybrid training diagram, a single remote instructor broadcasts training to multiple sites](./images/hybrid_training.png)

With this model, we aim to bring training events to the trainees while reducing the environmental impact of instructor travel.

> <tip-title>Combine with Australia Biocommon paper</tip-title>
> This is an excellent companion paper: {% cite Hall_2021 %}
{: .tip}

# What is an hybrid training event?

*To illustrate a typical Gallantries event and the different roles (1 person can have several roles), we have decided to share a story with several fictional characters*

**Max** is a training coordinator of an international research consortium. They recently surveyed researchers in the consortium on their need in HTS data analysis. The results were overwhelming: over 60 scientists at the several locations would like to learn how to analyze their own RNA-seq data. To fulfill this demand, Max decided to organize a Gallantries event: a 3-days workshop with 4 different locations simultaneously (Greece, Estonia, France and Spain). Max is the **global organizer** of this event, in charge of finding the date, contact and coordinate with the local hosts, advertize the event, find the instructors, etc. After the event, Max will aggregate all feedback from participants, helpers and hosts and share them.

Max first contacted 4 **local hosts** (1 at each site):

One of them is **Imani** who is based in Greece. Imani is in charge of finding a suitable room (preferably with computers), checking the local setup, recruiting local helpers, advertizing the event locally to participants, and organizing the local catering and social event. Imani will also give the introduction and wrap-up on each day, and collecting feedback from participants.

Imani recruited 2 **local helpers** for each day. One of the helpers, **Casey**,  has some previous experience with RNA-seq data analysis and Galaxy. Before the workshop, they went through the training material and tested it. They will help participants during the workshop when they are stuck or get different results than the instructors. They will also give direct feedback to the instructor about the pace, any possible local issues etc.

Max also recruited 4 **instructors**, **Farah** among them. Farah is a trained bioinformatician and experienced instructor, regularly giving training to scientists on HTS data analysis. They will teach in the morning of the 2nd day (introduction and the first steps of RNA-seq data analysis). Farah is located in Germany and teaches from a room at their institute in front of their computer. During their session, they will adapt their pace given the feedback they receive from the local helpers, check the status of participants job on a dedicated page, and will also answer questions from participants written on chat.

**Alex** is a PhD student in molecular biology based in Greece. They would like to learn about RNA-seq data analysis to be able to analyze the data they generated. They heard about the workshop and join one site for the workshop close to their institute. As a **participant**, Alex will actively participate in the workshop by running their first RNA-seq data analysis given the instructor's instructions, ask for help from local helpers when stuck, will raise their questions on the participant chat, and will give feedback using sticky notes and the dedicated feedback form.

# Cost of an event

With the hybrid training model, the cost of organizing and participating in the workshop are minimized: the instructors do not need to travel to the venue and the event can be the closest as possible to participants (e.g. in their institute).

We recommend that local hosts organize drinks and coffee for the breaks and, if possible, lunch. These will be the major costs of hosting such an event.

To cover those costs, local hosts can ask for a small participation fee. This will also increase the number of registered participants showing up.

# Workshop Checklists

To help you organize an hybrid event, we have created some checklists, by timing but also [by role](#checklists-by-role). Most of the items in these checklists are not specific to Gallantries events.

## Before the workshop

### Global organizers

{% include topics/teaching/tutorials/hybrid-training/organizer-before.md %}

### Local hosts

{% include topics/teaching/tutorials/hybrid-training/host-before.md %}

### Local helpers

{% include topics/teaching/tutorials/hybrid-training/helper-before.md %}

### Instructors

{% include topics/teaching/tutorials/hybrid-training/instructor-before.md %}

## During the workshop

Templates for chat
How chat works
emphasize communication between helpers
put sticky notes there quickly

### Global organizers

{% include topics/teaching/tutorials/hybrid-training/organizer-during.md %}

### Local hosts

{% include topics/teaching/tutorials/hybrid-training/host-during.md %}

### Local helpers

{% include topics/teaching/tutorials/hybrid-training/helper-during.md %}

### Instructors

{% include topics/teaching/tutorials/hybrid-training/instructor-during.md %}

## After the workshop

Debrief
    Sticky notes collection from sites
    share experience back: feedback from sites (extract form from issue in github)
    form feedback from instructors + helpers

### Global organizers

{% include topics/teaching/tutorials/hybrid-training/organizer-after.md %}

### Local hosts

{% include topics/teaching/tutorials/hybrid-training/host-after.md %}

### Local helpers

{% include topics/teaching/tutorials/hybrid-training/helper-after.md %}

### Instructors

{% include topics/teaching/tutorials/hybrid-training/instructor-after.md %}

# Checklists by role

## Global organizers

### Before the workshop

{% include topics/teaching/tutorials/hybrid-training/organizer-before.md %}

### During the workshop

{% include topics/teaching/tutorials/hybrid-training/organizer-during.md %}

### After the workshop

{% include topics/teaching/tutorials/hybrid-training/organizer-after.md %}

## Local hosts

### Before the workshop

{% include topics/teaching/tutorials/hybrid-training/host-before.md %}

### During the workshop

{% include topics/teaching/tutorials/hybrid-training/host-during.md %}

### After the workshop

{% include topics/teaching/tutorials/hybrid-training/host-after.md %}

## Local helpers

### Before the workshop

{% include topics/teaching/tutorials/hybrid-training/helper-before.md %}

### During the workshop

{% include topics/teaching/tutorials/hybrid-training/helper-during.md %}

### After the workshop

{% include topics/teaching/tutorials/hybrid-training/helper-after.md %}

## Instructors

### Before the workshop

{% include topics/teaching/tutorials/hybrid-training/instructor-before.md %}

### During the workshop

{% include topics/teaching/tutorials/hybrid-training/instructor-during.md %}

### After the workshop

{% include topics/teaching/tutorials/hybrid-training/instructor-after.md %}


topic: teaching
tutorial: assessment
title: Assessment and feedback in training and teachings
difficulty: intermediate
time_estimation: 3h30m
tools_json: []
requirements_json: []
tags_json: ["elixir", "train-the-trainers"]
content_hash: db5ef916883ecbfdef20620dec4b7a42
last_modified: 2026-02-26T21:12:18.330220
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/assessment/tutorial.html

content:
**Assessment** is the evaluation or estimation of the nature, quality, or ability of someone or something, while **feedback** integrate the output of the assessment in the learning process.

Assessment and feedback phase falls in the overall Nicholls process ({% cite via2020course %}, {% cite nicholls2002developing %}):

![A flow chart of the five phases of the instructional design. The 1st phase is to select the Learning Outcomes, followed by the selection of the corresponding Learning Experiences. The third phase is to select the content that is relevant to the LO. The fourth phase is the development of the necessary assessments for the learners, with the final phase being the evaluation of the effectiveness of the LE and the LO, at which point the flow connects back to the first phase for the refinement of the process.](images/NichollsPhases.png "5 phases of instructional design described by Nicholls et al in Developing teaching and learning in higher education. London, UK: Routledge. pp 51-75")

Teaching goal for this session is to develop an understanding of **different types of feedback and assessments**, when to give and receive feedback, and for which **purpose**.

# Assessments

There are different types of assessments:

Assessment | Timing | Objectives
--- | --- | ---
**Pre-course assessment** | Before the course | verify the target audience of the course
**Preventive assessment** | Beginning of the course | final adjustments of the course to the reality of the participants
**Formative assessment** | During the course | pilot in real time if learning is taking place
**Summative assessment** | Right after the course | measure and evaluate the knowledge and skills acquired
**Strategic evaluation** | After the course, long time after the course | measure the adequacy, quality and impact of the course

They can be categorised in different ways:

![A representation of the mental model for assessment, as a tree. It shows the various categories of assessment, as well as the optimal placement of them during the teaching process.](images/assessment-categories.png)

> <hands-on-title>Diagnostic Assessment - {% icon time %} 15 min - Groups of 4</hands-on-title>
>
> Discuss how you could collect information from learners at the beginning of a course or lesson and how you can use it
{: .hands_on}

<!--{% include _includes/quiz.html id="diagnostic_assessment.yaml" %}-->

## Formative vs summative assessment

**Formative** assessment takes place **during** teaching and learning. Its purpose is to help both instructors and learners to become aware of what the focus should be. Its purpose is to **improve the quality of students' learning**, not to provide evidence for evaluating or grading students.

**Summative** assessment takes place **at the end** of teaching and learning. Its purpose is to evaluate learners' learning by comparing it against a standard or benchmark, and provide evidence for evaluating or grading students.

> <comment-title>At the end of teaching</comment-title>
>
> For summative assessment, 'the end' does not necessarily mean the end of an entire course or module of study.
> Summative assessments may be distributed throughout a course, after a particular unit (or collection of topics) has been taught, but graded any how.
{: .comment}

What are the methods for summative assessment? You do remember some of the methods used in summative assessment when you are a student or from being a teacher:
- Questionnaires
- Interviews
- Observations
- Testing: specific test created by the teacher or establishment made to include all points of a unit or specific information taught in a given time frame
- Projects: a culminating project that synthesizes knowledge

Summative assessment should be
- Aligned to LOs (levels of complexity, what you taught, the level you taught it)
- Objective
- Feasible

![A cartoon figure showing the main difference between formative and summative assessment, when making soup. When the chef tastes the soup while preparing, it's formative assessment. When the guests taste the soup after its done, it's summative asssessment.](images/FormativeSummativeAssessment.png "Allegorie to explain formative and summative assessment. When the chef tastes the soup and notices that there is not enough salt or spices, they can still add more. When the guests taste the soup, what's done is done (the chef can't change anything in the recipe for these guests). But they can still change the recipe for the future guests (future students). In formative assessment, the instructor can direct the learning on the fly, correct the direction, address mistakes/misconceptions/missing knowledged. In summative assessment, the learners have passed a test and they can't go back in time to do it better. The mark was decided. A summative assessment can indicate how the teaching was (if the instructor realises that the whole class of learners did poorly in the exam, this indicates that the teaching may be a cause), but in the best case scenario she can improve her course for the next cohort of learners (not for those who had finished the course/semester/year)")

In training, the scope here, the focus is on formative assessment (collect information / feedback on how learning is progressing). We will from now on only talk about formative assessment.

Formative assessment takes place **during** teaching and learning. Its purpose is to help both instructors and learners to become aware of what the focus should be. Its purpose is to **improve the quality of student learning**, _not_ to provide evidence for evaluating or grading students.

Formative assessment collect feedback (information) from learners' on their learning progress :
- **Goals and objectives**: if learners goals and objectives are aligned to the course's goals and outcomes
- **Prior knowledge**: which knowledge gaps need to be filled before moving on
- **Mental models**: whether their mental models are correct
- **Frequent mistakes**: which types of mistakes need special attention
- **Ability to perform a task**

> <hands-on-title>Collecting formative assessment - {% icon time %} 10 min - Groups of 4</hands-on-title>
>
> 1. Find an example on how to collect information on **learners' mental models**
> 2. Find an example on how to collect information on **learners' goals and objectives**
> 3. Find an example on how to collect information on **learners' frequent mistakes**
>
{: .hands_on}

<!--{% include _includes/quiz.html id="formative_assessment.yaml" %}-->

It should be highlighted that the **best practice** for teaching is to:
- Constantly **monitor** what is happening to students' learning
- Investigate when things do not **proceed as planned** or expected
- Act upon to **improve** learning
- Provide feedback to learners on how to progress

## Collecting data from learners through formative assessment

Formative assessment & feedback can be done in many different ways:
- Providing **diagnostic questionnaires**
- Asking **questions** to learners and getting responses orally
- Asking them to **describe the strategy** they would adopt to solve a problem
- Asking them to **solve a problem** in groups, or individually but in front of the class
- Using **brainstorming** and **discussions**

### Diagnostic questionnaires

#### Questionnaires

**Anonymous questionnaires** allow to have an idea of the level of knowledge of the whole group of learners. 

**Non-anonymous** and personal questionnaires allow to find out if the learner has a necessary pre-required knowledge, in the negative case indicate an appropriate teaching choice to palliate this lack.

#### Instant feedbacks or minute cards

Before each long break, for example lunch or between days, we can have learners to complete **minute cards** to share anonymous feedback. At an in-person workshop, paper sticky notes double as minute cards, with the two different colors used for positive and constructive feedback. At an online workshop, this may be done by making a copy of the [Virtual Minute Card Template](https://docs.google.com/forms/d/1p7iOV5HNvy4POS4g6eottY8RSfKq4kaoKz1-jIFYTMI/template/preview) from the Carpentries on Google Forms. Other tools can work as well, but we do recommend making sure that feedback is private and anonymous. A public-facing sticky note board will receive different (and less useful) feedback.

During long breaks, instructors read through the minute cards and look for patterns. At the start of each half day, the Instructors take a few minutes to address commonly raised issues with the whole class. The non-teaching Instructor can also type answers to the questions in the Etherpad.

Another way to get instant feedback is the **Fist of five**. It is a simple format for asking for agreement or disagreement, or understanding on a topic by a show of hands. Each hand signal has a different meaning:
- Fist: "I do not understand"
- 1 finger – index finger: "I have (major) concerns that need to be heard and resolved immediately."
- 2 Fingers – index finger and middle finger or thumb and index finger: "I have (minor) concerns that we need to resolve now."
- 3 fingers – thumb, index finger and middle finger or index finger, middle finger and ring finger: "I have (minor) concerns but can live with the continuying. My concerns should be resolved soon."
- 4 Fingers – Index finger, middle finger, ring finger and little finger or thumb, index finger, middle finger and ring finger: "I got it."
- 5 fingers (Five) – Showing open hand: "I am enthusiastic and we can continue"

#### Multiple Choice Question (MCQ) to evaluate mental models

Look at this example of a MCQ

- Question: "What is 27+15 ?"
- Possible answers
    - 42
    - 32
    - 312
    - 33
- Correct answer: 42, but each of the other answers provides valuable insights

> <question-title>Multiple Choice Questions and mental models</question-title>
>
> In the previous MCQ, which misconception is associated with
> 1. 32
> 2. 312
> 3. 33
>
> > <solution-title></solution-title>
> > 1. If they answer 32, they are throwing away the carry completely.
> > 2. If they answer 312, they know that they can't just discard the carried '1', but doesn't understand that it's actually a ten and needs to be added into the next column. In other words, they are treating each column of numbers as unconnected to its neighbors.
> > 3. If they answer 33 then they know they have to carry the 1, but are carrying it back into the same column it came from.
> {: .solution}
{: .question}


### Group activities

Many group activities are possible
- Solve problems in groups
- Do exercises in front of the class
- Asking to describe the strategy they would adopt to solve a problem
- Using brainstorming

    Brainstorming should address a specific question; sessions addressing multiple questions are inefficient. The brainstorming technique can be used to build a concept map as a group.

- Shared notes for
    - Collaborative note taking
    - Output of group activities
    - Exercises
    - Opinions, ideas, comments, thoughts
    - Questions, answers

    Tools like Google docs, Etherpad or HackMD could be used

> <hands-on-title>How to integrate the result of formative assessment / feedback in your course? - {% icon time %} 5 min - Silent reflection</hands-on-title>
>
> 1. In this session, how did we assess your learning?
> 2. Choose one of the techniques to collect feedback, and describe in the how you can integrate the result in your lesson, on the fly
>
{: .hands_on}

<!--{% include _includes/quiz.html id="integrate_formative_assessment.yaml" %}-->

## How frequent should be formative feedback

<!--{% include _includes/quiz.html id="frequency_formative_assessment.yaml" %}-->

> <hands-on-title>How frequent should formative feedback be? - {% icon time %} 5 min - Self-reflection</hands-on-title>
>
> - In this session, how many feedback opportunities have you had so far?
>
>   > <solution-title>Get feedback often</solution-title>
>    > In this tutorial so far, there were at least 3 feedback opportunities
>    {: .solution}
> 
> - How frequent do you think formative feedback should be?
>
{: .hands_on}

![A figure with a line plot, showing that the learners' attention peaks at ~7-10' and then drastically drops after 15+ minutes.](images/AttentionCurve.jpg "Attention Curve")

Formative assessment should be frequent: ideally **every 5 min**, at least every 10-15 min. Indeed, the attention span is 10-15 min and attention should be refocused after that.

# Dealing with feedback

Feedback is hard.

> <hands-on-title>How to deal with feedback - discuss experiences  - {% icon time %} 15 min - Groups of 4</hands-on-title>
>
> - Share one or two experiences that you have had in either offering or receiving feedback.
> - Share your strategy in dealing with this feedback. What were the challenges, and what worked?
> - Write it in the shared notes and discuss with the rest of the group.
{: .hands_on}

![A cartoon showing that a single negative feedback among a lot of positive can still be the only one that one remembers eventually.](images/gettingFeedback.jpg "Getting feedback")

Humans focus more on negative feedback than on positive (you are not alone). How to deal with (bad) feedback received from trainees:
* **Breath deeply**
* Try to see the point in the criticism, **learn from it**
* **Don't take it personally**

Trainees feedback should be considered along other forms of quality evidence:
* Review what they have effectively learned (in exams)
* Consider your own experience of teaching
* Discuss with colleagues and friends
* Look at the feedback from past sessions of the same course
* Look at the response rates
* Look at the counter examples (contradictions)
* Look at the repetitive patterns (not at only one single answer)

![A humorous cartoon showing a parent teaching a child on how to provide constructive feedback when playing with other children.](images/givingGoodFeedback.jpg "Learn and teach others to give good feedback")

# Conclusions


topic: teaching
tutorial: galaxy-admin-training
title: Galaxy Admin Training
difficulty: advanced
time_estimation: 60m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: []
content_hash: a3cc2792041ef86e2241d73c5fdb2f49
last_modified: 2026-02-26T21:12:18.331292
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/galaxy-admin-training/tutorial.html

content:
Setting up and running a Galaxy Admin Training is not a very complicated process thanks to a significant amount of work that has been put into making it easy and quick.

![GAT logo is the GTN logo over a space background and text reading galaxy admin traiing.](../../images/gat.png)

This tutorial has multiple audiences who are all adressed within this one tutorial. We encourage everyone involved in hosting a GAT event to read this, so you are aware of all of the moving parts which are required for such an event.

> <agenda-title></agenda-title>
>
> In this tutorial, we will see:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Planning Your Training

First consider the requirements for your training:

- How many days will the training be?
- What topics do you want to cover?
- How many helpers will you need?
- Will the training be online or in person? Teaching online is slower, so you may not be able to cover as many topic as in person; you may also need more helpers.
- Do you have your own infrastructure, or do you need to request infrastructure?

We recommend [checking out an example schedule from the GAT repository](https://github.com/galaxyproject/admin-training/tree/2020-barcelona) and modifying it based on your needs.

## Requesting Infrastructure

If you do not have a cloud available that can, at minimum:

- setup VMs which are accessible from where your students are
- expose ports 20, 80, 443 (optional), 8080 (optional but nice)

Then consider [contacting UseGalaxy.eu](mailto:galaxy@informatik.uni-freiburg.de?subject=Hosting%20a%20GAT) and let us know you'd like to host a GAT event. Tell us the number of students and dates of the event.

# Provisioning the Infrastructure

Skip to the section on setting up below that is appropriate for you

## [EU admins] Setting up VMs

> <hands-on-title>EU: Setting up VMs</hands-on-title>
>
> 1. Open a PR editing the [count variable](https://github.com/usegalaxy-eu/infrastructure/blob/master/gat.tf#L1) and increasing it to the desired number of VMs.
>
> 2. Merge
>
> 3. Once deployed, clone/pull the repository locally and run `./bin/process-training-output.sh`
>
>    ```
>    $ ./bin/process-training-output.sh
>    ubuntu  gat-0.training.galaxyproject.eu ...
>    ubuntu  gat-1.training.galaxyproject.eu ...
>    ```
>
> 4. Place this in the [GAT Machines](http://gxy.io/gatmachines) spreadsheet in the correct tab.
>
{: .hands_on}

## [Non-EU admins] Setting up the VMs

> <hands-on-title>Non-EU: Setting up VMs</hands-on-title>
>
> 1. If you have your own cloud, setup VMs per student. We strongly recommend:
>
>    - Ubuntu 18.04+
>    - 8Gb RAM
>    - 2 VCPUs
>
>    If you're using terraform, you can take inspiration from [UseGalaxy.eu's terraform configuration](https://github.com/usegalaxy-eu/infrastructure/blob/master/gat.tf)
>
> 2. Place the list of usernames and IPs and passwords in the [GAT Machines](http://gxy.io/gatmachines) spreadsheet in the correct tab.
>
{: .hands_on}

## [Everyone] Bootstrapping the VMs

Once your VMs are running, great! Now you'll need to bootstrap the instances and prepare them for student use.

The GAT team maintains some infrastructure to handle the bootstrapping in [the GAT repository](https://github.com/galaxyproject/admin-training/tree/2020-bcc/bootstrap-instances)

We use a more complicated hosts file in the project, as we have multiple pools of VMs across global regions. This is seen in the `workshop_eu` and `workshop_oz` groups, and the corresponding commands in the Makefile. However, this distinction of regions is not necessary. The most simple hosts file looks like:

```
# Your Machines
[workshop_instances]
192.0.2.1
192.0.2.2
192.0.2.3
...

# Some variables for those machines
[workshop_instances:vars]
ansible_host_key_checking = false
ansible_user = ubuntu
ansible_become = true
ansible_ssh_private_key_file = ~/admintraining.key
set_password = true      # Generate a random password
```

> <hands-on-title>Everyone: Setting up VMs</hands-on-title>
>
> 1. `git clone https://github.com/galaxyproject/admin-training/` and change into that repo
>
> 2. `cd bootstrap-instances`
>
> 3. In the `hosts` file, edit the `[workshop_eu]` section to list all of your IPs or DNS entries.
>
>    You can specify `ansible_user=something` if it uses a different username, and `ansible_password=password` for each machine. E.g.
>
>    A range of DNS entries (`gat-0` to `gat-39`)
>
>    ```
>    [workshop_]
>    gat-[0:39].training.galaxyproject.eu
>    ```
>
>    Some IPs
>
>    ```
>    [workshop_eu]
>    192.0.2.1
>    192.0.2.2
>    192.0.2.3
>    192.0.2.4
>    ```
>
>    Some IPs with additional information
>
>    ```
>    [workshop_eu]
>    192.0.2.1 ansible_password=2121
>    192.0.2.2 ansible_password=1212
>    192.0.2.3 ansible_password=4321
>    192.0.2.4 ansible_password=1235
>
>    [workshop_instances:vars]
>    ansible_user = myuser
>    ```
>
>    Remove the `workshop_oz` from the top of the file under `[workshop_instances:children]`
>
> 4. Setup a python virtualenv and activate it.
>
> 5. Install the requirements (listed in `requirements.txt`)
>
> 6. Run `make all`
>
{: .hands_on}

The last command runs the `playbook.yml`, which in turn does a large number of things:

- Bootstraps python on the machine if it isn't available already (Ubuntu stopped including it for some reason.)
- (Generates if needed) and copies an SSH key to all of the machines to make login easier (this is stored in `id_rsa` and `id_rsa.pub` in the same directory as the makefile.)
- This key is set as the ubuntu user's key, and added to their authorized_keys, to permit them to run ansible easily on that machine (if they don't configure the local connection correctly.)
	- (When pulsar is in use) the pulsar machines are provisioned identically to the ones where Galaxy is setup, so the students can login passwordlessly to their pulsar machine.
- Updates packages (slow)
- Installs basic dependencies (emacs, vim, nano, git, etc.)
- Adds the `gat-cli` script to `/usr/bin/gat`
- Optionally sets a password to the machine
	- If you set `set_password=true` in the hosts file, you can set a password on machines.
	- This is useful when your machines only have an SSH key on them, and no password set for students.
- Reboots the machines

# Testing

Once your VMs are ready, you should test ALL of the lessons you intend to teach. Most of the training should be fine, but sometimes changes in e.g. Galaxy versions or the availability of newer versions of some ansible modules will indicate you should test the training as you plan to teach it, and update the training materials where relevant.

# Starting Your Training

We recommend providing a website [similar to our GitHub repository](https://github.com/galaxyproject/admin-training/tree/2020-barcelona) (or using our repository! Ask us for a branch.) with at minimum the following links:

- **Q&A** pointing to a Google document where students can ask questions.
	- In our experience this is an excellent format for discussion: it allows students to ask as many questions as they have, and responses can be given in real-time right below each question.
	- Additionally images and complex formatting is easy
	- Lastly, it's *anonymous* which many students prefer.
- **Chat** pointing to `https://gxy.io/gatchat` (or your preferred channel)
- **VM List** pointing to `https://gxy.io/gatmachines` (or your own spreadsheet)
- The slides and tutorials for your training.

# During Your Training

During planning for BCC2020 we found that monitoring student progress would be extremely difficult, it was not a situation we had encountered before as this was the first remote GAT during the pandemic. So we developed a small utility, the `gat-cli`, which assists in monitoring students' progress.

```
$ gat
Galaxy Admin Training (gat) tool:

  gat status-ansible     [Admin] Check status of ansible training
  gat status-galaxy      [Admin] Check status of ansible-galaxy training
  gat status-cvmfs       [Admin] Check status of cvmfs training
  gat status-pulsar      [Admin] Check status of pulsar training
```

Each of these commands run a couple checks on their local machine. For example the `status-galaxy` command checks:

- Check that the `postgresql` service is running
- Does a db named `galaxy` exist
- Does http://localhost:8080/api/version respond with *some* content
- Check that the `galaxy` service is running
- Check that the `nginx` service is running
- Does https://localhost:443/api/version respond with *some* content

However, the `gat` command is only available on the VMs, so we've written an ansible task (wrapped in the Makefile) which SSHs into every machine, and runs the gat command for you:

```
$ make check-galaxy-eu
gat-19    postgres ✘ ✘ galaxy(http) ✘ SysD-gxy ✘ SysD-nginx ✘ galaxy(ssl) ✘
gat-22    postgres ✔ ✘ galaxy(http) ✘ SysD-gxy ✘ SysD-nginx ✘ galaxy(ssl) ✘
gat-2     postgres ✔ ✔ galaxy(http) ✔ SysD-gxy ✘ SysD-nginx ✘ galaxy(ssl) ✘
gat-15    postgres ✔ ✘ galaxy(http) ✘ SysD-gxy ✘ SysD-nginx ✘ galaxy(ssl) ✘
gat-24    postgres ✔ ✘ galaxy(http) ✘ SysD-gxy ✘ SysD-nginx ✘ galaxy(ssl) ✘
gat-6     postgres ✘ ✘ galaxy(http) ✘ SysD-gxy ✘ SysD-nginx ✘ galaxy(ssl) ✘
```

You can see a checkmark reported for every step completed by students, giving you a nice overview of how many students have completed each step, and if you're ready to move on. Additionally you know precisely which students you should reach out to check in with, if they aren't progressing. The `-eu` again refers to the specific pool of machines, and if you're using a different hosts file, with different group names, you may need to edit the Makefile accordingly.

There are other `make check-` commands for each of the `gat` status commands. Run `make` to list all of them.

# After Your Training

Once your training is concluded, go through the questions students have asked in the Google Doc, and consider contributing them back to the training materials with Tips and Question boxes covering these student questions. Here is [an example pull request](https://github.com/galaxyproject/training-material/pull/1922) when we did this after BCC2020.

# Conclusion


topic: teaching
tutorial: experiences
title: Teaching experiences
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: b4b7ca406bbece36ebe70f7e9f9d878e
last_modified: 2026-02-26T21:12:18.330980
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/experiences/tutorial.html

content:
We all teach in different ways, share your training philosophy and help other instructors find the way that is best for them!

Some instructors have contributed their ideas. You can read their stories below.

# Teaching stories

{% assign stories = site.pages | where:"layout", "training_philosophy" %}
{% assign contributors = site.data['contributors'] %}

{% for p in stories %}
    {% assign username = p.username %}
<div id="{{ username }}">
    <h2>{{ contributors[username].name | default: username }}</h2>
    <div class="hall-of-fame-hero">
        <a href="{{ site.baseurl }}/hall-of-fame/{{ username }}/" class="thumbnail">
            <img src="https://avatars.githubusercontent.com/{{ username }}" alt="{{ username }}'s avatar">
        </a>
        <div class="contact-items">
            {% if contributors[username].github != false %}
                <a title="GitHub" href="https://github.com/{{ username }}">
                    {% icon github %}
                </a>
            {% endif %}
            {% if contributors[username].email %}
                <a title="E-mail" href="mailto:{{ contributors[username].email }}">
                    {% icon email %}
                </a>
            {% endif %}
            {% if contributors[username].gitter %}
                <a title="Gitter" href="https://gitter.im/{{ contributors[username].gitter }}">
                    {% icon gitter %}
                </a>
            {% endif %}
            {% if contributors[username].twitter %}
                <a title="Twitter" href="https://twitter.com/{{ contributors[username].twitter }}">
                    {% icon twitter %}
                </a>
            {% endif %}
            {% if contributors[username].linkedin %}
                <a title="LinkedIn" href="https://www.linkedin.com/in/{{ contributors[username].linkedin }}">
                    {% icon linkedin %}
                </a>
            {% endif %}
            {% if contributors[username].orcid %}
                <a title="ORCID" href="https://orcid.org/{{ contributors[username].orcid }}">
                    {% icon orcid %}
                </a>
            {% endif %}
        </div>
    </div>
    {{ p.content | markdownify }}
</div>
{% endfor %}


topic: teaching
tutorial: learner_participation_engagement
title: Training techniques to enhance learner participation and engagement
difficulty: intermediate
time_estimation: 60m
tools_json: []
requirements_json: []
tags_json: ["elixir", "train-the-trainers"]
content_hash: 74116d5245717039aacedeb59a73bb7f
last_modified: 2026-02-26T21:12:18.335702
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/learner_participation_engagement/tutorial.html

content:
Effective training

<!-- GTN:IGNORE:002 -->

> <comment-title>Resources / Reading recommendation</comment-title>
>
> This tutorial is significantly based on the 3rd session of ELIXIR Train the Trainer curriculum, but also:
> - [Teaching Tech Together {% cite wilson2019teaching %}](https://teachtogether.tech)
> - [How Learning works, {% cite ambrose2010learning %}](https://www.academia.edu/6743627/How_Learning_Works_Seven_Research_Based_Principles_for_Smart_Teaching_Susan_A_Ambrose_Michael_W_Bridges_Michele_DiPietro_Marsha_C_Lovett_Marie_K_Norman_with_a_Foreword_by_Richard_E_Mayer)
> - Understanding how we learn {% cite weinstein2018understanding %}
> - [Instructor Training, The Carpentries](https://carpentries.github.io/instructor-training/08-motivation/index.html)
>
> To get ready to teach this tutorial
> - Read the chapter "In the Classroom" from [Teaching Tech Together](https://teachtogether.tech/en/index.html#s:classroom)
{: .comment}


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Effective training & good trainer

> <hands-on-title>What makes training effective and a good trainer? - {% icon time %} 10 min - Discussion in groups of 2</hands-on-title>
>
> - Recall concrete examples of your past training experience
> - Share with your colleague
> - List 3 keywords for effective training
> - List 3 keywords for a good trainer
>
{: .hands_on}

{% include _includes/quiz.html id="effective_training_trainer.yaml" %}

**Training / teaching is effective** if
- brings learners to Learning Objectives (LO)
- is engaging
- is well designed

A **trainer / teacher is effective** when they
- facilitates learning
- sets clear LOs
- identifies appropriate LEs
- is engaging
- is inspiring
- is empathetic
- is accessible

To help trainers, GOBLET ([Global Organisation for _Bioinformatics_ Learning, Education & Training (GOBLET)](https://www.mygoblet.org/)) developed this skill matrix:

![Skill matrix with 2 rows and 2 columns. Top-left - Communication: Verbal communication skills, Written communication skills, Presentation Skill. Top-right - Planning & management: Session planning, Curriculum planning, Event management. Bottom-left: Expertise & knowledge: Subject area knowledge, User application awareness, Knowledge of training methodology. Bottom-right - Trainee engagement: Flexibility in delivery, Empathy with trainees, Understanding & knowledge of trainees. ](images/goblet_matrix.png "Skills matrix for a good trainer. Source: <a href="https://www.mygoblet.org/sites/default/files/goblet_events/GOBLET-TTT-061113.pdf">Global Organisation for Bioinformatics Learning, Education and Training (GOBLET)</a>")

The idea is that there are **four areas in which a good trainer should be or become skilled**. In many cases, trainers focus on the area of communication and are less aware about their "level" in the other areas.

This matrix is then a **tool for self evaluation**:
- In which area(s) do you feel you would need improvement?
- In which area(s) do you feel you are ok?


> <hands-on-title>Reflect on your skills as a trainer? - {% icon time %} 5 min - Silent reflection</hands-on-title>
>
> - Look at the GOBLET Skill Matrix
> - Reflect on the following questions
>   - Which areas do you need to develop in?
>     - Communication
>     - Planning & management
>     - Trainee engagement
>     - Expertise & knowledge
>   - Which skills do you need to improve in **communication**?
>     - Written communication skills
>     - Presentation skills
>     - Verbal communication skills
>   - Which skills do you need to improve in **planning & management**? 
>     - Session planning
>     - Curriculum planning
>     - Event management
>   - Which skills do you need to improve in **trainee engagement**?
>     - Flexibility in delivery
>     - Empathy with trainees
>     - Understanding & knowledge of trainees
>   - Which skills do you need to improve in **expertise & knowledge**?
>     - Subject area knowledge
>     - User application awareness
>     - Knowledge of training methodology
>
{: .hands_on}

<!--{% include _includes/quiz.html id="trainer_skills.yaml" %}-->

> <details-title>GOBLET</details-title>
>
> [Global Organisation for _Bioinformatics_ Learning, Education & Training (GOBLET)](https://www.mygoblet.org/) aims to coordinate world-wide bioinformatics training activities:
> - to share, not duplicate, effort;
> - to share, not duplicate, cost;
> - to work together in a mutually respectful way towards common solutions and a sustainable future
>
{: .details}

# Effective training practices for effective learning

We are now going to watch a short video (13 min) from Eric Mazur, professor at the Harvard University,

{% include _includes/youtube.html id="Z9orbxoRofI" title="Effective training practices" %}

In this video, Eric Mazur discusses some of these teaching practices and a **paradigm change: from the traditional lecture to active learning-based approaches**. Can you recognise the main features of this change of paradigm?


> <hands-on-title>Reflect on Mazur's interview - {% icon time %} 5 min - Silent reflection</hands-on-title>
>
> - Write something that really impressed you in Mazur’s interview
>
{: .hands_on}


<!--{% include _includes/quiz.html id="mazur_interview.yaml" %}-->

From the Eric Mazur's interview but also as Mitchell Waldrop pointed out in {% cite waldrop2015we %}, learners who **actively interact with the material, the teacher, and other learners**:
- will **learn better and more**
- will **better remember** what they learn
- will be **more able to apply** their knowledge to different fields

In other words, for learning to actually occur, instructors should select **teaching practices (learning experiences)** that **promote active, interactive and collaborative learning**, what Mazur expresses as **"learning by doing"**.

Let's discuss a few more insights about active learning and stress the main differences between the most used technique worldwide (Lecturing) and the active learning approach. You will see that we are talking about an actual change of paradigm.

## Active, interactive and collaborative learning

From {% cite felder2009active %}, **Active Learning** (also called "Guide on the side") is anything course-related that all students in a class session are called upon to do other than simply watching, listening and taking notes (aka typical teachning practice or "Sage on stage" approach):

Typical teaching practice: lecturing | Active learning
--- | ---
Teacher-centered | Student-centered
Teaching by telling | Teaching by questioning
Teacher speaks and students listen | Collaborative learning
Few interactions student/teacher and student/student | High interactivity (student/teacher, and student/student)
Few questions | Discussion with teacher follows, not precedes
The same few students engaged | All students engaged

Do you recognise the paradigm change proposed by Eric Mazur?

With the help of Mazur, you hopefully started believing that active, interactive and collaborative learning is worth considering and deserves you give it a go in one of your future training courses.Should this be the case, you may have questions like:
- how do I choose the active learning techniques for my lessons?
- what "active learning" means in terms of "teaching / training practices"?
- which is the most effective technique I can incorporate in my teaching?
- is there any "ideal" teaching technique I should absolutely learn?

There is **not "the" ideal teaching technique nor the "most effective" teaching technique.**

You may remember from the Nichols' steps of curriculum design, that it is **essential that you align learning experiences to the learning outcomes** of your course. In other words, for each LO, you should identify the learning experience(s) that will best support the achievement of the LO.

More generally, to decide how to teach and choose the **most appropriate learning experience(s)**, you may use **three criteria**:
- What is your **purpose / goal** (is it to inspire learners? is it to ensure they will remember a concept?)
- The **Bloom's level** at which you intend to teach (a specific topic; this may change from topic to topic)
- The **learning outcome(s)** the learning experience is designed for

   For example, a lecture is not suitable to teach learners implement an algorithm. It may show how to do it, so that they may be able to describe how to do it, but if you want them to be able to do it, you will have to choose a learning experience where learners will have the chance to practice algorithm implementation.

Inspired by Eric Mazur and the idea of active learning, {% cite via2020course %} created this useful table to reinforce the concepts discussed in previously and provide with criteria to choose the **most suitable learning experience(s) for the Bloom's level, TG(s) and LO(s)**:

| Learning experience | Highest Bloom's levels supported | Example TG(s) _"This LE will allow me to..."_                                                                                                                       | Example LO(s) _"Learners will be able to..."_                                                                                 |
| ------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Lecture, webinar    | Remember, Comprehend             | Inspire learners, ignite learners' enthusiasm, clarify/explain a concept, provide an overview, give context, summarise content                                    | _list_ the key points of the lecture/webinar, _summarise_ take home message(s)                                              |
| Exercise, practical | Apply, Analyse                   | Help learners digest course materials, solve typical problems, apply knowledge, show how to do things with appropriate guidance, give an idea of how a tool works | _follow_ a set of instructions or protocol, _calculate_ a set of results or outcomes from a given protocol                  |
| Flipped class       | Apply, Analyse                   | Teach learners how to formulate questions, help learners to memorise new information & concepts, or analyse & understand course materials                         | _summarise_ the content material, *ask* appropriate questions                                                             |
| Peer instruction    | Synthesise, Evaluate             | Prepare learners to defend an argument, give learners opportunities to explain things, thereby helping to develop critical thinking & awareness                   | _explain_ how they solved an exercise, _evaluate_ others' choices/decisions, _diagnose_ errors in the exercise-solving task |
| Group discussion    | Synthesise, Evaluate             | Give learners opportunities to practice questioning, develop new ideas & critical thinking                                                                        | _communicate_ their own ideas, _defend_ their own opinions                                                                  |
| Group work          | Synthesise, Evaluate             | Promote collaborative work & peer instruction, provide opportunities for giving/receiving feedback, & digesting course materials                                  | _provide_ feedback on their peers' work, _share_ ideas, _explain_ the advantages of team-work                               |
| Problem-solving     | Synthesise, Evaluate             | Promote learner abilities to identify & evaluate solutions, develop new ideas, make decisions, evaluate decision effectiveness, troubleshoot                      | _diagnose_ faulty reasoning or an underperforming result, _correct_ errors                                                  |

> <comment-title>Learning experience vs Teaching practice</comment-title>
>
> In the table, Learning experience (from the learner point of view) is the Teaching practice (from the teacher point of view).
>
{: .comment}

You should be kind of familiar with some of the learning experiences listed in the table, e.g., lecture/webinar, group discussion, work in groups, problem solving. Some of you might not be familiar with the techniques of "peer instruction" and "flipped classroom".

These two techniques can be implemented in many different ways and incorporated into your course in a "light way" or very extensively. This will depend on your familiarity with the technique. At the beginning, our suggestion is that you try with something "minima", test it, see how you feel and how learners receive it, and then you increase the amount of learning experiences involving peer instruction and flipped classroom.

## Peer instruction

Peer instruction is a 1-to-1 instruction done in a scalable way by **interleaving formative assessment with learner discussion**.

> <details-title>Peer-to-peer or peer-learning: what is it? - {% icon time %} 2-min video</details-title>
>
> <iframe width="560" height="315" src="https://www.youtube.com/embed/MYa4hgcMRNc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
>
{: .details}

In its "traditional" form, the teacher
1. Gives a brief introduction to the topic.
2. Gives learners a multiple choice question (MCQ) that probes for their misconceptions
3. Has all the learners vote on their answers to the MCQ
    2. If the learners all have the **right** answer, move on
    3. If they all have the same wrong answer, **address** that specific misconception
    4. If they have a mix of right and wrong answers, give them several minutes to argue with each other in groups of 2–4, then vote again

You may also be creative and implement this technique in your own way.

> <details-title>Concept Tests at Avanti's Learning Centre in Kanpur - {% icon time %} 2-min video</details-title>
>
> <iframe width="560" height="315" src="https://www.youtube.com/embed/2LbuoxAy56o" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
>
{: .details}

> <details-title>Does that work?</details-title>
>
> It has been studied extensively in a wide variety of contexts, including programming (references??). Learners value peer instruction even at first contact.
>
> Group discussion significantly improves learners' understanding. It uncovers gaps in their reasoning and forces them to clarify their thinking. Re-polling the class then lets the teacher know if they can move on or if further explanation is necessary. A final round of additional explanation after the correct answer is presented gives learners one more chance to solidify their understanding.
>
> Peer discussion actually does enhance understanding. Even when none of the learners in a discussion group originally knew the correct answer. As long as there is a diversity of opinion within the group, their misconceptions cancel out.
>
{: .details}

## Flipped classroom

In its standard approach,

1. Before the course, out of class, students prepare to participate in class activities
2. During the course in the class, students practice applying key concepts with feedback
3. After the course, out of class, students check their understanding and extend their learning

Flipping class requires a **lot of preparation**. The teacher need to explain the method to learners and make an "agreement" with them: they have to commit to work on the materials you will give them prior the flipped class and you will commit to give them "manageable" materials.

Don't start by flipping an entire course. Start small. Test. Add up.

# Other teaching practices for active learning

- **Brief question-and-answer sessions**: a classic to distribute attention

    > <details-title>Recommendation for brief question-and-answer sessions</details-title>
    >
    > When doing, the teachers should **make sure that everybody has a voice and that attention is fairly distributed**
    >
    > Tools like sticky notes could be used. Each learners write their name on a sticky note and put it on their laptop. Each time the teacher calls on them or answers one of their questions, they take their sticky note down. Once all the sticky notes are down, everyone puts theirs up again. It makes it easy for the teacher to see who they haven't spoken with recently and to avoid unconscious bias and interacting preferentially with their most extroverted learners
    >
    > Without a check like this, it is too easy to create a feedback loop in which<>
    > - Extroverts get more attention, leads to them doing better and leads to them getting more attention
    > - Quieter, less confident, or marginalized learners are left behind
    >
    > Such tools shows learners that attention is being distributed fairly. When they _are_ called on, they won't feel like they're being picked on
    >
    {: .details}

- **Think, pair, share**: a lightweight technique that helps people improve ideas through discussion with their peers

    > <details-title>Think, pair, share</details-title>
    >
    > In this form of peer instruction, Teacher proposes an activity, a question, a problem and then follow 3 steps>
    >
    > 1. **THINK**: Each person starts by thinking individually about it and jotting down a few notes
    > 2. **PAIR**: students explain their ideas to each other in pairs, merging them or selecting the most promising
    > 3. **SHARE**: a few pairs present their ideas to the whole group
    >
    > For example, a teacher asks a question (a problem to solve)
    > 1. Students **THINKS** singularly. The teacher can check results (technology can be used for that)
    > 2. Students talks in **PAIRS** and find an agreement. The teacher can check results
    > 3.  Students **SHARE** their answers under the teacher's guide
    >
    > Why does it work?
    > It forces people to externalize their cognition and gives a chance to spot and resolve gaps or contradictions in their ideas _before_ exposing them to a larger group
    {: .details}

- **Take notes together / shared notes**

    > <details-title>Shared notes</details-title>
    >
    > This form of real-time elaboration forces to organize and reflect on material as it's coming in, which in turn increases the likelihood of transfer it to long-term memory
    >
    > It also allows people to compare what they think they're hearing with what other people are hearing, and helps to fill in gaps and correct misconceptions right away.
    >
    > It gives the more advanced learners in the class something useful to do. Rather than getting bored and doing something else during class, they take the lead in recording what's being said. They are kept engaged.
    >
    > It allows less advanced learners to focus more of their attention on new material.
    >
    > Finally, it provides the notes that are usually more helpful *to learners* than those the teacher would prepare in advance. Learners are more likely to write down what they actually found new rather than what the teacher predicted would be new.
    {: .details}

- **Work in groups**, e.g. with projects

- **Pair programming**

    > <details-title>Pair programming</details-title>
    >
    > <iframe width="560" height="315" src="https://www.youtube.com/embed/vgkahOzFH2Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
    >
    > Used for Software development, two programmers work together on one computer:
    > - One person (the driver) does the typing
    > - The other (the navigator) offers comments and suggestions
    > The two switch roles several times per hour. This practice is found effective in professional work
    >
    > For learning, 2 learners work together on 1 computer. They can help each other during practical exercises, clarify each other’s misconceptions when solution is presented and also discuss common interests during breaks.
    >
    > It is a good way to teach. It increases success rate in introductory courses, leads better software, higher learner confidence in their solutions and more benefits for learners from underrepresented groups
    {: .details}

- **Discussions / debate** with all learners (as for brief Q&A session, teachers should make sure that everybody has a voice)

- **Brainstorming** with all learners (as for brief Q&A session, teachers should make sure that everybody has a voice)

- **Hands-on activities**, e.g. running tools on their own, commands, analysis

    Galaxy is a great tool for that. That is why tutorials from the GTN are developed around that ideas.

    > <details-title>Recommendation for brief question-and-answer sessions</details-title>
    >
    > During training session, teachers could use sticky notes as status flags. They Give each learner two sticky notes of different colors, such as orange and green.
    >
    > If someone has completed an exercise and wants it checked, they put the green sticky note on their laptop. If they run into a problem and need help, they put up the orange one
    >
    > It is much better than having people raise their hands. It is more discreet and so students are more likely to actually do it. They keep working while their flag is raised rather than trying to type one-handed
    > And teacher can quickly see from the front of the room what state the class is in.
    >
    > It is particularly helpful when people in mixed-ability classes are working through material at their own speed
    >
    {: .details}

- Have learners **make predictions**

    Research has shown that people learn more from demonstrations if they are asked to predict what's going to happen. It fits naturally into live coding / analysis: after adding or changing a few lines of a program, the teacher asks the class what is going to happen when it runs.

    If the example is even moderately complex, prediction can serve as a motivating question for a round of peer instruction.

- **Build an artefact**, e.g. concept map

<!--{% include _includes/quiz.html id="active_learning_strategies.yaml" %}-->

> <hands-on-title>Which strategies for active learning training did you use? - {% icon time %} 5 min - Silent reflection</hands-on-title>
>
> Which strategies for active learning training did you use? (Add +)
> - Peer instruction 
> - Flipped-classroom 
> - Brief question-and-answer sessions
> - Think, pair, share 
> - Take notes together / shared notes 
> - Work in groups
> - Pair programming 
> - Discussions
> - Hands-on activities
> - Have learners make predictions / presentations
> - Build an artefact (e.g. concept map) 
>
{: .hands_on}

# Practices and behaviors to improve delivery and learning

About teaching practices / learning experiences, we learned to
1. **Select learning experiences according to the criteria discussed earlier**
    - reflect on your TG(s)
    - write LOs AND identify the corresponding Bloom's level
2. **Select learning experiences allowing learners to achieve LOs**
3. **Always remember that learning occurs BY DOING**
4. **Be aware of other practices and behaviours supporting:**
    - interactivity
    - a positive and engaging learning environment
    - active and collaborative learning
    - stimulating lessons
    - frequent feedback

Other teaching practices that can improve content delivery. Specially supporting
- **interactivity**
- a **positive and engaging learning environment**
- **active and collaborative** learning
- **stimulating** lessons
- **frequent feedback**

Here are some examples of practices to achieve goals above. None of the practices described here are essential

- **Start with introductions**
    - **Begin by introducing yourself**: If you're an expert, tell them a bit about how you got to where you are. If you're only two steps ahead of them, emphasize what you and they have in common. Whatever you say, the goals are to make yourself more approachable and encourage their belief that they can succeed.

    - **Introduction of learners to each other**
        - In a class of a dozen, it can be done verbally
        - In a larger class or if they are strangers to one another, you can have them each write a line or two about themselves in the shared notes or split into small groups or breakout rooms.

- **Pay attention on how you set up the learning environment** (both in presence and online)

    - **Learners environment**: Send setup instructions per email before and have everyone run some tests at the beginning to check
    - **Yours**: Same as learners and food / drinks

    > <tip-title>Check our dedicated tutorials to set up Galaxy for teaching</tip-title>
    >
    > - [Organizing a workshop]({% link topics/teaching/tutorials/organize-workshop/tutorial.md %})
    > - [Running a workshop as instructor]({% link topics/teaching/tutorials/running-workshop/tutorial.md %})
    > - [Set up a Galaxy for Training]({% link topics/teaching/tutorials/setup-galaxy-for-training/tutorial.md %})
    > - [UseGalaxy.eu's Training Infrastructure as a Service]({% link topics/teaching/tutorials/setup-tiaas-for-training/tutorial.md %})
    >
    {: .tip }

- **Introduce blended multimedia materials to create engaging activities**
- **Collect instant feedback** (see [dedicated tutorial]({% link topics/teaching/tutorials/assessment/tutorial.md %}))
- **Let learners do recaps**: Organise recap sessions at the end of lessons or at the beginning of the following ones where learners are actively involved. You may ask them to do the recaps.

- **Repeat each questions** aloud back to them before answering  to ensure you and all the other learners understood the question and gives you a a chance to redirect the question to something you're more comfortable answering

- **Introduce physical exercises, short, relaxing breaks**: In day-long training courses, you may introduce very short sessions of stretching (or even one moment meditation and then doing 1 minute meditation together). You might feel ridiculous, but you have no idea of how learners appreciate these types of activities and how these create a relaxed climate in the class.

- **Introduce challenges or games**: Gamification can be a very powerful and engaging technique.   Before using gamification, it is though better to learn how to do it.

- **Never teach alone. Teach together**. Whenever possible: With two or more trainers in the classroom, the class can be observed from different angles, it is easier to detect learners who are not comfortable or struggling or lagging behind. It is possible to provide one-to-one support without stopping the lesson flow. It is also important that trainers teaching together provide feedback to each other.

    > <details-title>Different models to teach together or co-teaching</details-title>
    >
    > - **Team teaching**: Both teachers deliver a single stream of content in tandem, taking turns like musicians taking solos.
    > - **Teach and assist**: Teacher A teaches while Teacher B moves around the classroom to help struggling learners.
    > - **Alternative teaching**: Teacher A provides a small set of learners with more intensive or specialized instruction while Teacher B delivers a general lesson to the main group.
    > - **Teach and observe**: Teacher A teaches while Teacher B observes the learners, collecting data on their understanding to help plan future lessons.
    > - **Parallel teaching**: The class is divided in two and the teachers present the same material simultaneously to each.
    > - **Station teaching**: The learners are divided into small groups that rotate from one station or activity to the next while teachers supervise where needed.
    >
    {: .details }

    > <tip-title>Recommendations for co-teaching</tip-title>
    >
    > If you and a partner are co-teaching:
    > - Take 2–3 minutes before the start of each class to confirm who’s teaching what. If you have time, try drawing or reviewing a concept map together.
    > - Use that time to work out a couple of hand signals as well. "You're going too fast", "speak up", "that learner needs help", and, "It’s time for a bathroom break" are all useful.
    > - Each person should teach for at least 10–15 minutes at a stretch, since learners will be distracted by more frequent switch-ups.
    > - The person who isn't teaching shouldn't interrupt, offer corrections or elaborations, or do anything else to distract from what the person teaching is doing or saying. The one exception is to ask leading questions if the learners seem lethargic or  unsure of themselves.
    > - Each person should take a couple of minutes before they start teaching to see what their partner is going to teach after they’re done, and then *not* present any of that material.
    > - The person who isn't teaching should stay engaged with the class, not catch up on their email. Monitoring the shared notes, keeping an eye on the learners to see who’s struggling, jotting down some feedback to give your teaching partner at the next break—anything that contributes to the lesson is better than anything that doesn’t.
    > - Take a few minutes when the class is over to congratulate or commiserate with each other
    {: .tip }

- **Assess prior knowledge and mental models** to tailor the lesson on learners' actual needs, to address misconceptions, to learn about your learners (see [dedicated tutorial]({% link topics/teaching/tutorials/assessment/tutorial.md %})). The more you know about your learners before you start teaching, the more you will be able to help them.

- **Learn learners names and use them**

- **Avoid homework in All-day formats**: Learners who have spent an entire day programming will be tired. If you give them homework to do after hours, they'll start the next day tired as well.

Each of the techniques presented here will make your classes better, but **you shouldn't try to adopt them all at once**. The reason is that every new practice increases *your* cognitive load as well as your learners', since you are all now trying to learn a new way to learn as well as the lesson's subject matter. If you are working with a group repeatedly, you can introduce one new technique every few lessons; if you only have them for a one-day workshop, it's best to pick just one method they haven't seen before and get them comfortable with that.

# Conclusion


<!--{% include _includes/quiz.html id="active_learning_strategies_bloom_levels.yaml" %}-->

> <question-title>What is the highest Bloom’s level supported by each teaching technique? - {% icon time %} 5 min - Silent reflection</question-title>
>
> 1. What is the highest Bloom's level supported by **Peer instruction**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Peer instruction**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Flipped-classroom**? 
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Brief question-and-answer sessions**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Think, pair, share**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Take notes together / shared notes**? 
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Work in groups**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Pair programming**? 
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Discussions**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Hands-on activities**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Have learners make predictions / presentations**?
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
> 1. What is the highest Bloom's level supported by **Build an artefact (e.g. concept map)**? 
>    - Remember
>    - Understand 
>    - Apply
>    - Analyze
>    - Evaluate 
>    - Create
>
>
{: .question}


topic: teaching
tutorial: gallantries-coursebuilder
title: Course Builder
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["cyoa"]
content_hash: 0b3d11103021aa6ffcf73e53eb2d48e4
last_modified: 2026-02-26T21:12:18.332372
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/gallantries-coursebuilder/tutorial.html

content:
As part of the [Gallantries Project](https://gallantries.github.io) we developed a small "Course Builder" which can help you build a schedule of pre-existing videos from the GTN Video library to use with your courses.

> <agenda-title></agenda-title>
>
> In this tutorial, we will see:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Try the Course Builder

You can try the [course builder](https://gallantries.github.io/course-builder/) online. It works as follows:

1. The "Welcome!" Tab

   Pick some modules from the left hand menu by clicking on them. They will turn blue when added to your event.

   > <hands-on-title>Add some modules</hands-on-title>
   > Add the following modules:
   >
   > Basics:
   > 1. Setup
   > 1. Code of Conduct
   > 1. Feedback
   > Sessions:
   > 1. Quality Control
   > 1. Mapping
   > Galaxy Intro:
   > 1. A Very Short Introduction to Galaxy
   > Transcriptomics:
   > 1. Introduction to Transcriptomics
   > 1. Reference based RNA-Seq data analysis
   {: .hands_on}

2. The "Session" Tab

   This tab allows you to organise your modules into sessions. Do you want to split up the materials by day? Or morning/afternoon? Or by week if you're running a longer course?
   
   > <hands-on-title>Create Sessions and Organise Content</hands-on-title>
   > 1. Create a new section
   >
   >    name: Galaxy Intro
   >    description: "Morning Session"
   >
   > 1. Create a new section
   >
   >    name: RNA-Seq
   >    description: "Afternoon Session"
   >
   > 1. Add Very Short Introduction to Galaxy to the first session by dragging and dropping it.
   > 1. Add RNA Seq modules to the second session
   {: .hands_on}

3. The "Configure Event" Tab

   This tab lets you add metadata to the event, it builds on the data in the video-library so you may not find your name in the list.
   
   > <hands-on-title>Update event metadata</hands-on-title>
   > 1. Change the event title and description
   > 1. Change the event dates
   > 1. Select some trainers
   > 1. Select some contacts (can be randomly, or yourself if you are present)
   > 1. Select some affiliated institutions
   {: .hands_on}

3. The "Export" Tab

   If you've made it this far you're ready to contribute this file to the Video Library which will provide a workshop webpage for you!

   > <hands-on-title>Create the event webpage</hands-on-title>
   > 1. Fork the [video-library repo](https://github.com/gallantries/video-library)
   > 1. Add your file to `docs/events/<your-event-identifier>/index.md`
   > 1. Create a pull request
   >    - If any instructors/institutes/organisers were missing from your
   >      selections, then write the missing information in the Pull Request
   >      and the maintainers will help you add it in the correct place.
   {: .hands_on}


topic: teaching
tutorial: setup-tiaas-for-training
title: Training Infrastructure as a Service
difficulty: intermediate
time_estimation: 10m
tools_json: []
requirements_json: []
tags_json: ["cyoa"]
content_hash: c6d4d8433a2c39dbf07d6dedf42f6ec0
last_modified: 2026-02-26T21:12:18.343557
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/setup-tiaas-for-training/tutorial.html

content:
UseGalaxy.eu has developed Training Infrastructure as a Service (TIaaS for short) which allows you to use Galaxy with a private queue for your training event. Your trainees' jobs won't wait in the main queue, and can be processed much more quickly than they might be otherwise. This can provide the experience of a local, private Galaxy combined with a public Galaxy that you are not responsible for maintaining. Additionally if something goes wrong, you can conveniently blame the Galaxy admins, rather than feeling the stress of debugging and fixing your private Galaxy.

![TIaaS Logo](../../images/tiaas-logo.png){: width="500px"}

UseGalaxy.eu has developed Training Infrastructure as a Service (TIaaS for short) which allows you to use Galaxy with a private queue for your training event. Your trainees' jobs won't wait in the main queue, and can be processed much more quickly than they might be otherwise. This can provide the experience of a local, private Galaxy combined with a public Galaxy that you are not responsible for maintaining. Additionally if something goes wrong, you can conveniently blame the Galaxy admins, rather than feeling the stress of debugging and fixing your private Galaxy.

> <agenda-title></agenda-title>
>
> In this tutorial, we will see:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Where are you

{% include _includes/cyoa-choices.html option1="Europe, the Middle East, and Africa (EMEA)" option2="Asia/Pacific (APAC)" option3="the Americas (AMER)" text="Galaxy has servers across the world, but it's best to choose one close to you. Here we customise this tutorial based on where you are, so, pick the location closest to where you'll be running your training." default="Europe, the Middle East, and Africa (EMEA)" %}

# Identify if TIaaS is Appropriate For Your Training

First consider the requirements for your training to see if TIaaS is a good fit for you:

- Do you need really special tools that are not already available on <span class="Europe-the-Middle-East-and-Africa-EMEA"><a href="https://usegalaxy.eu">UseGalaxy.eu</a></span><span class="AsiaPacific-APAC"><a href="https://usegalaxy.org.au">UseGalaxy.org.au</a></span><span class="the-Americas-AMER"><a href="https://usegalaxy.org">UseGalaxy.org</a></span>?
  - The UseGalaxy.* servers support most training workflows.
  - But we only install publicly available and open tools: <span class="Europe-the-Middle-East-and-Africa-EMEA"><a href="https://github.com/usegalaxy-eu/usegalaxy-eu-tools">EU Tool Repository</a></span><span class="AsiaPacific-APAC"><a href="https://github.com/usegalaxy-au/usegalaxy-au-tools">Australian Tool Repository</a></span><span class="the-Americas-AMER"><a href="https://github.com/galaxyproject/usegalaxy-tools/">American Tool Repository</a></span>
- Do you need extra guarantees that the server will be online?
  - The big UseGalaxy.* servers have good uptime, for services run by academic groups, but it cannot make promises regarding availability.
  - Additionally this server can experience occasional slow downs due to usage by other groups and users. If you need more guarantees, please find an alternative.

# How TIaaS Works

<p class="Europe-the-Middle-East-and-Africa-EMEA">
We have several groups of virtual machines (VMs) attached to UseGalaxy.eu that run user jobs. For trainings we attach a new group of VMs that is specially labelled for that training. When normal users run tools on our server, these jobs are instructed to avoid the training pools by default.
</p>
<p class="AsiaPacific-APAC">
We have a pool of virtual machines (VMs) dedicated to training, that run training specific jobs only.
</p>
<p class="the-Americas-AMER">
Jobs submitted while in a TIaaS group have an artificially capped memory limited, and walltime. By lowering these parameters we can be sure the jobs will execute more quickly, but this only works for training data. If you run a training with "real data" it may require too much memory or computation time, and those jobs will be killed.
</p>

When your users join a training, using a special URL provided to you, they then are placed in a special training group. Their jobs will then preferentially run on a training machine, and, in the event there is no more capacity, they will run on the main queue. If a spot on a training VM opens up first, they will run there rather than continuing to wait in the main queue.

# The Application Process

> <hands-on-title>Apply for TIaaS Training</hands-on-title>
>
> 1. <span class="Europe-the-Middle-East-and-Africa-EMEA"><a href="https://usegalaxy.eu/tiaas/new/">Fill out the form</a></span><span class="AsiaPacific-APAC"><a href="https://usegalaxy.org.au/tiaas/new/">Fill out the form</a></span><span class="the-Americas-AMER"><a href="https://usegalaxy.org/tiaas/new/">Fill out the form</a></span>
>
> 2. The admin team will review the request and get in touch with you as needed to identify the compute resources you need for your training
>
> 3. The admin team will inform you of the URL you should provide to your participants during the training. They can open this URL and they will be added to the special training group
>
{: .hands_on}

# The Student's Process

We have a "test" TIaaS training setup on UseGalaxy.eu, which will never have compute resources associated with it, but you can use it to test the process of signing up, if you wish:

> <hands-on-title>Join a TIaaS Training</hands-on-title>
>
> 1. Click on this link: [https://usegalaxy.eu/join-training/test](https://usegalaxy.eu/join-training/test)
>
> 2. In the background you are added to a group in Galaxy. If you were to run jobs they would be tagged with this special queue.
>
{: .hands_on}

That's it, you're now in the "test" TIaaS group. It's really that easy for students.

# The Status Dashboard

Once your students are registered, and you're running your training, a common question instructors ask are "Are you all done?" and students are often not as vocal or repsonsive as we would like. So the TIaaS service now has a dashboard you can view which shows you the queue status for everyone in your training group. This works by finding all of the members of that training group, and checking all jobs that were created in the last 3 hours. For all of those jobs, this is displayed as a simple dashboard with the status of these jobs:

It shows:

- Overview of queue (how many are in state new/queued/ok/error)
- Overview split by tools (how many people are done running Fastqc?)
- A full listing of the queue

![TIaaS Queue Status](../../images/tiaas-status.png "The TIaaS Status dashboard gives you an overview of all jobs states (are they ok or not), as well as a breakdown by tool. This is useful for finding out if everyone is finished running FastQC this morning and if they mostly worked OK. Finally it gives you a detailed breakdown, shown in the order they were submitted. This can give you a more detailed feeling for how the students are progressing through the tutorial.")


> <hands-on-title>View the dashboard</hands-on-title>
>
> 1. The status dashboard is just the same URL as to join the group, with `/status` at the end: [https://usegalaxy.eu/join-training/test/status](https://usegalaxy.eu/join-training/test/status)
>
{: .hands_on}

# Before Your Training

**At least 2 weeks** before your training:

1. Run through your planned trainings on the UseGalaxy.eu server to ensure everything is available including data libraries and tools.
2. If anything is missing, contact the admins and they will work to resolve the missing training material resources.
3. Follow the rest of the training material's [guide to preparing a workshop]({% link topics/teaching/tutorials/organize-workshop/tutorial.md %})

# During Your Training

1. Check out your status dashboard
2. Watch for problems and contact the admins if
   - Jobs are spending abnormally long in the queue
   - Jobs are failing unexpectedly
2. Remind your participants to submit bug reports if they experience any tool errors
3. Follow the rest of the training material's [guide to running a workshop]({% link topics/teaching/tutorials/running-workshop/tutorial.md %})


topic: teaching
tutorial: gallantries-async
title: Asynchronous training
difficulty: intermediate
time_estimation: 1h
tools_json: []
requirements_json: []
tags_json: []
content_hash: ab4d614007623cae194fbde2b7849059
last_modified: 2026-02-26T21:12:18.332231
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/teaching/tutorials/gallantries-async/tutorial.html

content:
**asynchronous training events**, i.e. MOOC style courses pair pre-recorded and written lesson materials with online participants following along at their own pace and convenience.

With this model, we aim to bring training events to the trainees while **eliminate** environmental impact of instructor *and* participant travel.

# What is an asynchronous training event?

*To illustrate a typical Gallantries/Smörgåsbord event and the different roles (1 person can have several roles), we have decided to share a story with several fictional characters*

**Taylor** is a training coordinator of an international research consortium. They recently surveyed researchers in the consortium on their need in HTS data analysis. The results were overwhelming: over 60 scientists at the several locations would like to learn how to analyze their own RNA-seq data. To fulfill this demand, Taylor decided to organize a Smörgåsbord style event: a 3-days online workshop. Taylor is the **global organizer** of this event, in charge of finding the date, contact and coordinate with the teachers, advertise the event, find the instructors, etc. After the event, Taylor will aggregate all feedback from participants, helpers and hosts and share them.

Taylor recruited 2 **helpers** for each day, from amongst the participating incitations. One of the helpers, **Casey**,  has some previous experience with RNA-seq data analysis and Galaxy. Before the workshop, they went through the training material and tested it. They will help participants during the workshop when they are stuck or get different results than the instructors. They will also give direct feedback to the instructor about the pace, any possible local issues etc.

Taylor also recruited 4 **instructors**, **Farah** among them. Farah is a trained bioinformatician and experienced instructor, regularly giving training to scientists on HTS data analysis. She will give the introduction and the first steps of RNA-seq data analysis lessons. Farah is located in Germany and teaches from a room at her institute in front of her computer. **2 weeks** before an event she needs to record her lesson as she would normally teach it during a live training, and provide that video to Taylor. Taylor will generate captions (e.g. YouTube's auto-captions) to Farah who must review and manually correct any captioning mistakes, before returning those to Taylor.

**Alex** is a PhD student in molecular biology based in Greece. They would like to learn about RNA-seq data analysis to be able to analyze the data they generated. They heard about the workshop and wish to join online. As a **participant**, Alex will actively participate in the workshop by running their first RNA-seq data analysis given the instructor's instructions, ask for help from helpers when stuck, will raise their questions on the participant chat, and will give feedback using sticky notes and the dedicated feedback form.

# Cost of an event

With the asynchronous training model, the cost of organizing and participating in the workshop are essentially eliminated: the instructors do not need to travel to the venue, participants can learn wherever is convenient for them, no rooms nor coffee must be organised.

# Event Timeline

Here is an example event timeline we provided for a past Smörgåsbord event:

Date        | Time Left | What's happening
---         | ---       | ---
31 January  | 6 weeks   | You tell us which sessions you want to record
4 February  | 5 weeks   | The final schedule is published
17 February | 4 weeks   | GTN CoFest! Here we will help you create workflows + workflow tests for your tutorials.
28 February | 2 weeks   | **Training Videos Due**. Final Course Materials online
7 March     | 1 week    | Captions deadline & 30-second intro videos due
14 March    | 0         | It begins!

We also provided instructors with an instructors guideline page [tailored to the event](https://gallantries.github.io/video-library/instructors) and more important, a set of recording guidelines [like the Gallantries'](https://gallantries.github.io/video-library/events/smorgasbord2/recording.html) which helped instructors know what was best practice during video recording.


# Workshop Checklists

To help you organize an asynchronous event, we have created some checklists, by timing but also [by role](#checklists-by-role). Most of the items in these checklists are not specific to Gallantries/Smörgåsbord events.

## Before the workshop

### Global organizers

{% include topics/teaching/tutorials/gallantries-async/organizer-before.md %}

### Local helpers

{% include topics/teaching/tutorials/gallantries-async/helper-before.md %}

### Instructors

{% include topics/teaching/tutorials/gallantries-async/instructor-before.md %}

## During the workshop

- Chat online on Slack or another platform
- Enjoy!

### Global organizers

{% include topics/teaching/tutorials/gallantries-async/organizer-during.md %}

### Local helpers

{% include topics/teaching/tutorials/gallantries-async/helper-during.md %}

### Instructors

{% include topics/teaching/tutorials/gallantries-async/instructor-during.md %}

## After the workshop

Debrief
    Sticky notes collection from sites
    share experience back: feedback from sites (extract form from issue in github)
    form feedback from instructors + helpers

### Global organizers

{% include topics/teaching/tutorials/gallantries-async/organizer-after.md %}

### Instructors

{% include topics/teaching/tutorials/gallantries-async/instructor-after.md %}



# Checklists by role

## Global organizers

### Before the workshop

{% include topics/teaching/tutorials/gallantries-async/organizer-before.md %}

### During the workshop

{% include topics/teaching/tutorials/gallantries-async/organizer-during.md %}

### After the workshop

{% include topics/teaching/tutorials/gallantries-async/organizer-after.md %}

## Local helpers

### Before the workshop

{% include topics/teaching/tutorials/gallantries-async/helper-before.md %}

### During the workshop

{% include topics/teaching/tutorials/gallantries-async/helper-during.md %}

## Instructors

### Before the workshop

{% include topics/teaching/tutorials/gallantries-async/instructor-before.md %}

### During the workshop

{% include topics/teaching/tutorials/gallantries-async/instructor-during.md %}

### After the workshop

{% include topics/teaching/tutorials/gallantries-async/instructor-after.md %}


topic: fair
tutorial: inveniordm-integration
title: Integrating InvenioRDM-compatible Repositories with Galaxy
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["fair", "InvenioRDM"]
content_hash: 34d69040b8b952c9dfd5ba357ac8b60f
last_modified: 2026-02-26T21:12:15.971818
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/inveniordm-integration/tutorial.html

content:
# Overview

Discover a more streamlined approach to research data management with Galaxy's integration with InvenioRDM-compatible repositories.

> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction

Any InvenioRDM-compatible repository can be connected to Galaxy, offering a streamlined experience in managing and analyzing your research data. With this integration, you can export research results directly from Galaxy to your institution's InvenioRDM repository or Zenodo, and import files from these repositories into Galaxy for reproducible analysis workflows.


<img style="float: right; width: 150px;" alt="InvenioRDM Logo" src="../../images/inveniordm-logo.svg" />

## InvenioRDM

[InvenioRDM](https://inveniosoftware.org/products/rdm/) is a research data management platform designed to help organizations effectively manage, publish, and showcase their research data. It provides an infrastructure for storing, sharing, and preserving research outputs, making it a valuable tool for researchers and institutions.

## Zenodo

<img style="float: right; margin-left: 1rem;" alt="Zenodo Logo" src="../../images/zenodo-gradient-logo.svg" />

[Zenodo](https://zenodo.org/) is an open repository for all scholarship, enabling researchers from all disciplines to share and preserve their research outputs, regardless of size or format. Free to upload and free to access, Zenodo makes scientific outputs of all kinds citable, shareable and discoverable for the long term.

It's worth noting that [Zenodo, in October 2023, migrated to InvenioRDM as its underlying technical platform](https://blog.zenodo.org/2023/10/13/2023-10-13-zenodo-rdm/). This move not only makes this integration fully compatible with Zenodo but also signals a commitment to enhanced features and scalability, further bolstering the integration's potential benefits for researchers.

# Getting started

For testing purposes, you can use the [Zenodo sandbox instance](https://sandbox.zenodo.org/) to explore the functionality of the plugin.

## Setting up your PAT (Personal Access Token)

To be able to upload files and browse protected records, you need to create an account and set up your PAT (Personal Access Token). In this case, in the Zenodo sandbox instance. To create a new token:

> <hands-on-title>Create Your Personal Access Token</hands-on-title>
>
> 1. Open the [Zenodo sandbox instance](https://sandbox.zenodo.org/) and `Log in` or create an account.
> 
> 2. Go to your `user settings` in the top right corner and select the `Applications` tab.
>
> 3. Then, click on `New token` and give it a name and the necessary permissions.
>
>    ![Zenodo PAT](../../images/zenodo-pat.png)
>
> > <comment-title></comment-title>
> > You will only be able to view and copy the access token when it is first created, so you should copy and securely store the token at this time. If you lose your copy of the access token, then you will need to generate a new one.
> >
> {: .comment}
>
> > <comment-title></comment-title>
> > As InvenioRDM is the underlying technical platform for Zenodo and the Zenodo sandbox instance, you will be able to follow the above steps 2 and 3 to create a PAT after you have logged in to your chosen repository service that uses InvenioRDM.
> >
> {: .comment}
> 
{: .hands_on}

## How to integrate InvenioRDM repositories in Galaxy 25.0

The new `Manage Your Repositories` section is available under `User → Preferences`. We will integrate the Zenodo Sandbox instance for demonstration purposes, but you can also configure your institution's InvenioRDM instance. Here’s how it works:

> <hands-on-title>Integration in Galaxy</hands-on-title>
> 
> 1. Navigate to `User → Preferences → Manage Your Repositories`. If you haven't set up any integrations yet, you'll see an empty list.
> ![Manage Your Remote File Sources](../../images/1-empty-file-sources-list.png)
>
> 2. Click `+ Create` to configure a new integration. You'll see a list of available integrations, including `S3, Dropbox, InvenioRDM, Zenodo`, and more (depending on your Galaxy server).
> ![Create New Integration](../../images/2-create-new.png)
> 
> 3. Select `InvenioRDM` to configure it and enter your **credentials or relevant connection details**. (This will be the access token created in the previous section if using the Zenodo sandbox.)
> ![Configure InvenioRDM](../../images/3-invenio-template-filled.png)
> 
> 4. Once set up, your InvenioRDM integration, in this case, `My Zenodo Sandbox`, will appear in the list where you can manage or delete it.
> ![InvenioRDM Integration](../../images/4-instance-created.png)
> 
> 5. Your selected integrations will appear first when browsing import/export locations. You can find them by clicking on the `Upload` button in the activity bar.
> ![Browse Remote Files](../../images/5-upload-import.png)
{: .hands_on}


> <comment-title></comment-title>
> We are using the Zenodo Sandbox for demonstration purposes. If you want to integrate the "real" Zenodo instance, we recommend using the Zenodo integration instead of InvenioRDM, already in the `+ Create` Galaxy integration step. This will improve the integration further by making it more prominent in some parts of the Galaxy interface. See the tutorial, [Uploading Data to Zenodo from Galaxy]({% link topics/fair/tutorials/zenodo-in-galaxy/tutorial.md %}), for a step by step guide.
>
{: .comment}


> <details-title>Do you want to add a different repository to Galaxy?</details-title>
> 
> There are a number of additional repository types that you can connect to Galaxy and use to import and export data.
>
> {% snippet faqs/galaxy/manage_your_repositories.md %}
>
{: .details}

# Importing records and files into Galaxy

Once you have integrated an InvenioRDM repository into Galaxy, you will be able to import records and files stored in the remote repository into Galaxy.

> <hands-on-title>Importing Files</hands-on-title>
> 
> 1. Open the Upload tool, and select `Choose remote files`
> ![Upload Tool](../../images/upload-tool.png)
>
> 2. Search for the Zenodo sandbox instance. Remember that this will only appear if the InvenioRDM plugin is configured to connect to the Zenodo sandbox in your Galaxy instance.
> ![Choose Remote Files](../../images/choose-remote-files.png)
>
{: .hands_on}

Once you have selected the Zenodo sandbox instance, you can browse public records and import them into your Galaxy history. You can choose to import the full record or individual files in the same way you would import files from any other remote source.

# Exporting your Galaxy history

You can export your history directly to Zenodo. A benefit of publishing your history on Zenodo is that Zenodo will generate a DOI for your history. You can then include this DOI in your publication and it can be used by other researchers to cite your work.

> <hands-on-title>Exporting a History</hands-on-title>
> 
> 1. From the history menu {% icon galaxy-history-options %}
, you can select `Export History to File`
>
> 2. Then choose `to RDM repository`. 
> 
> 3. You will be prompted to select the InvenioRDM repository you want to export your history to. 
> 
> 4. Once you have selected the repository, you can decide whether to create a new record or upload the history to an existing draft record.
> ![Export History to RDM Repository](../../images/export-history.png)
>
{: .hands_on}

You can always edit the record metadata directly from the Zenodo sandbox web interface. Once you are satisfied with the record, you can publish it, generating a DOI that others can use to cite your research. Of course, in the case of the sandbox instance, the DOI is not minted and your data is not stored permanently. But the experience is the same as with the production instance.

# Conclusion

In this tutorial, we have connected an InvenioRDM-compatible repository, such as a Zenodo repository, to a Galaxy instance. We have also imported files and records from Zenodo into Galaxy and exported a Galaxy History to Zenodo.

# References

1. [InvenioRDM and Zenodo integration in Galaxy](https://galaxyproject.org/news/2024-05-03-inveniordm-integration/)
2. [Updates to your InvenioRDM and Zenodo integration in Galaxy
   ](https://galaxyproject.org/news/2025-03-10-inveniordm-integration-update/)


topic: fair
tutorial: fair-metadata
title: Metadata
difficulty: intermediate
time_estimation: 40M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 2e5e3eef4eef01fc6104f281b4967e6f
last_modified: 2026-02-26T21:12:15.970035
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-metadata/tutorial.html

content:
The term metadata is defined along with what constitutes rich metadata.  Learners will be able to recall examples of community/domain standards that apply to data and metadata.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



# Metadata and the FAIR Principles
Metadata relates most directly to the following 10 FAIR Principles highlighted in Table 2.1.  We will discuss and signpost these in this episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a globally unique and persistent identifier<br>**F2. data are described with rich metadata (defined by R1 below)**<br>**F3. metadata clearly and explicitly include the identifier of the data it describes**<br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>**A2. metadata are accessible, even when the data are no longer available** |
| To be Interoperable:        | **I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation** <br>**I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data**                                                                                         |
| To be Reusable:             | **R1. meta(data) are richly described with a plurality of accurate and relevant attributes** <br>**R1.1. (meta)data are released with a clear and accessible data usage license**<br>**R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards**       |

Table 2.1: The 15 FAIR Guiding Principles. Principles relating directly to metadata are highlighted in **black**.

# What is metadata?

Metadata is information that describes your data - it is data about data.

The provision of ‘rich’ metadata is key to FAIR since it allows data to be found, and enables other researchers to interpret data appropriately. ‘Rich’ in this context refers to extensive metadata, often connecting data to other data or terms (even in other datasets), with **qualified references**, Qualified references: terms used to describe relationships to pieces of (meta)data., specifying how they are connected.

If a researcher is given access to a dataset (a spreadsheet in CSV format, for example), the data is not usable without meaningful column headings and context. For example, what is this data about?  Is it part of a larger dataset and, if so, how is it related?  What are the column headings representing, and what are the rows representing?  Additionally, if the values in the individual cells are device or assay measurements, what device was used, and what assay?  

If you look at the example of the spreadsheet below (Figure 2.1) the metadata are coloured blue and the data are in orange.  The data in this example are clinical observations for a cohort of patients and the metadata are the descriptive column headings and additional information about the origin of the data.  The column headings provide context for the data values in each cell.  The contact information enables a researcher to direct questions about data reuse and context.  This raises an interesting point though, because metadata should be rich enough to allow any person to reuse the underlying data without ambiguity (FAIR Principles F2, R1).  

Though  this Figure 2.1 shows the difference between metadata and data, it does not exemplify the use of rich metadata. 

![spreadsheet](../../images/figure2-1_human-welsh-cohort.png "A spreadsheet showing the relationship between data (orange) and metadata (blue)")


We are given only 2 pieces of **data provenance**, Data provenance: metadata describing the origin of a piece of data, including information such as version, original location of the data, and usually an audit trail up to the current version, in this example (i) the study name “LONDON DIABETES COHORT” which could help discover other documentation about the project; (ii) the name of the person who presumably led the study.  While these may help to search for further information, for a human user, this would still be fraught with issues since both names may not uniquely identify the study or its lead.  For a machine agent, such as a script, interpretation is made difficult because the metadata is not clearly marked-up (represented formally with tags).

To fix this, contact details for investigators could be included.  Often [ORCID IDs](https://info.orcid.org/what-is-orcid/) associated with names are used since they uniquely identify an individual and do not pose problems when an investigator moves institutions and the original (email) address changes.  Coupled with this, investigators can be formally assigned a role in the provenance metadata allowing people to contact the appropriate person, for example, “data producer”, “data manager”.

A project URL should be used where possible, ideally one that can act as a **persistent identifier**, Globally unique and persistent identifier: a permanent reference to a digital resource, usually given as a URL, that takes the user to that resource, for the dataset.  The landing page for that URL should provide more context for the dataset, and describe high-level information such as how the dataset relates to other project outputs, as well as providing other provenance information (dates, times, places, people).  This could include more metadata about the type of study, objectives, protocols, dates, release versioning and so on.  At the end of this episode, resources are given to help define metadata elements that could be included, including a project called [Dublin Core](https://www.dublincore.org/specifications/dublin-core/) which defines [15 such elements](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/) or metadata fields.

Terms of access and reuse are missing which could be rectified by including a [data licence](https://rdmkit.elixir-europe.org/licensing#what-licence-should-you-apply-to-your-research-data), which often appears as part of the metadata, usually at the bottom of a webpage hosting data.
There could be ambiguity around the acronym, “BPM”, used in the third column header, so this should be defined within a glossary of acronyms and or ideally hyperlinked to a definition in an existing ontology.

There are issues too with the data, as well as the metadata. The second column DISEASE TYPE could be better designed. Two pieces of information (data) are depicted in the same column: disease type (diabetes) and disease stage (early/late).  Ideally, these should be in 2 separate columns allowing researchers to subset on stage and disease type independently for downstream analysis. There are also four different terms used for diabetes (“Diabetes Mellitus II”, “Diabetes”, “Diabetes Mellitus” and “Diabetes Mellitus I”), which again does not allow a researcher to subset data efficiently.  To fix this you would use defined terms within an existing vocabulary or ontology.  The following accesses a {% cite diseaseontology %} we could use, where each term (for example, “type 2 diabetes mellitus”) is described and assigned a unique ID.  In the example above you would use this unique ID, or the associated descriptive  term, to tag all patients with the same disease, identically.  This then makes the data sub-setable and machine-readable.


> <question-title></question-title>
>
> Look at the final row of data in Figure 2.1.  There is a missing disease type.  What problems might this cause?
>
> > <solution-title></solution-title>
> >
> > This isn’t a metadata problem, but it is worth mentioning that missing data can cause more ambiguity.  Does the blank mean the data were not available, or that the data were available but not recorded, or that the individual has no disease? This has implications on how the data are interpreted, and it is likely this data row would be discarded from an analysis.  Here you would be looking to use terms to denote missing, for example, “no disease”, “not recorded”.
> >
> {: .solution}
{: .question}


# Writing FAIR metadata

We have discussed already how rich metadata enables a dataset to be reused and interpreted correctly.  In the context of the FAIR principles, the previous exercise illustrates two of these, namely that _“(Meta)data are richly described with a plurality of accurate and relevant attributes”_ (FAIR Principle R1) and that _“(Meta)data are associated with detailed provenance”_ (FAIR Principle R1,2).  Further to this, the suggested use of the published {% cite diseaseontology %} for data, illustrates a further three principles, where _“(Meta)data use **vocabularies**,Vocabularies: (or controlled vocabulary) is a dictionary of terms you can use when producing (meta)data, that follow FAIR principles”_ (FAIR Principle I2), and _“(Meta)data meet domain-relevant **community standards**, Community standards: standard guidelines used to structure and exchange data, usually supported by community-developed resources and/or software, (FAIR Principle R1.3). The use of hyperlinks specifically to terms in the ontology means that Metadata includes *qualified references*, Qualified references: terms used to describe relationships to pieces of (meta)data. , to other Metadata (FAIR Principle I3).   From the previous exercise, the [disease ontology](https://disease-ontology.org/) provides the vocabulary for the different types of diabetes: type 1 diabetes mellits {% cite diseaseontology %} and type 2 diabetes mellitus {% cite diseaseontology %}.   

The FAIR Guiding Principles also highlight the importance of providing rich metadata to enable researchers to **find** datasets such that _“Data are described with rich metadata”_ (FAIR Principle F2).  More often than not, a researcher will find data through searching its metadata, usually via an online or a database search.  Information on how this can be achieved is discussed in the next episode on data registration.

The use of vocabularies and cross-references is fundamental to data interoperability.  Interoperable (meta)data can be linked and combined across studies, aided by consistent,  compatible and machine-readable curation.  [FAIRsharing](https://fairsharing.org/search?fairsharingRegistry=Standard) is a useful registry of vocabularies, and standards, while more comprehensive ontology lists are maintained by [OLS (Ontology Lookup Service)](https://www.ebi.ac.uk/ols/index) and [BioPortal](https://bioportal.bioontology.org/).

The previous exercise also touches on **machine-readability**, Machine-readable: (meta)data is supplied in a structured format that can be read by a computer, of (meta)data through mention of using controlled terms in the “DISEASE TYPE” column to allow subsetting.  The [Open Data handbook](https://opendatahandbook.org/glossary/en/terms/machine-readable/) gives a nice overview of machine-readable (meta)data but in short, it is (meta)data supplied in a defined and structured format that can easily be read by an appropriate script or piece of software.  If we use our example of a spreadsheet in comma-separated value form (CSV format), the (meta)data will be organised into cells, in a format that is interoperable with many software.  This would not be true if the same data were made available as a screenshot, highlighting that human-readable data may not be machine-readable.

# Rich metadata in public data repositories

Help with rich metadata curation is often supported by public data repositories, and data deposition is one way you can improve its level of FAIRness.  During submission, metadata is composed and linked, making it understood, accessible and searchable.

The figure below shows a screenshot of a dataset hosted by the [BioStudies](https://www.ebi.ac.uk/biostudies/) (Figure 2.2).  BioStudies is a public database holding descriptions for biological studies and their (meta)data, and is often used by researchers to provide **primary identifiers** to supplementary information described in publications.


![A screenshot of a plant RNAseq dataset housed in BioStudies showing rich metadata like accession, study type, publication, etc.](../../images/figure2-2_rnaseq-database.png "A FAIR dataset in a public repository {% cite RubénSchlaen %}")

> <question-title></question-title>
>
> Look at the example given in Figure 2.2. What metadata is given?
>
> > <solution-title></solution-title>
> >
> > Metadata can be categorised into 3 basic types (i) descriptive (what the data is); (ii) structural (how the data was generated, i.e. provenance); (iii) administrative (data owners, contributors and funders).  In this example, all 3 are given. <br> **Administrative metadata**: authors and organisations underneath the dataset title, and the information in “Publication” <br> **Structural metadata**: “Protocols” and data links on the right-hand side of the page <br>  **Descriptive metadata**: All other metadata on the left-hand side of the page, describing the study type through to “Samples” and “Assays and Data”.  Of note here is the use of controlled vocabularies for metadata such as Organism ([taxonomy database](https://www.ncbi.nlm.nih.gov/taxonomy)) and BioStudy-specific terms for fields such as Experimental Factors, Technology.  These allow a researcher to search for all experiments in the database around a theme or type: with queries of the type: “get me all Arabidopsis RNAseq data”.<br> 
> >
> {: .solution}
{: .question}


# Using community standards for (meta)data 
Public databases often serve communities and specific types of data, and may often use community standards for metadata curation.  These standards include, usually,  open-access **ontologies** that can be used by researchers to annotate their (meta)data.  The [FAIRsharing](https://fairsharing.org/search?fairsharingRegistry=Standard) initiative provides a curated, searchable resource to help find many of these.  The [disease ontology](https://disease-ontology.org/) we have mentioned already in the first exercise has its [own page](https://fairsharing.org/FAIRsharing.8b6wfq) in FAIRsharing.
Another useful resource serving the data needs of specific communities is [RDMkit](https://rdmkit.elixir-europe.org/).  RDMkit is an online research data management toolkit for Life Sciences, and as part of its product hosts pages for domain-specific best practices and guidelines.  [Domain pages](https://rdmkit.elixir-europe.org/your_domain) signpost detail and promote relevant considerations, tools and resources.


> <question-title></question-title>
>
> Familiarise yourself with the [Bioimaging data](https://rdmkit.elixir-europe.org/bioimaging_data) domain page on RDMkit.  Read the section on “Standard (meta)data formats” about a third of the way down. <br> What 3 URLs are given to help a researcher gather appropriate metadata for their images?
>
> > <solution-title></solution-title>
> >
> > [OME model](https://docs.openmicroscopy.org/ome-model/6.3.1/) XML-based representation of microscopy data. <br> [Quality assessment working groups](https://quarep.org/) <br> REMBI {% cite Sarkans2021 %} <br>
> >
> {: .solution}
{: .question}


# Useful resources
- More about Metadata: [RDMkit](https://rdmkit.elixir-europe.org/metadata_management), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/introduction/metadata-fair.html)
- Definition of Machine readability: [Research Data Alliance (RDA)](https://www.rd-alliance.org/groups/long-tail-research-data-ig/wiki/enable-machine-readability.html)
- Definition of Qualified identifiers: [GO FAIR](https://www.go-fair.org/fair-principles/i3-metadata-include-qualified-references-metadata/)
- More about Data licensing: [RDMkit](https://rdmkit.elixir-europe.org/licensing),[ FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/reusability/ATI-licensing.html)
- Definition of Metadata community standards: [FAIRDOM](https://fair-dom.org/community_standards)
- Registry of FAIR metadata standards: [FAIRsharing](https://fairsharing.org/search?fairsharingRegistry=Standard)
- Making interoperable metadata: [Ed-DaSH](https://carpentries-incubator.github.io/fair-bio-practice/06-being-precise/index.html), [Ten simple rules for making a vocabulary FAIR](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009041),<br> [EOSC Guidelines for research institutes to enable discoverability of research data](https://eosc-portal.eu/sites/default/files/Final-D4.4%20Guidelines%20for%20RIs%20to%20enable%20discoverability%20of%20research%20data-pdf.pdf)
- Useful metadata projects and tools: [Dublin Core](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/), [ISA framework](https://isa-tools.org/index.html), [COPO](https://f1000research.com/articles/9-495)


topic: fair
tutorial: data-management-peatlands
title: Introduction to Data Management Plan (DMP) for Peatland Research and PeatDataHub
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["fair", "open", "data management plan"]
content_hash: 57e07cb8f998c4cb0cf2dd076fa8126e
last_modified: 2026-02-26T21:12:15.932287
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/data-management-peatlands/tutorial.html

content:
Peatland monitoring and research generates data that is used for management, conservation and policy related to these ecosystems. To ensure that your project data is collected systematically and can be analysed, shared, and re-used it is important to have a data management plan.

This tutorial is a short introduction to writing a Data Management Plan (DMP) with examples for peatland monitoring projects and initiatives.

You can find more information on DMPs in the [Research Data Management Toolkit for Life Sciences]( https://rdmkit.elixir-europe.org/)

> <agenda-title></agenda-title>
>
> In this tutorial we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# What is a Data Management Plan?

A Data Management Plan (DMP) is a live document that provides a framework for managing data material during and after the research project.

A data management plan will facilitate the process of collecting and managing your peatland data and will allow you to discuss with your team how to share data, considering the requirements of your organisation and funders.

# Key components of a DMP
DMP templates vary by institution and funding agencies. 

| DMP component  | What to include|
| ------------- | ------------- |
| Descriptions of the data  |    &bull; Project details <br> &bull; Types of data <br> collected and generated <br> &bull; Origin of data <br> &bull; Format and size of data |
|Data collection/generation | &bull; Data collection methods <br> &bull; Data quality standards| 
|Data management, documentation, and curation | &bull; Storage and accessibility <br> &bull; Metadata standards and documentation <br> &bull; Curation and preservation |  
|Data sharing and access | &bull; Where the data will be shared <br> &bull; When will the data be available |
|Data security (where relevant)|&bull; Risks to data security<br> &bull; GDPR considerations|
|Capabilities| &bull; Evaluation of institutional capabilities to preserve and manage data|
|Maintaining and implementing the Data Management Plan|&bull; Actions to maintain and deliver the DMP |
|Environmental considerations|&bull; Energy cost considerations related to data management|
|Responsibilities|&bull; Agreement on the responsibilities of named individuals for data management|
|Relevant institutional, departmental or study policies on data sharing and data security| &bull; Review of institutional policies for data management|

## Hands-on: Create a DMP for peatland research
* Download the DMP template for [Peatland Research and Monitoring](https://zenodo.org/records/11185774).
* Fill in the DMP template using the prompt questions.
* Review your DMP with your team and collaborators.

# FAIR data management solutions

After you have filled in the DMP template, review how you can make your DMP FAIR.

There are several ways to set up FAIR (Findable, Accessible, Interoperable, Reusable) data management plans (DMPs) 

* Findable (F): Data description and collection or reuse of existing data.  Using metadata and community recognised standards for curation that enable someone to find the data.
* Accessible (A): Standardised authentication or authorisation (e.g. HTTP, HTTPS) for open and restricted data access.
* Interoperable (I): Documentation and data quality, and community standards for data exchange (e.g. tab-delimited text).
* Reusable (R): Storage and backup supported by legal and ethical requirements, as well as data release policies.

More information is available in the [FAIR data management tutorial]({% link topics/fair/tutorials/data-management/tutorial.md %})

# Conclusion

Creating a DMP will allow you and your institution to make the most of your peatland data. This introduction to DMP aims to highlight the usefulness of data management for peatland research and monitoring.



##  Suggested reading
* [FAIR in a nutshell]({% link topics/fair/tutorials/fair-intro/tutorial.md %}) 
* [The published FAIR Guiding Principles: Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016)]({% cite Wilkinson2016 %})
* [Recipes for data FAIRification, written by domain experts giving real-world examples: FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html)
* [Documentation and frameworks for data FAIRification. Each of the 15 FAIR principles is put into context with real data examples: GO FAIR](https://www.go-fair.org/fair-principles)
* [FAIR walkthrough using examples from across all academic disciplines: How to FAIR](https://howtofair.dk/)

## Resources
* [How to write a data-sharing agreement between collaborators](https://www.youtube.com/watch?v=iaZInoaHa04)
* [DMP online](https://dmponline.dcc.ac.uk)
* [DSWizard](https://bio.tools/tool/Data_Stewardship_Wizard)
* [WaterLands](https://waterlands.eu/)
* [PeatDataHub](https://peatdatahub.net/)


topic: fair
tutorial: ro-crate-workflow-run-ro-crate
title: Workflow Run RO-Crate Introduction
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["ro-crate"]
content_hash: ad797cb2966ab84d379ea334ff0cd026
last_modified: 2026-02-26T21:12:15.985689
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-workflow-run-ro-crate/tutorial.html

content:
The [Workflow Run RO-Crate](https://www.researchobject.org/workflow-run-crate/) working group is developing a series of [RO-Crate profiles](https://www.researchobject.org/ro-crate/profiles.html) for representing the provenance of computational workflow executions. The set includes three profiles, which represent provenance with increasing level of detail:

* [Process Run Crate](https://w3id.org/ro/wfrun/process) can be used to describe the execution of one or more tools that contribute to a generic computation;
* [Workflow Run Crate](https://w3id.org/ro/wfrun/workflow) also describes computational processes, but in this case the execution is supposed to be orchestrated by a computational workflow (e.g., a Galaxy workflow);
* [Provenance Run Crate](https://w3id.org/ro/wfrun/provenance) is based on Workflow Run Crate, but provides additional specifications for describing the details of each step's execution.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}


## Process Run Crate

Consider a simple computation where we select the first four lines of a text file, and then we select the last three lines out of the initial selection. Suppose we start with the following input file, `lines.txt`:

```
one
two
three
four
five
```

Then we run:

```
head --lines 4 lines.txt >sel1.txt
```

The `sel1.txt` file will then be:

```
one
two
three
four
```

Now we run:

```
tail --lines 3 sel1.txt >sel2.txt
```

Obtaining the following `sel2.txt`:

```
two
three
four
```

A Process Run Crate that describes the above computation will contain the three text files, plus an `ro-crate-metadata.json` file structured as follows:

```json
{
    "@context": "https://w3id.org/ro/crate/1.1/context",
    "@graph": [
        {
            "@id": "ro-crate-metadata.json",
            "@type": "CreativeWork",
            "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},
            "about": {"@id": "./"}
        },
        {
            "@id": "./",
            "@type": "Dataset",
            "conformsTo": {"@id": "https://w3id.org/ro/wfrun/process/0.1"},
            "hasPart": [
                {"@id": "lines.txt"},
                {"@id": "sel1.txt"},
                {"@id": "sel2.txt"}
            ],
            "mentions": [
                {"@id": "#runHead"},
                {"@id": "#runTail"}
            ]
        },
        {   "@id": "https://w3id.org/ro/wfrun/process/0.1",
            "@type": "CreativeWork",
            "name": "Process Run Crate",
            "version": "0.1"
        },
        {
            "@id": "https://www.gnu.org/software/coreutils/head",
            "@type": "SoftwareApplication",
            "name": "head",
            "softwareVersion": "8.30"
        },
        {
            "@id": "https://www.gnu.org/software/coreutils/tail",
            "@type": "SoftwareApplication",
            "name": "tail",
            "softwareVersion": "8.30"
        },
        {
            "@id": "#runHead",
            "@type": "CreateAction",
            "name": "Select first four lines",
            "description": "head --lines 4 lines.txt >sel1.txt",
            "endTime": "2023-04-27T15:02:56+02:00",
            "instrument": {"@id": "https://www.gnu.org/software/coreutils/head"},
            "object": {"@id": "lines.txt"},
            "result": {"@id": "sel1.txt"},
            "agent": {"@id": "https://orcid.org/0000-0002-1825-0097"}
        },
        {
            "@id": "#runTail",
            "@type": "CreateAction",
            "name": "Select first four lines",
            "description": "tail --lines 3 sel1.txt >sel2.txt",
            "endTime": "2023-04-27T15:04:02+02:00",
            "instrument": {"@id": "https://www.gnu.org/software/coreutils/tail"},
            "object": {"@id": "sel1.txt"},
            "result": {"@id": "sel2.txt"},
            "agent": {"@id": "https://orcid.org/0000-0002-1825-0097"}
        },
        {
            "@id": "lines.txt",
            "@type": "File"
        },
        {
            "@id": "sel1.txt",
            "@type": "File"
        },
        {
            "@id": "sel2.txt",
            "@type": "File"
        },
        {
            "@id": "https://orcid.org/0000-0002-1825-0097",
            "@type": "Person",
            "name": "Josiah Carberry"
        }
    ]
}
```

The RO-Crate describes two actions, `#runHead` and `#runTail`, which represent, respectively, the execution of the `head` and `tail` programs, referenced via the `instrument` property. Each action references its input(s) via `object` and its output(s) via `result`. Note how the two actions are connected by the fact that the output of `#runHead` is the input of `#runTail`: they form an "implicit workflow", whose steps have been executed manually rather than by a higher level software tool. The actions also refer to their executor via the `agent` property. The following diagram shows the relationships between the various entities:

<img alt="head-tail diagram" src="img/head-tail-diagram.svg" width="600" />

For more information, you can see the [Process Run Crate](https://w3id.org/ro/wfrun/process) profile, which contains the full specification.


## Workflow Run Crate

Suppose now that we write a script to automate the above process:

```bash
#!/usr/bin/env bash

set -euo pipefail

die() {
    echo $1 1>&2
    exit 1
}

nargs=3
if [ $# -ne ${nargs} ]; then
    die "Usage: $0 input_file head_lines tail_lines"
fi
input_file=$1
head_lines=$2
tail_lines=$3

output_file="chunk.txt"

workdir=$(mktemp -d)
head_output="${workdir}/temp.txt"
head --lines ${head_lines} "${input_file}" >"${head_output}"
tail --lines ${tail_lines} "${head_output}" >"${output_file}"
rm -rf "${workdir}"
```

This script can be considered a workflow, since it orchestrates the execution of other applications, managing the flow of data between them, to get the final output. Note that the script is parametrized, so every time it is run we can choose a different input file and different values for the number of lines to extract with `head` and `tail`. If you name the script `head_tail.sh`, you can obtain the same result as above by running:

```
bash head_tail.sh lines.txt 4 3
```

The result will be stored in the `chunk.txt` file.

A Workflow Run Crate to represent a run of this workflow would contain the workflow itself (`head_tail.sh`), the input and output files (`lines.txt`, `chunk.txt`) and an ro-crate-metadata.json file like the following:

```json
{
    "@context": "https://w3id.org/ro/crate/1.1/context",
    "@graph": [
        {
            "@id": "ro-crate-metadata.json",
            "@type": "CreativeWork",
            "about": {"@id": "./"},
            "conformsTo": [
                {"@id": "https://w3id.org/ro/crate/1.1"},
                {"@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0"}
            ]
        },
        {
            "@id": "./",
            "@type": "Dataset",
            "conformsTo": [
                {"@id": "https://w3id.org/ro/wfrun/process/0.1"},
                {"@id": "https://w3id.org/ro/wfrun/workflow/0.1"},
                {"@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0"}
            ],
            "hasPart": [
                {"@id": "head_tail.sh"},
                {"@id": "lines.txt"},
                {"@id": "chunk.txt"}
            ],
            "license": {"@id": "http://spdx.org/licenses/CC0-1.0"},
            "mainEntity": {"@id": "head_tail.sh"},
            "mentions": {"@id": "#runHeadTail"}
        },
        {
            "@id": "https://w3id.org/ro/wfrun/process/0.1",
            "@type": "CreativeWork",
            "name": "Process Run Crate",
            "version": "0.1"
        },
        {
            "@id": "https://w3id.org/ro/wfrun/workflow/0.1",
            "@type": "CreativeWork",
            "name": "Workflow Run Crate",
            "version": "0.1"
        },
        {
            "@id": "https://w3id.org/workflowhub/workflow-ro-crate/1.0",
            "@type": "CreativeWork",
            "name": "Workflow RO-Crate",
            "version": "1.0"
        },
        {
            "@id": "head_tail.sh",
            "@type": ["File", "SoftwareSourceCode", "ComputationalWorkflow"],
            "name": "Head-Tail",
            "programmingLanguage": {"@id": "#bash"},
            "input": [
                {"@id": "#input_file-param"},
                {"@id": "#head_lines-param"},
                {"@id": "#tail_lines-param"}
            ],
            "output": [
                {"@id": "#output_file-param"}
            ]
        },
        {
            "@id": "#input_file-param",
            "@type": "FormalParameter",
            "additionalType": "File",
            "name": "input_file",
            "valueRequired": "True"
        },
        {
            "@id": "#head_lines-param",
            "@type": "FormalParameter",
            "additionalType": "Integer",
            "name": "head_lines",
            "valueRequired": "True"
        },
        {
            "@id": "#tail_lines-param",
            "@type": "FormalParameter",
            "additionalType": "Integer",
            "name": "tail_lines",
            "valueRequired": "True"
        },
        {
            "@id": "#output_file-param",
            "@type": "FormalParameter",
            "additionalType": "File",
            "name": "output_file"
        },
        {
            "@id": "#bash",
            "@type": "ComputerLanguage",
            "identifier": "https://www.gnu.org/software/bash/",
            "name": "Bash",
            "url": "https://www.gnu.org/software/bash/"
        },
        {
            "@id": "#runHeadTail",
            "@type": "CreateAction",
            "name": "Run of head_tail.sh",
            "endTime": "2023-05-04T17:02:01+02:00",
            "instrument": {"@id": "head_tail.sh"},
            "object": [
                {"@id": "lines.txt"},
                {"@id": "#head_lines-pv"},
                {"@id": "#tail_lines-pv"}
            ],
            "result": [
                {"@id": "chunk.txt"}
            ],
            "agent": {"@id": "https://orcid.org/0000-0002-1825-0097"}
        },
        {
            "@id": "lines.txt",
            "@type": "File",
            "exampleOfWork": {"@id": "#input_file-param"}
        },
        {
            "@id": "#head_lines-pv",
            "@type": "PropertyValue",
            "exampleOfWork": {"@id": "#head_lines-param"},
            "name": "head_lines",
            "value": "4"
        },
        {
            "@id": "#tail_lines-pv",
            "@type": "PropertyValue",
            "exampleOfWork": {"@id": "#tail_lines-param"},
            "name": "tail_lines",
            "value": "3"
        },
        {
            "@id": "chunk.txt",
            "@type": "File",
            "exampleOfWork": {"@id": "#output_file-param"}
        },
        {
            "@id": "https://orcid.org/0000-0002-1825-0097",
            "@type": "Person",
            "name": "Josiah Carberry"
        }
    ]
}
```

Note how the crate has a `mainEntity` that points to the workflow (`head_tail.sh`). This is in conformance with the [Workflow RO-Crate](https://w3id.org/workflowhub/workflow-ro-crate/1.0) profile. In addition to Workflow RO-Crate, Workflow Run Crates also conform to the [Process Run Crate](https://w3id.org/ro/wfrun/process) profile and to the [Workflow Run Crate](https://w3id.org/ro/wfrun/workflow) profile itself, as you can see from the `conformsTo` entries.

One of the main additions with respect to Process Run Crate is the representation of parameter "slots" (which can take different values according to the specific run) via `FormalParameter`. In this case we have three input parameters:

 * `#input_file-param`, corresponding to the input file
 * `#head_lines-param`, corresponding to the number of lines to select with the `head` program
 * `#tail_lines-param`, corresponding to the number of lines to select with the `tail` program

The `#runHeadTail` action has three `object` entries, which represent the actual values used for the parameters in this specific run. The first item is the `lines.txt` file, while the other two are represented via [PropertyValue](https://schema.org/PropertyValue) instances, where the `value` property provides the values taken by the parameters. Crucially, the `exampleOfWork` property connects each parameter value to the corresponding `FormalParameter`. Similarly, the `chunk.txt` file matches the output parameter `#output_file-param`.

An example of a Workflow Run Crate for a Galaxy workflow is available at [https://doi.org/10.5281/zenodo.7785860](https://doi.org/10.5281/zenodo.7785860).

You can export a Galaxy workflow invocation as a Workflow Run Crate by following [this tutorial](https://by-covid.github.io/gtn/ro-crate-in-galaxy/).


## Provenance Run Crate

[Provenance Run Crate](https://w3id.org/ro/wfrun/provenance) further extends Workflow Run Crate, specifying how to represent the actions corresponding to individual steps in the workflow. The extent to which this can be done may depend on how the workflow is written and how the workflow engine operates. For instance, the shell script we used in the previous section does not keep the intermediate output from the execution of `head`: it's stored in a temporary directory, which is deleted at the end of the script.

A workflow management system that supports Provenance Run Crate is [StreamFlow](https://streamflow.di.unito.it/), which executes [CWL](https://www.commonwl.org/) workflows. Let's write a CWL workflow that implements our head-tail application. We start by writing `headtool.cwl`, a wrapper for the `head` tool:

```yaml
class: CommandLineTool
cwlVersion: v1.0

baseCommand: head

inputs:
  lines:
    type: int
    inputBinding:
      position: 1
      prefix: "--lines"
  input:
    type: File
    inputBinding:
      position: 2
outputs:
  output:
    type: File
    outputBinding:
      glob: head_out.txt
stdout: head_out.txt
```

We then do the same for `tail`, calling the wrapper `tailtool.cwl`:

```yaml
class: CommandLineTool
cwlVersion: v1.0

baseCommand: tail

inputs:
  lines:
    type: int
    inputBinding:
      position: 1
      prefix: "--lines"
  input:
    type: File
    inputBinding:
      position: 2
outputs:
  output:
    type: File
    outputBinding:
      glob: tail_out.txt
stdout: tail_out.txt
```

Finally, we write the workflow, `head_tail.cwl`:

```yaml
class: Workflow
cwlVersion: v1.0

inputs:
  input_file:
    type: File
  head_lines:
    type: int
  tail_lines:
    type: int
outputs:
  output_file:
    type: File
    outputSource: tail/output

steps:
  head:
    in:
      lines: head_lines
      input: input_file
    out: [output]
    run: headtool.cwl
  tail:
    in:
      lines: tail_lines
      input: head/output
    out: [output]
    run: tailtool.cwl
```

Now that we have the workflow, let's write the file that contains the parameter settings for our specific run, `params.yml`:

```yaml
input_file:
  class: File
  location: lines.txt
head_lines: 4
tail_lines: 3
```

Now, install StreamFlow (we're going to need a version with RO-Crate support). To avoid possible conflicts, consider installing in a clean [virtual environment](https://docs.python.org/3/library/venv.html).

```
pip install 'streamflow==0.2.0.dev4'
```

Now run the workflow:

```
cwl-runner head_tail.cwl params.yml
```

To generate the RO-Crate, we're going to need the workflow's name in the StreamFlow database:

```console
$ streamflow list
NAME                                  TYPE EXECUTIONS 
f6b2273d-a945-4b27-aa98-dcfe0b778f4c  cwl  1
```

In your case, the name might be different. Now generate the RO-Crate with:

```
streamflow prov f6b2273d-a945-4b27-aa98-dcfe0b778f4c
```

taking care to replace the workflow name with the actual one you got from `streamflow list`. This will generate a zip archive containing the RO-Crate. Unzip the archive to get the RO-Crate directory:

```
unzip -d f6b2273d-a945-4b27-aa98-dcfe0b778f4c{,.crate.zip}
```

If you inspect `f6b2273d-a945-4b27-aa98-dcfe0b778f4c/ro-crate-metadata.json`, you will find a similar structure to the one seen for Workflow Run Crate, but with a lot more detail. Most notably, there will be three actions: in addition to the one representing the execution of the whole workflow, there will be two more, representing the execution of `headtool.cwl` and of `tailtool.cwl`. Each of this points to the corresponding tool via `instrument`, and to its inputs and outputs via `object` and `result`. The intermediate file resulting from the `head` run (`head_out.txt`) is included, and listed both as an output of the `head` run and as an input of the `tail` run. This situation is similar to what we've seen in the [Process Run Crate example](#process-run-crate), but this time everything has been orchestrated by a workflow engine on the basis of a formal workflow description.

More information on the entities represented in a Provenance Run Crate is available in the [specification](https://w3id.org/ro/wfrun/provenance).

An example a Provenance Run Crate representing an execution of a digital pathology image annotation workflow is available at [https://doi.org/10.5281/zenodo.7669622](https://doi.org/10.5281/zenodo.7669622).


topic: fair
tutorial: ro-crate-galaxy-best-practices
title: Best practices for workflows in GitHub repositories
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["ro-crate"]
content_hash: b7f35a5f43ab9b16fda3a50f49bdb69c
last_modified: 2026-02-26T21:12:15.972988
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.html

content:
A workflow, just like any other piece of software, can be formally correct and runnable but still lack a number of additional features that might help its reusability, interoperability, understandability, etc.

One of the most useful additions to a workflow is a suite of tests, which help check that the workflow is operating as intended. A test case consists of a set of inputs and corresponding expected outputs, together with a procedure for comparing the workflow's actual outputs with the expected ones. It might be the case, in fact, that a test may be considered successful even if the actual outputs do not match the expected ones exactly, for instance because the computation involves a certain degree of randomness, or the output includes timestamps or randomly generated identifiers.

Providing documentation is also important to help understand the workflow's purpose and mode of operation, its requirements, the effect of its parameters, etc. Even a single, well structured README file can go a long way towards getting users started with your workflow, especially if complemented by examples that include sample inputs and running instructions.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn about the best practices that the Galaxy community 
> has created for workflows.
>
> 1. TOC
> {:toc}
>
{: .agenda}

> <tip-title>Using your own workflow</tip-title>
>
> This tutorial assumes that you already have a Galaxy workflow that you want to apply best practices to. You can follow along using any workflow you have created or imported during a previous tutorial (such as [A short introduction to Galaxy]({% link topics/introduction/tutorials/galaxy-intro-short/workflows/index.md %})).
>
{: .tip}

## Community best practices

Though the practices listed in the introduction can be considered general enough to be applicable to any kind of software, individual communities usually add their own specific sets of rules and conventions that help users quickly find their way around software projects, understand them more easily and reuse them more effectively. The Galaxy community, for instance, has a [guide on best practices for maintaining workflows](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html) and a built-in Best Practices panel in the workflow editor (see the tip below).

{% snippet faqs/galaxy/workflows_best_practices.md %}

> <hands-on-title>Apply best practices for workflow structure</hands-on-title>
>
> 1. Open your workflow for editing and find the **Best Practices** panel (see the tip above).
> 1. Resolve the warnings that appear until every item has a green tick.
>
{: .hands_on}

The [Intergalactic Workflow Commission (IWC)](https://github.com/galaxyproject/iwc) is a collection of highly curated Galaxy workflows that follow best practices and conform to a specific GitHub directory layout, as specified in the [guide on adding workflows](https://github.com/galaxyproject/iwc/blob/main/workflows/README.md#adding-workflows). In particular, the workflow file must be accompanied by a [Planemo test file](https://planemo.readthedocs.io/en/latest/test_format.html) with the same name but a `-test.yml` extension, and a `test-data` directory that contains the datasets used by the tests described in the test file. The guide also specifies how to fulfill other requirements such as setting a license, a creator and a version tag. A new workflow can be proposed for inclusion in the collection by opening a pull request to the [IWC repository](https://github.com/galaxyproject/iwc): if it passes the review and is merged, it will be published to [iwc-workflows](https://github.com/iwc-workflows). The publication process also generates a metadata file that turns the repository into a [Workflow Testing RO-Crate](https://crs4.github.io/life_monitor/workflow_testing_ro_crate), which can be registered to [WorkflowHub](https://workflowhub.eu/) and [LifeMonitor](https://www.lifemonitor.eu/).


## Best practice repositories and RO-Crate

The [repo2rocrate](https://github.com/crs4/repo2rocrate) software package allows to generate a [Workflow Testing RO-Crate](https://crs4.github.io/life_monitor/workflow_testing_ro_crate) for a workflow repository that follows community best practices. It currently supports Galaxy (based on IWC guidelines), Nextflow and Snakemake. The tool assumes that the workflow repository is structured according to the community guidelines and generates the appropriate [RO-Crate](https://w3id.org/ro/crate/) metadata for the various entities. Several command line options allow to specify additional information that cannot be automatically detected or needs to be overridden.

To try the software, we'll clone one of the iwc-workflows repositories, whose layout is known to respect the IWC guidelines. Since it already contains an RO-Crate metadata file, we'll delete it before running the tool.

```bash
pip install repo2rocrate
git clone https://github.com/iwc-workflows/parallel-accession-download
cd parallel-accession-download/
rm -fv ro-crate-metadata.json
repo2rocrate --repo-url https://github.com/iwc-workflows/parallel-accession-download
```

This adds an `ro-crate-metadata.json` file at the top level with metadata generated based on the tool's knowledge of the expected repository layout. By specifying a zip file as an output with the `-o` option, we can directly generate an RO-Crate in the format accepted by WorkflowHub and LifeMonitor:

```bash
repo2rocrate --repo-url https://github.com/iwc-workflows/parallel-accession-download -o ../parallel-accession-download.crate.zip
```


## Generating tests for your workflow

What if you only have a workflow, but you don't have the test layout yet? You can use Planemo to generate it.

```bash
pip install planemo
```

As an example we will use this [simple workflow](https://github.com/crs4/life_monitor/blob/50cdb790ff125613aa07e70cb439e3a36b82d0bf/interaction_experiments/workflow_examples/galaxy/sort-and-change-case/sort-and-change-case.ga), which has only two steps: it sorts the input lines and changes them to upper case. Follow these steps to generate a test layout for it:


> <hands-on-title>Generate Workflow Tests With Planemo</hands-on-title>
> 1. Download [the workflow](https://raw.githubusercontent.com/crs4/life_monitor/50cdb790ff125613aa07e70cb439e3a36b82d0bf/interaction_experiments/workflow_examples/galaxy/sort-and-change-case/sort-and-change-case.ga) to a `sort-and-change-case.ga` file.
> 1. Download [this input dataset](https://raw.githubusercontent.com/crs4/life_monitor/50cdb790ff125613aa07e70cb439e3a36b82d0bf/interaction_experiments/workflow_examples/galaxy/sort-and-change-case/input.bed) to an `input.bed` file.
> 1. Upload the workflow to Galaxy (e.g., [Galaxy Europe](https://usegalaxy.eu/)): from the upper menu, click on "Workflow" > "Import" > "Browse", choose `sort-and-change-case.ga` and then click "Import workflow".
> 1. Rename the uploaded workflow from `sort-and-change-case (imported from uploaded file)` to `sort-and-change-case` by clicking the pencil icon next to the workflow name. 
> 1. Start a new history: click on the "+" button on the History panel to the right.
> 1. Upload the input dataset to the new history: on the left panel, go to "Upload Data" > "Choose local files" and select `input.bed`, then click "Start" > "Close".
> 1. Wait for the file to finish uploading (i.e., for the loading circle on the dataset's line in the history to disappear).
> 1. Run the workflow on the input dataset: click on "Workflow" in the upper menu, locate `sort-and-change-case`, and click on the play button to the right.
> 
>   ![Workflow Entry](img/workflow-entry.png)
> 
> 1. This should take you to the workflow running page. The input slot should be already filled with `input.bed` since there is nothing else in the history. Click on "Run Workflow" on the upper right of the center panel.
> 
>    ![Workflow Run Page](img/workflow-run-page.png)
> 
> 1. Wait for the workflow execution to finish.
> 1. On the upper menu, go to "Data" > "Workflow Invocations", expand the invocation corresponding to the workflow just run and copy the invocation's ID. In my case it says "Invocation ID: 86ecc02a9dd77649" on the right, where `86ecc02a9dd77649` is the ID.
>  
>    ![Workflow Invocation](img/workflow-invocation.png)
> 
> 1. On the upper menu, go to "User" > "Preferences" > "Manage API Key". If you don't have an API key yet, click the button to create a new one. Under "Current API key", click the button to copy the API Key on the right.
> 
>    ![API key](img/api-key.png)
> 
> 1. Run `planemo workflow_test_init --galaxy_url https://usegalaxy.eu --from_invocation INVOCATION_ID --galaxy_user_key API_KEY`, replacing `INVOCATION_ID` with the actual invocation ID and `API_KEY` with the actual API key. If you're not using the Galaxy Europe instance, also replace `https://usegalaxy.eu` with the URL of the instance you're using.
> 1. Browse the files that have been created - `sort-and-change-case-tests.yml` and `test_data/`
>
> Optionally see this tip for more details:
>
> {% snippet faqs/gtn/gtn_workflow_testing.md %}
>
{: .hands_on}

> <question-title></question-title>
>
> 1. How do the files in `test_data/` relate to your Galaxy history?
> 2. Look at the contents of `sort-and-change-case-tests.yml`. What are the expected outputs of the test?
>
> > <solution-title></solution-title>
> >
> > 1. The files in `test_data/` correspond to the output files in the history, though some of the names are different:
> >   1. `bed_input.bed` has the same name in the history - this is the input file we uploaded
> >   2. `sorted_bed.bed` corresponds to the `Sort on data 1` step (you can confirm this by viewing the file contents)
> >   2. `uppercase_bed.tabular` corresponds to the `Change case on data 2` step (you can confirm this by viewing the file contents)
> > 2. The expected outputs are `test-data/sorted_bed.bed` and `test-data/uppercase_bed.tabular`. This means that when the workflow is run on the input (`test-data/bed_input.bed`), it is expected to produce two files that look exactly like those outputs.
> >
> {: .solution}
{: .question}

To build up the test suite further, you can invoke the workflow multiple times with different inputs, and use each invocation to generate a test, using the same command as before:

```bash
planemo workflow_test_init --galaxy_url https://usegalaxy.eu --from_invocation INVOCATION_ID --galaxy_user_key API_KEY
```

Each invocation should test a different behavior of the workflow. This could mean using different datatypes for inputs, or changing the workflow settings to produce different results.

> <hands-on-title>Generate tests for your own workflow</hands-on-title>
> 1. Create a new folder on your computer to store the workflow.
> 1. Download the Galaxy workflow you updated to follow best practices earlier in this tutorial. You can do this by going to the Workflow page and clicking {% icon galaxy-download %} **Download workflow in .ga format**.
> 1. Create a new Galaxy history, and run the workflow on some appropriate input data.
> 1. Use `planemo` to turn that workflow invocation into a test case.
{: .hands_on}


## Adding a GitHub workflow for running tests automatically

In the previous section, you learned how to generate a test layout for an example Galaxy workflow. This procedure also gives you the file structure you need to populate the GitHub repository in line with community best practices. One thing is still missing though: a GitHub workflow to test the Galaxy workflow automatically. Let's create this now.

At the top level of the repository, create a `.github/workflows` directory and place a `wftest.yml` file inside it with the following content:

```yaml
name: Periodic workflow test
on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:
jobs:
  test:
    name: Test workflow
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 1
    - uses: actions/setup-python@v1
      with:
        python-version: '3.7'
    - name: install Planemo
      run: |
        pip install --upgrade pip
        pip install planemo
    - name: run planemo test
      run: |
        planemo test --biocontainers sort-and-change-case.ga
```

Replacing `sort-and-change-case.ga` with the name of your actual Galaxy workflow. You can find extensive [documentation on GitHub workflows](https://docs.github.com/en/actions/using-workflows) on the GitHub web site. Here we'll give some highlights:

 * the `on` field sets the GitHub workflow to run:
   * automatically every day at 3 AM
   * when manually dispatched
 * the steps do the following:
   * check out the GitHub repository
   * set up a Python environment
   * install Planemo
   * run `planemo test` on the Galaxy workflow

An example of a repository built according to the guidelines given here is [simleo/ccs-bam-to-fastq-qc-crate](https://github.com/simleo/ccs-bam-to-fastq-qc-crate), which realizes the Workflow Testing RO-Crate setup for [BAM-to-FASTQ-QC](https://workflowhub.eu/workflows/220).

Your workflow is now ready to add to GitHub! If you're not familiar with GitHub, follow these instructions to create a repository and upload your workflow files: [Uploading a project to GitHub](https://docs.github.com/en/get-started/start-your-journey/uploading-a-project-to-github).


topic: fair
tutorial: med-chem-data
title: Data management in Medicinal Chemistry
difficulty: introductory
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "data-management", "medicinal-chemistry", "computational-chemistry"]
content_hash: 90fafe76f1a52d7c8bc52e11c9055bb1
last_modified: 2026-02-26T21:12:14.644291
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/med-chem-data/tutorial.html

content:
The development of medicinal chemistry is advancing very rapidly. Big pharmaceutical companies, research institutes and universities are working on ground-breaking solutions to help patients combat all kinds of diseases. During that development process, tons of data are generated – not only from the lab environment but also from clinical trials. Given that the discovery of more potent, safer and cheaper drugs is the ultimate goal of all research bodies, we should all focus on making the data we gather FAIR: **F**indable, **A**ccessible, **I**nteroperable, and **R**eusable to push the boundaries of drug development even further. 

With the currently available methods such as artificial intelligence, machine learning, many toolkits, software and access to various databases, managing big data is now inherently linked to medicinal chemistry and helps to make this area as efficient as it can be. 

In this tutorial, we will therefore explore some concepts related to medicinal chemistry, explore the available chemical and pharmacological databases, and perform some basic data analyses using the Galaxy interface. 

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Principles of medicinal chemistry research

## Drug-likeness
There are many factors that medicinal chemists take into account while designing new drugs. It is essential to estimate the properties of the molecule before synthesising it in the lab, so the structures of drug candidates are usually compared to the existing drugs by so-called drug-likeness. This includes intrinsic properties of a compound that will lead to favourable ADMET (**A**bsorption, **D**istribution, **M**etabolism, **E**xcretion, **T**oxicity) characteristics. Below are some properties usually assessed:
- Lipophilicity
- Size
- Polarity
- Insolubility
- Unsaturation
- Flexibility

Additionally, there have been some rules developed to help estimate drug-likeness, amongst which:
- Lipinski rule ({% cite Lipinski2001 %})
- Ghose rule ({% cite Ghose1999 %})
- Veber rule ({% cite Veber2002 %})
- Egan rule ({% cite Egan2000 %})
- Muegge rule ({% cite Muegge2001 %})
- Abbott bioavailability score ({% cite Martin2005 %})


As you see, there are a number of characteristics to consider before even synthesising the molecule in a lab. One of the online tools that help to summarise all the most important information about intrinsic properties and drug-likeness of compounds is called [SwissADME](http://www.swissadme.ch/) and is a free web tool to evaluate pharmacokinetics, drug-likeness and medicinal chemistry friendliness of small molecules ({% cite Daina2017 %}). 

Below we briefly discuss two probably the most well-known rules: Lipinski's and Veber's rules.

## The Lipinski rule of 5

This rule of thumb developed back in 1997 by Christopher Lipinski and colleagues tries to predict the likelihood that a given small molecule can be orally active. The Lipinski rule of 5 favours molecules as potential oral drug candidates if:
-	the molecular mass is less than 500
-	the calculated logarithm of the octanol−water partition coefficient (clogP) is less than 5
-	they have up to 5 H-bond donors
-	they have up to 10 H-bond acceptors

However, nowadays there are more and more drugs being developed which don’t comply with those rules and regardless are still effective. There are voices from the scientific community, pointing out that “We are in danger of repeating our past mistakes if we assume these new modalities are not 'drug-like' and cannot be oral drugs because they are not [rule of 5] compliant” (Michael Shultz from Novartis, cited in {% cite Mullard2018 %} where the authors re-assess the rule of 5). Then, in {% cite OHagan2014 %} we read "This famous "rule of 5" has been highly influential in this regard, but only about 50 % of orally administered new chemical entities actually obey it." 


## Veber's rule

In ({% cite Veber2002 %}) we read that the commonly applied molecular weight cutoff at 500 does not itself significantly separate compounds with poor oral bioavailability from those with acceptable values. The authors explain that on average both the number of rotatable bonds and polar surface area or hydrogen bond count tend to increase with molecular weight which may in part explain the success of the molecular weight parameter in predicting oral bioavailability. Their observations suggest that compounds will have a high probability of good oral bioavailability if they meet the two criteria: 
- 10 or fewer rotatable bonds - increased rotatable bond count has a negative effect on the permeation rate
- a polar surface area no greater than 140 Å <sup>2</sup> (or 12 or fewer H-bond donors and acceptors) - reduced polar surface area correlates better with increased permeation rate than does lipophilicity


# Big data for drug discovery

Have you ever wondered how new drugs are developed? It is actually quite a long and complex pathway and not only takes lots of time but also resources. The scheme below will help you understand the scale of that process.

![A scheme showing drug discovery pipeline: starting from target identification, hit, lead, lead optimisation, pre-clinical testing, clinical evaluation in phases I - III, and finally regulatory approval. Lead optimisation also includes iterative steps of designing, making, testing and analysing the proposed compounds.]({% link topics/computational-chemistry/images/medchem.png %} "A scheme showing drug discovery pipeline")

As you can see, to get just one drug to the market, often thousands of structures are assessed to find the most effective and safe compound. That's lots of information being generated in the whole process! The information can be stored in repositories and databases so that subsequent drug discovery process in a similar area is easier, faster and cheaper. Managing big data and using machine learning and computational chemistry methods is crucial in the lead optimisation step. By reviewing chemical databases and studying molecular docking simulations, we can save much time and resources to synthesise only those compounds that are the most promising based on *in-silico* methods. Even after synthesising the optimised 'leads', we still get lots of data from tests and analyses. By using appropriate cheminformatic tools and software, we can use the results to drive forward this iterative process of lead optimisation.


## Ten Vs of big data for drug discovery

Hopefully, now you have an idea of what the process of drug discovery looks like. To understand why managing data in medicinal chemistry is so challenging, let's have a closer look at the "Ten Vs" of big data {% cite Zhao2020 %}:

- **V**olume: size of data. 
- **V**elocity: speed of new data generation. 
- **V**ariety: various formats of data. 
- **V**eracity: quality of data. 
- **V**alidity: authenticity of data. 
- **V**ocabulary: terminology of data. 
- **V**enue: platform of data generation. 
- **V**isualization: view of data. 
- **V**olatility: duration of data usefulness. 
- **V**alue: the potential of data usefulness to reduce the cost of drug discovery and development.

Medicinal chemists must keep in mind those features while both searching for data and publishing it. In this way we can focus on the main aspects of data management, try to improve the availability and normalisation of data, and be aware of limitations of repositories and inconsistencies in data quality. 

## FAIR MedChem
As you saw before, there are many properties that the compounds can be searched by, such as molecular weight, H-bond donors or acceptors, polarity, etc. Sometimes there is a need to assess other molecular properties but it might happen that not all the information is included in the database. This is one of the reasons why new repositories are being developed – they are more specific and gather particular properties of interest. How much easier the life of scientists could be if the relevant data was publicly available, well-ordered and contained the needed metadata? By submitting the data with the necessary information to the repository is a good way to make the data FAIR. This will make it:
- **F**indable, as the data will be given specific identifiers
- **A**ccessible, as the data will be available online, open and free where possible
-	**I**nteroperable, as the repository will often enforce the use of formalised, consistent language
-	**R**eusable, as the data will be released under a license with detailed provenance
In the repositories 


# Chemical and pharmacological databases

Currently, there are lots of publicly available databases storing information about hit molecules, chemical structures and properties, drug targets, pharmaceuticals… In the recent paper {% cite Zhao2020 %} the authors collated relevant databases to advance computer-aided drug discovery and divided them into several main groups:
- Chemical collections
- Drug / drug-like compounds
- Drug targets, including genomics and proteomics data
- Biological data from assay screening, metabolism, and efficacy studies
- Drug liabilities and toxicities
- Clinical databases

Below you will find the databases listed under corresponding categories, all taken from {% cite Zhao2020 %} paper. 

There are many more databases available, and many are still being developed. They are often quite specific and contain certain types of compounds (eg. [PROTACs](http://cadd.zju.edu.cn/protacdb/) or are aimed at a particular disease. If the listed databases are not specific enough for your research, you can try luck by searching smaller, more specific databases. 

## Chemical collections

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [Enamine REAL Database](https://enamine.net/hit-finding/compound-collections/real-database) | Tool used to find new hit molecules using large-scale virtual screening and for searching analogues to hit molecules | >700 million compounds that comply with ‘rule of 5’ and Veber criteria |
| [ZINC](http://zinc.docking.org/) | Contains compound information including 2D/3D structure, purchasability,   target, and biology-related information | >230 million compounds in 3D formats and >750 million compounds for analogue searching |
| [PubChem](https://pubchem.ncbi.nlm.nih.gov) | Contains chemical molecule (mostly small molecule) information, including chemical structures, identifiers, chemical and physical properties,   biological activities, safety and toxicity data | 97 million compounds, 236 million substances, 268 million bioactivities |
| [ChemSpider](https://www.chemspider.com/) | Free chemical structure database providing fast access to >78 million   structures, properties, and associated information | >78 million compound structures |
| [SCUBIDOO](http://kolblab.org/scubidoo/index.php) | Freely accessible database that currently holds 21 million virtual products originating from a small library of building blocks and collection of robust organic reactions | 21 million virtual products |
| [ChEMBL](https://www.ebi.ac.uk/chembl/) | Manually curated database of bioactive molecules with drug-like properties; brings together chemical, bioactivity, and genomic data to aid translation of genomic information into effective new drugs. | >1.9 million compounds, 1.1 million pieces of assay information |
| [TCM-Mesh](http://mesh.tcm.microbioinformatics.org/) | Integration of a database and a data-mining system for network pharmacology analysis of all respects of traditional Chinese medicine,   including herbs, herbal ingredients, targets, related diseases, adverse effect, and toxicity | 383 840 compounds, 6235 herbs |
| [Super Natural II](http://bioinf-applied.charite.de/supernatural_new/index.php) | Contains natural compounds, including information about corresponding 2D   structures, physicochemical properties, predicted toxicity class and potential vendors | 325 508 natural compounds |
| [BIAdb](https://webs.iiitd.edu.in/raghava/biadb/index.html) | Comprehensive database of benzylisoquinoline alkaloids, containing information about 846 unique benzylisoquinoline alkaloids | ~846 unique benzylisoquinoline alkaloids |

## Drug / drug-like compounds

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [AICD](http://956023.ichengyun.net/AICD/index.php) | Anti-Inflammatory Compounds Database (AICD) deposits compounds with potential antiinflammation activities | 79 781 small molecules |
| [DrugBank](https://go.drugbank.com/) | Unique bioinformatics and cheminformatics resource that combines detailed drug data with comprehensive drug target information | 13 441 drug entries |
| [ReFRAME](https://reframedb.org/) | Screening library of 12 000 molecules assembled by combining three databases (Clarivate Integrity, GVK Excelra GoStar, and Citeline   Pharmaprojects) to facilitate drug repurposing | 12 000 molecules |
| [SuperDRUG2](http://cheminfo.charite.de/superdrug2/) | Contains approved/marketed drugs with regulatory details, chemical structures (2D and 3D), dosage, biological targets, physicochemical properties, external identifiers, adverse effects, and PK data | >4600 active pharmaceutical ingredients |
| [Drugs@FDA database](https://www.fda.gov/drugs/drug-approvals-and-databases/drugsfda-data-files) | Information about drugs from FDA | ~23 391 drug application records |
| [e-Drug3D](https://chemoinfo.ipmc.cnrs.fr/MOLDB/index.php) | Contains 1930 molecular structures approved by FDA between 1939 and 2019   with a molecular weight <2000 | 1930 drugs |

## Drug targets, including genomics and proteomics data

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [BindingDB](https://www.bindingdb.org/bind/index.jsp) | Public, web-accessible database of measured binding affinities, focusing chiefly on interactions of proteins considered to be candidate drug targets with ligands that are small,   drug-like molecules | 1 756 093 binding data, for 7371 protein targets and 780 240 small molecules |
| [Supertarget](http://insilico.charite.de/supertarget/index.php?site=home) | An extensive web resource for analyzing drugtarget interactions. | 332 828 drug-target interactions |
| [Ligand Expo](http://ligand-expo.rutgers.edu/index.html) | Provides chemical and structural information about small molecules within structure entries of Protein Data   Bank. | 30 440 entries of ligand |
| [PDBeChem](https://www.ebi.ac.uk/pdbe-srv/pdbechem/) | Consistent and enriched library of ligands, small molecules, and monomers referenced as residues and hetgroups in PDB entries | >29 922 ligands |
| [PDBbind-CN](http://www.pdbbind-cn.org/) | Provides essential linkage between energetic and structural information of biomolecular complexes, which is helpful for various computational and statistical studies on molecular recognition in biological systems | 19 588 biomolecular complexes |
| [STITCH](http://stitch.embl.de/) | Database integrating information about interactions from metabolic pathways, crystal structures, binding experiments, and drug–target relationships | Interactions between 300 000 small molecules and 2.6 million proteins from 1133 organisms |
| [BioGRID](https://thebiogrid.org/) | The Biological General Repository for Interaction Datasets is an open-access database on protein, genetic, and chemical interactions for humans and all major model organisms | 1 753 686 protein and genetic interactions, 28 093 chemical associations and 874 796 posttranslational modifications from major model organisms |
| [Binding MOAD](http://bindingmoad.org/) | Created from a subset of Protein   Data Bank (PDB), containing every high-quality example of ligand-protein binding. | 36 047 protein-ligand structures, and 13 353 binding data |
| [GPCRdb](https://gpcrdb.org/) | Contains data from GPCRs, including crystal structures, sequence alignments, and receptor mutations; can be visualized in interactive diagrams; provides online analysis tools | 15 149 proteins, and 144 917 ligands |
| [Guide to Pharmacology](https://www.guidetopharmacology.org/) | IUPHAR/BPS guide to pharmacology is an open-access, expert-curated database of molecular interactions between ligands and their targets. | 2937 targets, and 9859 ligands |
| [GLASS](https://zhanglab.ccmb.med.umich.edu/GLASS/) | GPCR-Ligand Association (GLASS)   database is a manually curated repository for experimentally validated   GPCR–ligand interactions; along with relevant GPCR and chemical information,   GPCRligand association data are extracted and integrated into GLASS from literature and public databases | ~ 277 651 unique ligands and 3048 GPCRs |

## Biological data from assay screening, metabolism, and efficacy studies

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [HMDB](https://hmdb.ca/about) | Freely available electronic database containing detailed information about small-molecule metabolites found in the human body | 114 162 metabolite entries |
| [SMPDB](http://smpdb.ca/) | Small Molecule Pathway Database (SMPDB) is an interactive, visual database containing >30 000 small-molecule pathways found in humans only | >30 000 small-molecule pathways |
| [TTD](http://db.idrblab.net/ttd/) | Therapeutic Target Database (TTD) is a database providing information about known and explored therapeutic protein and nucleic acid targets,   targeted disease, pathway information and corresponding drugs directed at each of these targets | 2589 targets, and 31 614 drugs |
| [BioCyc](https://biocyc.org/) | Collection of 7615 pathway/genome databases; each database in BioCyc collection describes genome and metabolic pathways of a single organism | 7615 pathway/genome databases |
| [BiGG](http://bigg.ucsd.edu/) | Metabolic reconstruction of human metabolism designed for systems biology simulation and metabolic flux balance modelling | 2004 proteins, 2766 metabolites, and 3311 metabolic and transport reactions |
| [BRENDA](https://www.brenda-enzymes.org/) | Main collection of enzyme functional data available to scientific   community | At least 40 000 different enzymes from >6900 different organisms |
| [Reactome](https://reactome.org/) | Curated, peer-reviewed knowledgebase of biological pathways, including metabolic pathways, and protein trafficking and signalling pathways | >9600 proteins, 9800 reactions, and 2000 pathways for humans |
| [BioModels Database](https://www.ebi.ac.uk/biomodels/) | Repository of computational models of biological processes; models   described from literature are manually curated and enriched with   cross-references | 6753 patient-derived genome-scale metabolic models, 112 898 metabolic   models etc. |
| [KEGG](https://www.genome.jp/kegg) | Database resource that integrates genomic, chemical, and systemic functional information | 18 652 metabolites |
| [CARLSBAD](http://carlsbad.health.unm.edu/carlsbad/?mode=home) | Database and knowledge inference system that integrates multiple bioactivity data sets to provide researchers with novel capabilities for mining and exploration of available structure-activity relationships (SAR)   throughout chemical biology space. | 932 852 CARLSBAD activities, 890 323 unique structure–target pairs, 3542   targets, 435 343 unique structures |
| [WOMBAT](http://dud.docking.org/wombat/) | Contains 331 872 entries, representing 1966 unique targets, with bioactivity annotations | 268 246 unique structures |
| [Open NCI Database](https://cactus.nci.nih.gov/ncidb2.2/) | Maintained by the National Cancer Institute; contains small-molecule information such as names, biological activities, and structures; a useful resource for researchers working in cancer/AIDS fields | >250 000 compounds |
| [NPACT](http://crdd.osdd.net/raghava/npact/) | Provides information on the plant-derived natural compounds, including structure, properties (physical, elemental, and topological), cancer type,   cell lines, inhibitory values (IC50, ED50, EC50, GI50), molecular targets,   commercial suppliers, and drug-likeness of compounds | 1574 entries |
| [PKPB_DB](https://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=204443) | Contains physiological parameter values for humans from early childhood through senescence; intended to be used in physiologically based (PB) PK modelling; also contains similar data for animals (primarily rodents) | N/A |

## Drug liabilities and toxicities

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [T3DB](http://www.t3db.ca/) | Unique bioinformatics resource that combines detailed toxin data with comprehensive toxin target information | 3678 toxins |
| [DrugMatrix](https://ntp.niehs.nih.gov/data/drugmatrix/) | One of world’s largest toxicogenomic reference resources | ~ 600 drug molecules and 10 000 genes |
| [ACToR](https://actor.epa.gov/actor/home.xhtml) | Includes computational toxicology information about compounds, including   HTS, chemical exposure, sustainable chemistry (chemical structures and physicochemical properties) and virtual tissue data | >500 000 chemicals |
| [SkinSensDB](https://cwtung.kmu.edu.tw/skinsensdb/) | Contains curated data from published AOP-related skin sensitization   assays | 710 unique chemicals |
| [SIDER](http://sideeffects.embl.de/download/) | Contains information on marketed medicines and their recorded adverse drug reactions, including frequency, drug and adverse effect classifications | 1430 drugs with 5868 side effect information |
| [LTKB Benchmark Dataset](https://www.fda.gov/science-research) | Contains drugs with potential to cause drug-induced liver injury in humans; established using FDA-approved prescription drug labels;   liver-toxicity-knowledge-base-ltkb/ltkb-benchmark-dataset | 287 prescription drugs |
| [CTD](http://ctdbase.org/) | Comparative Toxicogenomics Database (CTD) is a premier public resource for literature-based, manually curated associations between chemicals, gene products, phenotypes, diseases, and environmental exposures | 13 378 unique chemicals and related information |

## Clinical databases

| Database | Description | Size (as of 29 October 2019) |
|:---:|:---:|:---:|
| [ClinicalTrials.gov](https://clinicaltrials.gov/) | Database of privately and publicly funded clinical studies conducted around the world | ~ 324 429 research studies in all 50 US states and 209 countries |
| [AACT database](https://aact.ctti-clinicaltrials.org/) | Publicly available relational database that contains all information   (protocol and result data elements) about every study registered in   ClinicalTrials.gov. Content is downloaded from ClinicalTrials.gov daily and loaded into AACT | ~ 324 429 research studies in all 50 US states and 209 countries |
| [EORTC Clinical Trials Database](https://www.eortc.org/clinical-trials-database/) | Contains information about EORTC clinical trials and clinical trials from other organizations with EORTC participation | N/A |
| [Exposome-Explorer](http://exposome-explorer.iarc.fr/) | Contains detailed information on the nature of biomarkers, populations and subjects where measured, samples analyzed, methods used for biomarker analyses, concentrations in biospecimens, correlations with external exposure measurements, and biological reproducibility over time | 908 biomarkers |
| [PharmaGKB](https://www.pharmgkb.org/) | A pharmacogenomics knowledge resource that encompasses clinical information about drug molecules | 733 drugs with their clinical information |


# Data management in chemistry using Galaxy

## Chemical file formats

Depending on the type of analysis you are going to perform, you will work with different file formats. Below are the most important data types, commonly used in cheminformatics, that you will likely see in Galaxy:
- SDF (.sd, .sdf) - stands for Structure-Data Format, and SDF files wrap the molfile (MDL Molfile) format. Stores information about the chemical structure and associated data of compounds in plain text. Files in SDF format can encode single or multiple molecules that are then delimited by lines consisting of four dollar signs. SDF files are formatted ASCII files that store information about the positions of the individual atoms (either in 2D or 3D space) that make up the molecule. The data on connectivity and hybridization state are also encoded, although their use is less frequent and often inconsistent.
- MOL (.mol) - an MDL Molfile for holding information about the atoms, bonds, connectivity and coordinates of a molecule. 
- MOL2 (.mol2) - a Tripos Mol2 file is a complete, portable representation of a SYBYL molecule. It is an ASCII file which contains all the information needed to
reconstruct a SYBYL molecule
- CML (.cml) - Chemical Markup Language (ChemML or CML) is an approach to managing molecular information using tools such as XML and Java. It supports a wide range of chemical concepts including molecules, reactions, spectra and analytical data, computational chemistry, chemical crystallography and materials
- InChI (IUPAC International Chemical Identifier) - a textual identifier for chemical substances, designed to provide a standard way to encode molecular information. The identifiers describe chemical substances in terms of layers of information — the atoms and their bond connectivity, tautomeric information, isotope information, stereochemistry, and electronic charge information ({% cite Heller2015 %})
- SMILES (.smi) - the **s**implified **m**olecular-**i**nput **l**ine-**e**ntry **s**ystem (SMILES) is a specification in the form of a line notation for describing the structure of chemical species using short ASCII strings. A linear text format which can describe the connectivity and chirality of a molecule ({% cite Weininger1988 %}) 
- PDB - the Protein Data Bank (PDB) file format is a textual file format describing the three-dimensional structures of molecules held in the [Protein Data Bank](https://www.rcsb.org/), now succeeded by the mmCIF format. It contains a description and annotation of protein and nucleic acid structures including atomic coordinates, secondary structure assignments, as well as atomic connectivity. In addition, experimental metadata is stored. ({% cite Berman2007 %})
- GRO (.gro) - a plain text file storing spatial coordinates and velocities (if available) of atoms during a molecular dynamics simulation, utilised by [GROMACS](https://manual.gromacs.org/archive/5.0.3/online/gro.html)

To visualise the structures held by the files with the positions of the individual atoms, you can use (NGL Viewer)[https://nglviewer.org/ngl/].

If you plan to work with molecular dynamics simulations, there are also some MD trajectory file formats that you might want to get familiar with. In Galaxy, you can convert between xtc, trr, dcd and netcdf files using a tool called {% tool [MDTraj file converter](toolshed.g2.bx.psu.edu/repos/chemteam/md_converter/md_converter/1.9.6+galaxy0) %}.

## Galaxy tools 

In Galaxy Chemical Toolbox there are dozens of tools that can be used for various analyses. Below we will only show a few, mostly related to data import, format conversion and some functions linked to what was discussed previously. 

Let's start with importing publicly available data. [Protein Data Bank](https://www.rcsb.org/) stores thousands of three-dimensional structural data of proteins nucleic acids and other biological molecules.

> <hands-on-title>Import PDB file from Protein Data Bank</hands-on-title>
>
> 1. Open [Protein Data Bank](https://www.rcsb.org/)
> 2. Type your search term in the box *"Enter search term(s), Entry ID(s), or sequence"*: `MAO B`
> ![A screenshot showing a search box in the upper right corner of the Protein Data Bank website.]({% link topics/computational-chemistry/images/pdb_search.jpg %} "Search box of the Protein Data Bank")
>3. Choose the structure of interest and copy its accession code: 2BK3
> ![A screenshot showing some of the resulting structures with their accession codes.]({% link topics/computational-chemistry/images/pdb_results.jpg %} "How to get accession code of the resulting structures.")
>4. Switch to Galaxy
>5. {% tool [Get PDB file](toolshed.g2.bx.psu.edu/repos/bgruening/get_pdb/get_pdb/0.1.0) %} with the following parameters:
>    - *"PDB accession code"*: `2BK3`
>
{: .hands_on}

This is often the first step of docking studies that you can learn about in other tutorials, for example in [Protein-ligand docking tutorial]({% link topics/computational-chemistry/tutorials/cheminformatics/tutorial.md %}).

You can import any molecule to Galaxy using SMILES notation. Below we show an example with benzenesulfonyl chloride and ethylamine. 

> <hands-on-title>Import SMILES</hands-on-title>
>
> 1. Copy SMILES of your molecule(s) of interest. In this example, we use benzenesulfonyl chloride (SMILES: C1=CC=C(C=C1)S(=O)(=O)Cl) and ethylamine (SMILES: CCN).
> 2. Click on the **{% icon galaxy-upload %} Upload data** button in the tools panel on the left-hand side.
> 3. Click on the **{% icon galaxy-wf-edit %} Paste/Fetch data** button twice. Two boxes will appear.
> 4. In the first box, under **"Name"** section, enter `Benzenesulfonyl chloride` and under **"Type"**: `smi`
> 5. Below paste the SMILES of benzenesulfonyl chloride: `C1=CC=C(C=C1)S(=O)(=O)Cl`
> 6. In the second box, under **"Name"** section, enter `Ethylamine` and under **"Type"**: `smi`
> 7. Below paste the SMILES of ethylamine: `CCN`
> 8. Click the **"Start"** button, then close the dialogue box. Your files are being added to your history!
>    
{: .hands_on}

As you learned from the previous section, there are many different formats used in computational chemistry. Of course, Galaxy allows you to interconvert between them - just have a look below!

> <hands-on-title>Convert the file format</hands-on-title>
>
> 1. {% tool [Compound conversion](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_compound_convert/openbabel_compound_convert/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-files %} *"Molecular input file"*: click on {% icon param-files %} *Multiple datasets* icon and choose both `Benzenesulfonyl chloride` and `Ethylamine`
>    - *"Output format"* - you have many options to choose from! Not only the formats but also the associated parameters. Just scroll the list and pick the one that is relevant to your downstream analysis. Here we use `MDL MOL format (sdf, mol)`
> 2. Rename {% icon galaxy-pencil %} the corresponding files `Benzenesulfonyl chloride SDF` and `Ethylamine SDF`. Before renaming the files, make sure to check either the starting dataset numbers or the atoms in the file in order not to confuse the two files! 
>
{: .hands_on}

We intentionally chose to work on benzenesulfonyl chloride and ethylamine. Do you know why? Well, if you think about the synthesis of sulfonamides, that's exactly what you need - a sulfonyl chloride and an amine. So... let's do some computational synthesis! 

> <hands-on-title>Run the reaction</hands-on-title>
>
> 1. {% tool [Reaction maker](toolshed.g2.bx.psu.edu/repos/bgruening/ctb_im_rxn_maker/ctb_im_rxn_maker/1.1.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Input file"*: `Benzenesulfonyl chloride SDF`
>    - *"Reagent options"*: Sulfonamide
>    - {% icon param-file %} *"Reagent file"*: `Ethylamine SDF`
>
{: .hands_on}

> <tip-title> Reaction maker tool works also with SMILES!</tip-title>
>
> Above we converted the *.smi* file into *.sdf* to show how {% icon tool %} **Compound conversion** tool works. However, it is worth pointing out that {% icon tool %} **Reaction maker** works also with SMILES files and the conversion happens automatically so that your *.smi* dataset is imported as *.sdf* input.
>
{: .tip} 

> <details-title>Which reactions I can run using this tool?</details-title>
>
> {% icon tool %} **Reaction maker** works with more reaction types, such as:
> - Amides synthesis
> - Nucleophilic aromatic substitution (SNAr mechanism)
> - Urea
> - Suzuki coupling
> - Sulfonamide
> - Reductive amination
> - N-alkylation
> - Ether coupling
> - Ester coupling
> - Benzimidazole
> - Triazole
> - Benzoxazole
> - Sarah Cu
> - Sarah Quat Am
>
{: .details}


The output is the *.txt* log file and the SDF file with the product molecule. If you have a look at that file, you'll see that there is just a list of atoms with their corresponding coordinates. To draw the resulting structure, we can use another tool.

> <hands-on-title>Visualisation of compounds</hands-on-title>
>
> 1. {% tool [Visualisation](toolshed.g2.bx.psu.edu/repos/bgruening/openbabel_svg_depiction/openbabel_svg_depiction/3.1.1+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecular input file"*: `SDF output for Reaction maker`
>    - *"Property to display under the molecule"* - there are many properties that you can choose from! Please note that sometimes if not enough information is provided, then the property might not be displayed. Let's choose a basic but useful property: `Molecular weight`
>    - *"Format of the resulting picture"*: `SVG`
>
{: .hands_on}

Since we spoke about the drug-likeness in the previous section, let's see how it works in practice, on the example of our "synthesised" molecule.

> <hands-on-title>Estimate the drug-likeness</hands-on-title>
>
> 1. {% tool [Drug-likeness](toolshed.g2.bx.psu.edu/repos/bgruening/qed/ctb_silicos_qed/2021.03.4+galaxy0) %} with the following parameters:
>    - {% icon param-file %} *"Molecule data in SDF or SMILES format"*: `SDF output for Reaction maker`
>    - *"Method"* - two possible methods to weight the features are available - one based on max. weight (QEDw,max), and the other on mean weight (QEDw,mo). There is also an option to leave features unweighted (QEDw,u) and we'll choose this one: `unweighted (QEDw,u)`
>    - *"Include the descriptor names as header"*: {% icon param-toggle %} `Yes`
>
{: .hands_on}

> <details-title>What are the assessed properties?</details-title>
>
> The eight properties used are molecular weight (MW), octanol–water partition coefficient (ALOGP), number of hydrogen bond donors (HBDs), number of hydrogen bond acceptors (HBAs), molecular polar surface area (PSA), number of rotatable bonds (ROTBs), number of aromatic rings (AROMs) and number of structural alerts (ALERTS).
>
{: .details}

In medicinal chemistry, we often base the new structures on scaffolds of existing drugs. Therefore, it is quite useful to be able to find similar structures - below is an example of how to do it in Galaxy.

> <hands-on-title>Search ChEMBL database for similar compounds</hands-on-title>
>
> 1. {% tool [Search ChEMBL database](toolshed.g2.bx.psu.edu/repos/bgruening/chembl/chembl/0.10.1+galaxy4) %} with the following parameters:
>    - *"SMILES input type"*: `File`
>    - {% icon param-file %} *"Input file"*: `Benzenesulfonyl chloride`
>    - *"Search type"*: `Substructure`
> 2. Rename {% icon galaxy-pencil %} the output file `Benzenesulfonyl chloride substructures`
>    
{: .hands_on}


> <tip-title>Search ChEMBL database tool works also with SDF!</tip-title>
>
> Even though the input format of the above tool is SMILES, it can automatically interconvert between *.sdf* and *.smi* formats, so you don't have to use {% icon tool %} **Compound conversion** tool beforehand. 
> 
{: .tip} 

> <details-title>Parameters of Search ChEMBL database tool</details-title>
>
>  {% icon tool %} **Search ChEMBL database** tool allows for searching the ChEMBL database for compounds which resemble a SMILES string. Two search options are possible:
> - similarity (searches for compounds which are similar to the input within a specified Tanimoto cutoff)
> - substructure (searches for compounds which contain the input substructure)
>   
> Results can be filtered for compounds which are
> - approved drugs
> - biotherapeutic
> - natural products
> - fulfil all of the Lipinski rule of five criteria
>
{: .details}

Let's repeat this step, but with Lipinski's Rule of Five which you learned about in the previous section of this tutorial!

> <hands-on-title>Search ChEMBL database for similar compounds using the Lipinski rule of five</hands-on-title>
>
> 1. Click the {% icon galaxy-refresh %} **Repeat** button on the previous dataset.
> 2. You will be redirected to {% tool [Search ChEMBL database](toolshed.g2.bx.psu.edu/repos/bgruening/chembl/chembl/0.10.1+galaxy4) %}. Leave all the parameters as before, except for one:
>    - *"Filter for Lipinski's Rule of Five"*: {% icon param-toggle %} `Yes`
> 2. Rename {% icon galaxy-pencil %} the output file `Benzenesulfonyl chloride Lipinski substructures`
>
{: .hands_on}


> <question-title></question-title>
>
> How many molecules have been filtered out?
>
> > <solution-title></solution-title>
> >
> > If you click on the dataset, you will see a short summary of how many molecules have been found.
> > When we didn't apply any filters, the tool found 45 molecules. After applying Lipinski's Rule of Five filter, 36 molecules satisfied the rules. 
> > ![Image showing the preview of the output datasets from searching benzenesulfonyl chloride substructures in the ChEMBL database, with and without Lipinski's Rule of Five filter.]({% link topics/computational-chemistry/images/substructures.png %} "Preview showing the number of molecules found in the ChEMBL database, before and after applying Lipinski's Rule of Five filter")
> >
> {: .solution}
>
{: .question}


## Galaxy workflow summary

Those are just a few tools, but they offer many parameters that you can tune depending on your analysis. Additionally, we provide you with an [example history](https://usegalaxy.eu/u/j.jakiela/h/data-management-in-medicinal-chemistry-tutorial) and the dedicated [workflow](https://usegalaxy.eu/u/j.jakiela/w/data-management-in-medicinal-chemistry-workflow) of this small tutorial. You are more than welcome to explore other tools in the Galaxy Chemical Toolbox though!

# Data-driven medicinal chemistry - yesterday, today and tomorrow

## Why do we need big data in med-chem?
In the article by {% cite Lusher2014 %} the role of the medicinal chemist is to make decisions about which of the infinite possibilities of new compounds should be made next. As the amount and variety of data on which to base these decisions grows, so too must the data analysis skills of the medicinal chemist. The authors claim that modern medicinal chemists should be able to recognise sources of relevant information, prepare raw data, use statistical tools, extract meaningful information, interpret results, recognise potential problems and make visualisations to communicate their findings to improve the quality of compounds being produced in research. According to the authors, this situation will require improved education and increased access to data and information management tools. This is where Galaxy comes in, offering not only a platform for data analysis with a plethora of the most commonly used tools in the field but also educational materials allowing anyone to learn and excel in analysis. 
 
The privilege of applying computational methods in medicinal chemistry pipelines allows to focus on the most promising compounds and to remove unsuitable ones before the stage of chemical synthesis. Not only is this approach more efficient but also more sustainable and can reduce the costs of the synthetic stage of drug discovery ({% cite Brown2018 %}).

Another important role that big data plays in drug discovery is target identification (and validation) which might be currently one the biggest challenges in medicinal chemistry. By introducing genomic data in drug discovery, we can produce more specific and effective medicines. Understanding the underlying causes of the disease and the biological targets are crucial in designing new drugs and hence should be inherently linked with this process, where possible. 

## Data challenges 
It is important to understand the limitations of the databases we use since if data is not well understood, the processing and analysis drawn may ultimately be flawed, following the ‘garbage in, garbage out’ principle. Here are some challenges mentioned in {% cite Brown2018 %} that you can explore more in-depth by referring to that article.
-	Errors
-	Reproducibility
-	Standardisation
-	Formatting of data
-	Information held in silos
-	Loss of contextual information
-	Limited and biased data

## Machine learning in med-chem
Given the huge number of molecules in the various databases, applying machine learning in cheminformatics and building predictive models of the physiochemical properties of molecules is more and more popular. It helps to predict absorption, distribution, metabolism, excretion, toxicity (ADMET), likelihood of interaction with the drug targets as well as off-target effects. It can be useful in elucidating complex protein-protein or drug-drug interaction networks, in understanding structure-activity relationships (SARs) and when combined with data from the ‘omics’ revolution. This approach also helps to identify bioisosteres and introduce the idea of scaffold hopping in molecular design ({% cite Brown2018 %}).

<!---
TO FURTHER IMPROVE THE TUTORIAL:
expand this section, include examples
-->

## Knowledge sharing
It is not only about gathering and managing the data but also building a culture of knowledge sharing. We work in big, international teams with well-established methods and well-equipped labs. There are so many opportunities for knowledge capture and exchange! The challenge emerges here though – how to efficiently connect and help each other? This is the question that many pharmaceutical companies try to answer, such as Merck & Co. who describe their approach to knowledge management and report on the multiple enduring and complementary teams and initiatives to capture and share knowledge ({% cite Beshore2022 %}).


topic: fair
tutorial: bioimage-metadata
title: FAIR Bioimage Metadata
difficulty: introductory
time_estimation: 15m
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["Data management", "Metadata", "Bioimaging"]
content_hash: 181e376092e598178ca87fbe67ba0298
last_modified: 2026-02-26T21:12:15.931800
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/bioimage-metadata/tutorial.html

content:
# FAIR Bioimaging

Submitting your data to a repository is a good way to make the data FAIR. This will make it:
- **F**indable, as the data will be given specific identifiers
- **A**ccessible, as the data will be available online, open and free where possible
-	**I**nteroperable, as the repository will often enforce the use of formalised, consistent language
-	**R**eusable, as the data will be released under a license with detailed provenance

# Examples of bioimage data repositories

But the question remains: where can I submit my data? Currently the main repositories where you can submit your images are: BioImage Archive, Electron Microscopy Public Image Archive (EMPIAR) and Image Data Repository (IDR). Let's have a look at the questions below to explore those repositories more in depth!

> <question-title></question-title>
>
> Listed below are three examples of Bioimage Data Repositories:
> - [IDR: Image Data Repository](https://idr.openmicroscopy.org/)
> - [EMPIAR: Electron Microscopy Public Image Archive](https://www.ebi.ac.uk/empiar/)
> - [BioImage Archive](https://www.ebi.ac.uk/bioimage-archive/)
>
> Visit their websites and find out what their scope is or what sorts of datasets they accept.
>
> > <solution-title></solution-title>
> >
> > - **IDR: Image Data Repository**: Curated datasets of cell and tissue microscopy images
> > - **EMPIAR: Electron Microscopy Public Image Archive**: Cryo-EM, Scanning Electron Microscopy, Soft X-ray tomography
> > - **BioImage Archive**: Everything else and some overlap with IDR and EMPIAR
> {: .solution}
>
{: .question}

> <tip-title>Repositories everywhere</tip-title>
>
> As well as these repositories, your Institute may have their own repository. For example, at the Warwick University, there is also [OMERO](https://warwick.ac.uk/fac/sci/med/research/biomedical/facilities/camdu/training/omero-warwick-guide_2.pdf) and [WRAP](https://wrap.warwick.ac.uk/).
>
{: .tip}


> <comment-title></comment-title>
> The repositories we are looking at in this course are for bioimage data, not medical data. There are other specialist repositories available if you have medical data.
{: .comment}

# Things to consider when choosing a repository

Now we know what repositories are available, but how to decide which one is best given the files we want submit? Try to work through the below Question box and find the answer!

> <question-title></question-title>
>
> Choose one repository from above and look through its documentation. Try to find:
> 1. What data formats are accepted?
> 2. What license is recommended to publish the data?
> 3. Are there specific instructions for large datasets?
>
> > <solution-title></solution-title>
> >
> > - **IDR: Image Data Repository**:
> >   1. The IDR uses the Bio-Formats library for reading imaging data. Bio-Formats supports over 150 proprietary and open file formats (see the [full list](https://bio-formats.readthedocs.io/en/stable/supported-formats.html)).
> >   2. It is strongly recommended that submitters make their datasets available under [CC-BY](https://creativecommons.org/licenses/by/4.0/) license.
> >   3. As specified on the [IDR website](https://idr.openmicroscopy.org/about/submission.html), dataset size is typically not an issue, but for sizes significantly larger than 1000 GB special planning may be needed.
> > - **EMPIAR: Electron Microscopy Public Image Archive**:
> >   1. Provide image data in the formats in which they are uploaded, but recommended is the use of common formats in the field including MRC, MRCS, TIFF, DM4, IMAGIC, SPIDER, MRC FEI, RAW FEI and BIG DATA VIEWER HDF5.
> >   2. All data in EMPIAR is freely and publicly available to the global community under the [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) license.
> >   3. As specified on the [EMPIAR page](https://www.ebi.ac.uk/empiar/deposition/manual/#manIntro), typically having more than 4000 files in a directory has a tendency to slow down access considerably. It is recommended in this case to sub-divide the directory into subdirectories with no more than 4000 files each. If you have a single file larger than 1 TB, contact EMPAIR in advance.
> >      To find out more, check the [FAQ page](https://www.ebi.ac.uk/empiar/faq).
> > - **BioImage Archive**:
> >   1. The BioImage Archive accepts all image data formats, although formats readable by [Bio-Formats library](https://bio-formats.readthedocs.io/en/stable/supported-formats.html), are preferable.
> >   2. According to [BioImage Archive Policies](https://www.ebi.ac.uk/bioimage-archive/help-policies/), all new data directly submitted to the BioImage Archive will be made available under a [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) licence, datasets brokered/imported from other resources may have other licenses though.
> >   3. There are different submission methods depending on data size:
> >      - Less than 50 GB total size, less than 20GB per file – use submission tool
> >      - Up to 1TB total size – use FTP
> >      - Anything larger – use Aspera
> >
> >        To find out more, check the [FAQ page](https://www.ebi.ac.uk/bioimage-archive/help-faq/).
> {: .solution}
>
{: .question}


# What metadata to collect

Whichever repository you choose, you will be required to upload some metadata along with your data. In an ideal world, you would remember everything about your data when you submit it. In reality, this is unlikely, the data could have been collected over a long time period or by different people. To overcome these challenges, it is best to collect metadata alongside imaging experiments, don’t leave it all to the end!
However, this raises further challenges. At the time of data acquisition, you probably won’t know which repository you will submit it to, what the study results will be, or even who will be the target audience for the data. So what metadata do you need to collect?

Currently, there is no standard for bioimages, so here is the general outline how to proceed:
- If you have chosen a repository, use their template/guidelines.
- Otherwise, use [REMBI](https://www.nature.com/articles/s41592-021-01166-8). These are published guidelines which are explained in detail in the [REMBI tutorial]({% link topics/fair/tutorials/bioimage-REMBI/tutorial.md %}). REMBI is useful as it should cover most of the metadata requirements of the repositories, even if you haven’t decided which one you want to use yet.
- For medical images, see the [DICOM](https://www.dicomstandard.org/) standard.

# How to store metadata

Metadata should be stored somewhere that it can be viewed and edited by collaborators. This helps everyone to stay on the same page with regard to what data should and has been collected. This is also useful if different people contribute to different aspects of the image acquisition, e.g. maybe one person prepared the sample, another imaged it under the microscope, and a third person did the post-processing. Each person can then update the metadata related to their part of the study.

Recommendations for storing metadata:
- If a repository has been chosen, use their template (if provided).
- Use a Delimited text file format, e.g. .csv, .tsv. You can use spreadsheet software and save to this format. Try to use a plain format, e.g. avoid merged/split cells.
-  Use data management software suitable for your data, e.g. [OMERO](https://www.openmicroscopy.org/omero/).

# Further steps
{% icon congratulations %} Congratulations on successfully completing this tutorial! If you want to know more about FAIR data management, we provide training that you can find in the [FAIR Data, Workflows, and Research]({% link topics/fair %}) section on GTN. For more detials on FAIR data management in bioimaging, see the [REMBI tutorial]({% link topics/fair/tutorials/bioimage-REMBI/tutorial.md %}), and if you want to dive into the imaging analysis straight away, feel free to choose one of the Galaxy [tutorials]({% link topics/imaging %})! Additionally, [Global BioImaging](https://globalbioimaging.org/) offers [training courses](https://globalbioimaging.org/international-training-courses/repository/image-data) dedicated to image data management, sharing, reuse, and image data repositories that were mentioned in this tutorial.


topic: fair
tutorial: zenodo-in-galaxy
title: Uploading Data to Zenodo from Galaxy
difficulty: intermediate
time_estimation: 30m
tools_json: []
requirements_json: []
tags_json: ["fair", "zenodo"]
content_hash: 6c1d2214b3bf56a129088a45faa83350
last_modified: 2026-02-26T21:12:15.986211
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/zenodo-in-galaxy/tutorial.html

content:
Discover a more streamlined approach to research data management with Galaxy's integration with Zenodo.

> <agenda-title></agenda-title>
> In this tutorial we will deal with:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Introduction

Zenodo can be connected to Galaxy, offering a streamlined experience in managing and analyzing your research data. With this integration, you can export research results directly from Galaxy to Zenodo, and import files from Zenodo into Galaxy for reproducible analysis workflows.

## Zenodo

<img style="float: right; margin-left: 1rem;" alt="Zenodo Logo" src="../../images/zenodo-gradient-logo.svg" />

[Zenodo](https://zenodo.org/) is an open repository for all scholarship, enabling researchers from all disciplines to share and preserve their research outputs, regardless of size or format. Free to upload and free to access, Zenodo makes scientific outputs of all kinds citable, shareable and discoverable for the long term.

It's worth noting that [Zenodo, in October 2023, migrated to InvenioRDM as its underlying technical platform](https://blog.zenodo.org/2023/10/13/2023-10-13-zenodo-rdm/). This move signals a commitment to enhanced features and scalability, further bolstering the integration's potential benefits for researchers.

# Getting started

## Setting up your PAT (Personal Access Token)

To be able to upload files and browse protected records, you need to create an account and set up your PAT (Personal Access Token). To create a new token:

> <hands-on-title>Create Your Personal Access Token</hands-on-title>
>
> 1. Open [Zenodo](https://zenodo.org) and `Log in` or create an account.
> 
> 2. Go to your `user settings` in the top right corner and select the `Applications` tab.
>
> 3. Then, click on `New token` and give it a name and the necessary permissions.
>
>    ![Zenodo PAT](../../images/production-zenodo-pat.png)
>
> > <comment-title></comment-title>
> > You will only be able to view and copy the access token when it is first created, so you should copy and securely store the token at this time. If you lose your copy of the access token, then you will need to generate a new one.
> >
> {: .comment}
> 
{: .hands_on}

## How to connect Zenodo to Galaxy 25.0

The new `Manage Your Repositories` section is available under `User → Preferences`. Here’s how it works:

> <hands-on-title>Connecting to Galaxy</hands-on-title>
> 
> 1. Navigate to `User → Preferences → Manage Your Repositories`. If you haven't set up any integrations yet, you'll see an empty list.
> ![Manage Your Remote File Sources](../../images/1-empty-file-sources-list.png)
>
> 2. Click `+ Create` to configure a new integration. You'll see a list of available integrations, including `S3, Dropbox, InvenioRDM, Zenodo`, and more (depending on your Galaxy server).
> ![Create New Integration](../../images/2-create-new.png)
> 
> 3. Select `Zenodo` to configure it and enter your **credentials or relevant connection details**. (This will be the access token created in the previous section if using the Zenodo sandbox.)
> ![Configure Zenodo](../../images/3-zenodo-template-filled.png)
> 
> 4. Once set up, the Zenodo integration, in this case named `My Zenodo`, will appear in the list where you can manage or delete it.
> ![Zenodo Integration](../../images/4-instance-created-zenodo.png)
> 
> 5. The Zenodo integration, along with any other integrations that you have set up, will appear first when browsing import/export locations. You can find them by clicking on the `Upload` button in the activity bar.
> ![Browse Remote Files](../../images/5-upload-import-zenodo.png)
{: .hands_on}


> <details-title>Do you want to add a different repository to Galaxy?</details-title>
> 
> There are a number of additional repository types that you can connect to Galaxy and use to import and export data.
>
> {% snippet faqs/galaxy/manage_your_repositories.md %}
>
{: .details}

# Importing records and files into Galaxy

Once you have connect Zenodo to Galaxy, you will be able to import records and files stored in Zenodo into Galaxy.

> <hands-on-title>Importing Files</hands-on-title>
> 
> 1. Open the Upload tool, and select `Choose remote files`
> ![Upload Tool](../../images/upload-tool.png)
>
> 2. Search for the Zenodo instance. Remember that this will only appear if the Zenodo plugin is configured to connect to Zenodo in your Galaxy instance.
> ![Choose Remote Files](../../images/choose-remote-files-zenodo.png)
>
{: .hands_on}

Once you have selected Zenodo, you can browse public records and import them into your Galaxy history. You can choose to import the full record or individual files in the same way you would import files from any other remote source.

# Exporting your Galaxy history

You can export your history directly to Zenodo. A benefit of publishing your history on Zenodo is that Zenodo will generate a DOI for your history. You can then include this DOI in your publication and it can be used by other researchers to cite your work.

> <hands-on-title>Exporting a History</hands-on-title>
> 
> 1. From the history menu {% icon galaxy-history-options %}
, you can select `Export History to File`
>
> 2. Then choose `to My Zenodo`. 
> 
> 3. You can decide whether to create a new record or upload the history to an existing draft record.
> ![Export History to Zenodo](../../images/export-history-zenodo.png)
>
{: .hands_on}

You can always edit the record metadata directly from the Zenodo web interface. Once you are satisfied with the record, you can publish it, generating a DOI that others can use to cite your research.

# Conclusion

In this tutorial, we have connected Zenodo to a Galaxy instance. We have also imported files and records from Zenodo into Galaxy and exported a Galaxy History to Zenodo. 

To see how to connect any InvenioRDM-compatible respository to Galaxy, see the tutorial, [Integrating InvenioRDM-compatible Repositories with Galaxy]({% link topics/fair/tutorials/inveniordm-integration/tutorial.md %}).

# References

1. [InvenioRDM and Zenodo integration in Galaxy](https://galaxyproject.org/news/2024-05-03-inveniordm-integration/)
2. [Updates to your InvenioRDM and Zenodo integration in Galaxy
   ](https://galaxyproject.org/news/2025-03-10-inveniordm-integration-update/)


topic: fair
tutorial: fair-clinical
title: Making clinical datasets FAIR
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "open", "data stewardship"]
content_hash: 4c9c75304f70a747132ee604a17c1320
last_modified: 2026-02-26T21:12:15.967691
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-clinical/tutorial.html

content:
The life science community is generally very good at sharing omics data on platforms such as [GEO](https://www.ncbi.nlm.nih.gov/geo/) and [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress).  However, the metadata and clinical data associated with the omics datasets are often incredibly sparse.  Tracking down the meta- and clinical data of omics datasets can be time-consuming for both data owner and researcher and is a significant hurdle when trying to explore enriched omics data.

In this tutorial, you will learn the benefits of making clinical datasets FAIR (Findable, Accessible, Interoperable, and Reusable) and how to make them FAIR.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Clinical datasets

> <hands-on-title>How would you describe what are clinical datasets? - {% icon time %} 2 minutes - Silent reflection</hands-on-title>
>
> * consider in general terms
> * consider more specifically
>   
{: .hands_on}

In general, the most common clinical data used in research are clinical reports (from trials and observational studies) and individual patient data.  In this section, we will explore some of the different types of clinical data.

## Metadata that accompanies transcriptomic data
In GEO and ArrayExpress the elements that we are describing as clinical data are the Characteristics or Source Characteristics.

> <hands-on-title>Explore GEO and ArrayExpress for characteristics - {% icon time %} 5 minutes</hands-on-title>
>
> 1. Click on the link below for ArrayExpress and open the section for Source Characteristics  
>    [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-12829 )
> 
> 2. Take a look at the Experimental Factors.  Notice that there are very few characteristics available for this dataset.
>
> 3. Click on the link below for GEO  
>    [GEO](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE65391)
>
> 4. Under the Samples section, click on one of the healthy controls.  Here you will see that this dataset has many characteristics.  Notice, however
>    that there is not much information gathered on the healthy controls
>
> 5. Click on one of the samples with systemic lupus erythematosus (SLE).  Notice here that there is more data collected on SLE subjects.
>
> 6. Explore ArrayExpress and/or GEO for datasets within the domain(s) that you are familiar with.  Take note of the availability of characteristic data for the different datasets
>
{: .hands_on}

## Interventional clinical trials
In these trials, usually, the participants have a condition or disease, and the research question is related to the effectiveness of a treatment.  Some of the participants are given the treatment(s) under investigation, and the other participants are given either a placebo or non-treatment drug.  Usually, these are under double-blind conditions, both participant and doctor do not know whether they are receiving/dispensing treatment or a placebo.  In addition, there may also be a set of control participants who do not have the condition or disease.  Data is usually collected at baseline or screening before any treatment or placebo is given, and then at least one time point in the future.  The breadth and type of data collected can vary across trials, and this can be limiting when trying to pool data from different trials.  The data collected could be minimal, with basic metadata to accompany the sample or imagining data.  But the data collection could also could be quite extensive, including quality of life questionnaires filled out by the patient.

An example of an interventional clinical trial is [TRACTISS](https://bmcmusculoskeletdisord.biomedcentral.com/articles/10.1186/1471-2474-15-21), where the aim of the study was to establish whether the drug rituximab improves clinical outcomes for Primary Sjögren's Syndrome patients.

## Observational clinical trials
In these trials, data is collected at least once, there may not be a follow-up collection.  The research question is usually related directly to the disease or condition, for example, understanding the biomarkers of a disease.  As with interventional clinical trials, the breadth of data collected varies on the trial.  There is often a control group but not always.  

[PRECISEADS](https://www.imi.europa.eu/projects-results/project-factsheets/precisesads) is an example of an observational clinical trial where the study's aim is molecular reclassification to find clinically useful biomarkers for systemic autoimmune diseases.


## Registries
Registries usually record (over a period of time), the disease progression of patients with the same disease or condition.  Often referred to a hub/collection point once the diagnosis is confirmed.  Patients are continually added to the registries and therefore registries can provide a large pool of patients with relatively similar data collected.  As these are registries of patients with a particular condition, there are no control participants.

Here are links to two registry projects.  Both websites describe the projects, the aims and the outputs of the studies.

[UK Sjögren's registry](https://www.sjogrensregistry.org/index.php)  
[UK JIA Biologics Registers](https://sites.manchester.ac.uk/bcrdbspar/) 

## Electronic health records (EHR)
These are generally the data collected when a person has contact with the health services. In England, this can be quite fragmented, as each service provider (i.e. GP, hospital, social care, etc.) collects data as an individual organisation, using one of a number of accredited packages for collecting and storing electronic health records.  

There are strict controls for researchers applying to access this data.  The research purpose must be approved before access is given, and the researchers usually receive data relevant to their research question.  In addition, the data received is usually anonymised to avoid the ability to identify individuals from the data.

[comment]: <> (This part may need rewording. My understanding is that EHRs are provided by: GPs where two main platforms are EMIS Web and TPP SystmOne and; NHS trusts with many different suppliers.  NHS England provides advice on accredited suppliers.)
[comment]: <> (SN - have rewritten most of the original paragraph and hope this covers the point above)

In the UK, Clinical Practice Research Datalink ([CPRD](https://cprd.com)) collects anonymised patient data from a network of GP practices links this data to a range of other health-related data and provides a longitudinal, representative UK population health dataset. 

The [Health Data Research Innovation Gateway](https://www.healthdatagateway.org/) can help researchers to find and discover UK health datasets, tools and other resources to further health research.

> <hands-on-title>Explore HDRUK Innovation Gateway for datasets - {% icon time %} 5 minutes</hands-on-title>
> 
> 1. Click on Datasets in the [Health Data Research Innovation Gateway](https://www.healthdatagateway.org/)
> 
> 2. In the filters, expand Publisher and in Search Filter type CPRD and enter  
> 
> 3. Select CPRD  
> 
> 4. Expand Phenotype and search for rheumatoid, and select Rheumatoid Arthritis  
> 
> 5. Take a look at the information about the two datasets  
> 
> 6. Clear the filters, and search for datasets related to your areas of interest  
>
{: .hands_on}

# The importance of making clinical datasets FAIR

## Associations between clinical features and omics
In some recent work in rheumatoid arthritis [The RA-MAP Consortium, 2022](https://www.nature.com/articles/s41597-022-01264-y), it was demonstrated that simple patient demographics such as sex, ethnicity and age are all drivers of expression variation in addition to disease activity.  Principal component driver plots highlighted critical associations between diverse clinical features and omics.  This shows how rich clinical information may be key to analysis in some, if not, many diseases.  

![Image of Unsupervised PCA Driver analysis of mRNA for whole blood, CD8 T-cells and CD14 Monocytes, showing that the clinical features Sex, Ethnicity and Age have some degree of association with principal components 1-5. ](../../images/RAMAP_graphic.png "Example of Unsupervised PCA Driver analysis of multi-omic compartments from the RA-MAP study, showing clinical features and their degree of association with Principal Components 1-5.  Image from: [RA-MAP, molecular immunologival landscapes in early rheumatoid arthritis and healthy vaccine recipients](https://www.nature.com/articles/s41597-022-01264-y#Tab1)")

> <hands-on-title>Do the datasets have all these characteristics - {% icon time %} 5 minutes</hands-on-title>
>
> For your domain, look at datasets on ArrayExpress and GEO and see how many of them have sex, age, ethnicity/race, and some measure of disease activity
>
{: .hands_on}

## Powering up cohorts for analyses and machine learning
For many diseases (or across a family of diseases), there is a need for well-powered cohorts with hundreds to thousands of participants, for analysis purposes.  FAIR principles have been widely promoted for omics datasets, but as shown, there are key challenges to meta-analysis.  This is illustrated by an example of collating 17 public rheumatoid arthritis studies, which was done as pre-analysis for the RA-MAP study.  The clinical data in most cases were sparse and incomplete.  For example, of the curated 17 studies, three did not include sex, 5 did not include age and only one study included ethnicity or race.  The limitations in the availability of clinical data substantially diminish the value of public data sets and on the surface to not adhere to FAIR principles.

[comment]: <> (Is there a link to the rheumatoid arthritis study example?)
[comment]: <> (SN - no link as this was pre-work before analysis, have changed the text to explain this)

Similarly, in order to make the most of machine learning techniques, datasets need to be reasonably large.  

## Time-saving
Usually, after finding potentially interesting datasets on GEO or ArrayExpress, the next step in trying to enrich these public datasets is to contact the PI of the study to request additional data.  From experience, the PI is usually happy to share additional data, especially if there have been previous collaborations.  However, even given the willingness to share, there is a time-consuming process of contacting the person who has the data and reaching agreement on what can and will be shared.  This at best will take weeks, but most likely months, especially if there is a lapse in communication between parties.

By implementing some of the techniques described in the next section can help release rich and FAIR data and metadata in a timely fashion. 

# Practical techniques to make clinical data FAIR
There are certain privacy concerns when making clinical data freely available through public libraries such as GEO and ArrayExpress. Data needs to be anonymised, or at least pseudo-anonymised.  

> <hands-on-title>What data would be problematic to share? - {% icon time %} 5 minutes</hands-on-title>  
>
> Take a look at the data below and consider which columns and data may be problematic to share  
> ![Image of a spreadsheet with the columns Variable Name, Data Item, Definition, Format, Validation Rule, Codes, Labels and Comments.  The spreadsheet gives examples of contents for each of these columns for ID, gender, age at onset, ethnicity and RF](../../images/Patient_data.png "An example of a data dictionary")
>
> > <solution-title></solution-title>  
> > * Remove columns C (FIRST), D (LAST), H (ADDRESS) and L (ZIP) as these are clear identifiers of the person  
> > * In terms of location, it is preferable to keep this as broad as possible.  In this case, keep column J (STATE) but drop columns I (CITY) and 
> > K (COUNTY).  It is debatable whether to keep them or not  
> > * Column B (BIRTHDATE) is another clear identifier but could be replaced by age  
> > * It is preferable not to keep dates (Columns (DATEOFASSESSMENT) and N (DATEOFDISEASEONSET))  
> >
> > We will discuss date handling in the next section  
> {: .solution}  
{: .hands_on}  

## Converting dates to time periods 
Avoid including dates and convert these into time periods.  For example, use date of birth and date of assessment to calculate Age (at assessment or baseline).  Similarly, calculate Age at Diagnosis, and Age at Onset, if dates are given.

If there are more than one visits, and if these are not within roughly defined time periods, for example, 4 weeks, or 12 weeks, calculate the number of days between the visits.

> <hands-on-title>Which columns would you change to time periods from dates? - {% icon time %} 10 minutes</hands-on-title>
> 
>  For the dataset below identify the column you would change to time periods and convert them to time periods
> ![Image of spreadsheet with columns ID, Birth Date, Race, Ethnicity, Gender, State, Date of Assessment and Date of Disease Onset as the columns, with 10 lines of data filled in below for each of these columns](../../images/Patient_dates.png "Patient data with dates")
>
> > <solution-title></solution-title>  
> > * Replace BIRTHDATE with AGE at DATEOFASSESSMENT  
> > * Replace DATEOFDISEASEONSET with AGEATONSET 
> > * Remove BIRTHDATE, DATEOFASSESSMENT and DATEOFDISEASEONSET
> > 
> > ![Image of spreadsheet with columns ID, Race, Ethnicity, Gender, State, Age and Age at Onset, with 10 lines of data filled in below for each of these columns](../../images/Datesremoved.png "Patient data with dates removed")  
> >
> {: .solution}  
{: .hands_on}  

## Comprehensive metadata for omics datasets
When publishing omics datasets to libraries such as GEO and ArrayExpress, consider at least including Age, Sex, Race/Ethnicity, and if possible some disease activity measure.  Publishing as much as possible is great for researchers to access enriched datasets, and also reduces fielding queries and requests to the PI.

## Data dictionaries
There are cases when PIs would rather not have the clinical datasets on free-to-access platforms.  In these situations, the clinical datasets can still be made FAIR.  By having clear documentation of the datasets, these documents can be published and attributed to the project or consortia.  For each individual clinical dataset, it is useful to have a data dictionary.

A data dictionary is the metadata of the dataset.  It should include details such as data type, value range or allowed values, any relationship to other data elements and their meaning or purpose.  The details of the data dictionary often originate from how the data was input either at the source or into the database.

Here’s an example of a data dictionary used on a project

![Image of an example of a data dictionary in a spreadsheet format](../../images/ExampleDataDict1.png "Example of a data dictionary")

> <tip-title>Key things to note</tip-title>  
>
> * The variable name is often the short form name of the data item  
> * The definition describes the data item more fully  
> * In the data dictionary example shown above, data items are encoded, and so their format is numeric, with constraints
> * The validation rule usually refers to instructions given when data was originally input, but it could be the calculation formula for the data item  
> * Codes and Labels here are the encoding definitions  
{: .tip}

> <hands-on-title>Create a data dictionary for the dataset below - {% icon time %} 10 minutes</hands-on-title>
>
> ![Image of a spreadsheet with columns ID, Race, Ethnicity, Gender, Age and Age at Onset, with 10 rows filled in below](../../images/DataDictionaryHandsOn.png "Create a data dictionary for this dataset")
>
> > <solution-title></solution-title>
> > Here is one example of a suitable data dictionary for the above dataset
> >
> > ![Image of a spreadsheet with the columns Variable Name, Data Item, Definition, Format, Validation Rule, Codes, Labels and Comments.  The spreadsheet gives examples of contents for each of these columns based on the previous image, for ID, race, ethnicity, gender, age and age at onset](../../images/DataDictSolution.png "Example of a data dictionary for the above dataset")
> >
> {: .solution}  
{: .hands_on}  

A key benefit of data dictionaries is that the information captured is generally not confidential or patient-sensitive and, therefore, there are few if any restrictions to sharing them.

## Data catalogues
Data catalogues are useful for teams, projects or consortia where there are many datasets being used and generated by the group.  Data catalogues are the metadata for the individual datasets, and they can provide context and provenance for each dataset within the group of datasets.

Here is an example of a data catalogue

![Image of a spreadsheet example of a data catalogue](../../images/DataCat.png "Example of a data catalogue")

You may not want to capture all these metadata, and you may want to include other items such as phenotype, links to any publications related to the dataset, and links to the published dataset.

Additionally, if there are complex data-sharing agreements, this information could be added to the catalogue (for internal use).

When projects and consortia are live, the corresponding data catalogue is a work in progress alongside the project and needs to be updated as required.  Although data catalogues can be somewhat time-consuming to set up, this is more than made up with the time saved looking for details about datasets, especially as people switch in and out of the team.

As with data dictionaries, data catalogues rarely capture any information that needs to be restricted access and usually can be shared in full.  The exception potentially may be the contact person's email address.  

More information about data catalogues can be found here  
 [Using a data catalogue within a research team](https://tess.elixir-europe.org/materials/rdmbites-using-a-data-catalogue-within-a-research-team)

## Publishing on Zenodo
Data dictionaries and data catalogues include a wealth of information about the clinical datasets they reference, and in general, contain no sensitive or confidential data.  [Zenodo](https://www.zenodo.org) is an open repository for research outcomes hosted by CERN.  It is free to use and each upload is citeable through a Digital Object Identifier (DOI).  Uploads are instantly available, with version control features.  Publishing on platforms such as Zenodo can help make your data findable.

Here, you can see published data dictionaries for a consortium
![Image of a screenshot of the CLUSTER Consortium page on the Zenodo website.  There are two entries for data dictionaries of datasets](../../images/CLUSTER_Zenodo.png "Screenshot of CLUSTER Consortium community page on Zenodo website")

To upload, create a login using GitHub ID, ORCID, or email.  Click on Upload (at the top of the page, left of centre), and follow the instructions.

## Managing Data Access Requests
Regardless of whether or not you publish data dictionaries and data catalogues, it is worth considering the mechanisms when receiving data access requests.  

When a project is ongoing, if there is a Data Management Committee (DMC), the responsibility for data access requests would be with the DMC.  It is really helpful to have a centralised point to provide data governance including guidelines for data sharing.

It is really worthwhile clarifying the process of handling data access requests to minimise effort and delays for both the data holder and the data requester.  This process may need to be revised as the project ends.

> <tip-title>Points to consider when handling data access requests</tip-title>  
> * Who will be the point of contact and the email address (e.g. create a general project email that is handled by the project administration team)
> * Expected length of time before requester receives a response
> * Involve the institution's legal team 
>   - Terms and conditions of use of data and/or samples
>   - Publication guidelines
> * Standardise the process with a Data Access Request Form  
{: .tip}

An example of a data access process is shown below
![Image of a four step process to request for data access.  Step 1 email the project.  Step 2 Discuss with the project what data or samples are being requested.  Step 3 approval by relevant committees.  Step 4 Decision communicated to requestor](../../images/DataAccessProcess.png "Example of a data access process")

This is an example of a [data and sample access request form](https://zenodo.org/record/8344400) that you might find useful as a starting point to create one for your project.

In the UK, for members of the UK Health Data Research Alliance, the HDRUK Innovation Gateway may also be a way to manage the data access request process.  This requires the metadata of the health datasets to be loaded to the Gateway web portal.  Researchers with an account can then search the portal and request access to datasets. The request access process and also creating collections of metadata are described in more detail in the [Data Custodian Playbook](https://discourse.healthdatagateway.org/t/data-custodian-playbook/980)

This work was funded by the ELIXIR-UK: FAIR Data Stewardship training UKRI award (MR/V038966/1).

[comment]: <> (need to also include acknowledgement of CLUSTER funding)


topic: fair
tutorial: fair-data-registration
title: Data Registration
difficulty: intermediate
time_estimation: 40M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 2ba6ea4c1d34b908c8c7cb92e7c91820
last_modified: 2026-02-26T21:12:15.968122
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-data-registration/tutorial.html

content:
The concept of data registration is defined as well as ways in which data registration can be achieved.  Learners will be able to describe why indexed data repositories are important as well as resources enabling you to choose a searchable repository.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}




# Data registration and the FAIR Principles
Data registration relates to the following 3 FAIR Principles (Table 3.1).  
We will discuss and signpost these in this Episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. **(meta)data are assigned a globally unique and persistent identifier**<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes<br>F4. **(meta)data are registered or indexed in a searchable resource**                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary<br> A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. **(meta)data are released with a clear and accessible data usage license**<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        

Table 3.1: The 15 FAIR Guiding Principles. Principles relating to data registration in **black**.

# What is data deposition and registration?

Data deposition and registration refer to the process of uploading data to a searchable resource and providing appropriate metadata to facilitate its discoverability.
For example, a data repository, where data and metadata can be uploaded, may enable it to be discovered, preserved and accessed.  Here we use the general term data repository to describe any online storage location that can host deposited (meta)data.

In the context of FAIR, data deposition relates to a number of the Guiding Principles. Firstly,  _“(meta)data are registered or indexed in a searchable resource, Indexed in a searchable resource: a resource where (meta)data are organised so that they can be queried based on defined fields, ”_ (FAIR Principle F4).  Searchable (indexed) metadata enables humans and computers to query and discover data of interest, though this depends on what is indexed.  Here, indexing refers to a process that occurs within the architecture of the data repository (local indexing) where metadata are organised so that they can be queried based on a defined field.  It is worth noting that community resources focused on a particular domain (for example, the human database in [Ensembl](https://www.ensembl.org/Homo_sapiens/Info/Index)) are better indexed for a particular community, rather than generic repositories (for example, [Zenodo](https://zenodo.org/)) which may not index the community specific components, and may focus on higher level metadata.  Indexing by an internet search engine is another example of this.  Google (and other search engines, such as Yahoo and Yandex) have an agreed vocabulary ([schema.org](https://schema.org/)), within web pages, that are ‘scraped’ and indexed.  While the focus of this vocabulary was originally intended for commercial products, community-specific efforts to facilitate discipline-specific indexing are underway (for example, [Bioschemas](https://faircookbook.elixir-europe.org/content/recipes/findability/seo/bioschemas-data-page.html)). 

# Why should I upload my data to a data repository?

Data repositories are generally preferred to file storage systems (such as Dropbox) or sharing data on an ad hoc basis since they often better support FAIR best practice.  Repositories will assign citable, _“globally unique and persistent identifiers”_ (FAIR Principle F1) to data, and in some cases enable a data submitter to apply a data usage licence through association with the resource (FAIR Principle R1.1).  

Although not exclusively, data repositories support the creation of metadata through curation interfaces providing drop-downs and text fields for metadata entry and validation.  Often in the case of a domain or data-specific data repository, such as BioStudies shown in the previous Episode, drop-downs for metadata curation will link community-endorsed vocabularies (FAIR Principle R1.3).

# Types of data repository

General public data repositories, such as [Zenodo](https://zenodo.org/), are multidisciplinary and permit registration and upload of open and closed access (meta)data.  Metadata curation is relatively high level and made searchable via indexing.  Relating to data in the Life Sciences, Zenodo is often used to publish and provide citable URLs to supplementary data within articles, usually in instances where a domain repository does not exist.

Institutional repositories work similarly and provide an online archive for hosting, indexing and preserving research output specific to an institution.  Typically these house more than data, providing a repository often for documents and articles.  Institutions will have their own systems supported locally or buy into company solutions. 

Discipline-specific repositories cater for communities and datatypes, and typically provide web interfaces to annotate rich metadata at the point when data are submitted.  Examples of these belong to the suite of data repositories at the [European Bioinformatics Institute (EBI)](https://www.ebi.ac.uk/submission/) where rich metadata creation is supported by teams of curators.


> <question-title></question-title>
>
> An example of a discipline-specific repository is [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress) database.  ArrayExpress stores data from high-through functional genomics assays, such as RNAseq, ChIPseq and expression microarrays.
> The data submission interface of ArrayExpress is called [Annotare](https://www.ebi.ac.uk/fg/annotare/login/).  Without creating a login, what help is given to a person looking to submit a dataset for the first time?
>
> > <solution-title></solution-title>
> >
> > Both a [submission guide](https://www.ebi.ac.uk/fg/annotare/help/index.html) and [YouTube video](https://www.youtube.com/watch?v=ANr2PTVy7JA) is provided.
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> Finding more help on how to upload data to specific repositories
> The [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html) is an online open resource housing specific ‘how to’ guides or recipes.  Use the FAIR Cookbook to find two recipes for “depositing data to Zenodo” and “registering datasets with Wikidata”, respectively.
>
> > <solution-title></solution-title>
> >
> > Open the **Findability** pulldown on the left hand banner to find recipes for the following:
> > [Depositing to generic repositories - Zenodo use case](https://faircookbook.elixir-europe.org/content/recipes/findability/zenodo-deposition.html) and [Registering Datasets in Wikidata](https://faircookbook.elixir-europe.org/content/recipes/findability/registeringDatasets.html).
> >
> {: .solution}
{: .question}




> <question-title></question-title>
>
> Choosing the right data repository for your data
> [FAIRsharing](https://fairsharing.org/) helps researchers identify suitable data repositories, standards and policies relating to their data.  Use this resource to identify data repositories for proteomic data.
>
> > <solution-title></solution-title>
> >
> > Access the search bar for the [FAIRsharing database registry](https://fairsharing.org/search?fairsharingRegistry=Database).  Search for proteomics and select “repository” under “Record Type”.
> > ![A screenshot of FAIRsharing showing the results for a search for all proteomics repositories.](../../images/figure3-1-fairsharing.png "FAIRsharing allows you to search for specific record types that are relevant for your area.")
> >
> {: .solution}
{: .question}


# Useful Resources
- Registries and lists of public repositories: [FAIR Cookbook](https://fairsharing.org/search?page=1&recordType=repository) and [nature](https://www.nature.com/sdata/policies/repositories) journal
- Publishing your data: [RDMkit](https://rdmkit.elixir-europe.org/data_publication)
- Using Bioschemas to embed metadata into webpages: [FAIR Cookbook Bioschemas](https://faircookbook.elixir-europe.org/content/recipes/findability/seo/bioschemas-datacatalog.html)


topic: fair
tutorial: ro-crate-intro
title: RO-Crate - Introduction
difficulty: introductory
time_estimation: 30M
tools_json: []
requirements_json: ["type: none", "type: none", "type: external", "type: \"internal\""]
tags_json: []
content_hash: 92e85555d9c4c4ab67f9f9fbcbfc9b92
last_modified: 2026-02-26T21:12:15.978663
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-intro/tutorial.html

content:
This tutorial assumes you have already completed [An overview of the RO-Crate concept and its implementations](https://gallantries.github.io/video-library/videos/ro-crates/intro/slides/) and have a basic understanding of working with JSON.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



In this tutorial, meant to be read along with the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.1/), we'll walk through the initial steps for creating a basic RO-Crate. You are invited to replicate the below steps on your local computer.


## Making a folder into an RO-Crate

In the simplest form, to describe some data on disk, an _RO-Crate Metadata File_ is placed in a folder alongside a set of files or folders.

First create a new folder `crate1/` and add a single file `data.csv` to represent our dataset:

```
"Date","Minimum temperature (°C)","Maximum temperature (°C)","Rainfall (mm)"
2022-02-01,16.0,28.4,0.6
2022-02-02,16.3,17.2,12.4
```

Next, to turn this folder into an RO-Crate we need to add the _RO-Crate Metadata File_, which has a fixed filename.
Create the file `ro-crate-metadata.json` using [Visual Studio Code](https://code.visualstudio.com/) or your favourite editor, then add the following JSON:

```json
{
  "@context": "https://w3id.org/ro/crate/1.1/context",
  "@graph": [

  ]
}
```

Your folder should now look like this:

![Folder listing of crate1, including data.csv and ro-crate-metadata.json](../../images/ro-crate-intro/crate1-folders.svg "Any folder can be made into an RO-Crate by adding <code>ro-crate-metadata.json</code>")

The presence of the reserved `ro-crate-metadata.json` filename means that `crate1` (and its children) can now be considered to be an **RO-Crate**. We call the top-level folder of the crate for the **RO-Crate Root** and can now refer to its content with relative file paths.

We also need to make some declaration within the JSON file to turn it into a valid _RO-Crate Metadata Document_, explained in the next session.


## JSON-LD preamble

The preamble of `@context` and `@graph` are JSON-LD structures that help provide global identifiers to the JSON keys and types used in the rest of the RO-Crate document. These will largely map to definitions in the [schema.org](http://schema.org/) vocabulary, which can be used by RO-Crate extensions to provide additional metadata beyond the RO-Crate specifications. It is this feature of JSON-LD that helps make RO-Crate extensible for many different purposes -- this is explored further in the specification's [appendix on JSON-LD](https://www.researchobject.org/ro-crate/1.1/appendix/jsonld.html). In short, only JSON keys (_properties_) and types defined this way can be used within the RO-Crate Metadata Document.

However, in the general case it should be sufficient to follow the RO-Crate JSON examples directly without deeper JSON-LD understanding. The RO-Crate Metadata Document contains a flat list of _entities_ as JSON objects in the `@graph` array. These entities are cross-referenced using `@id` identifiers, rather than being deeply nested. This is one major difference from JSON structures you may have experienced before.  The `@type` keyword associates an object with a predefined type from the JSON-LD context. Almost any property can alternatively be used with an `[]` array to provide multiple values.

The rest of this tutorial, and indeed most of the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.1/), specify which entities can be added to the `@graph` array.


## RO-Crate Metadata descriptor

The first JSON-LD _entity_ to add has the `@id` value of `ro-crate-metadata.json` to describe the JSON file itself:


```json
{
    "@id": "ro-crate-metadata.json",
    "@type": "CreativeWork",
    "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},
    "about": {"@id": "./"}
}
```

This required entity, known as the [RO-Crate Metadata Descriptor](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#ro-crate-metadata-file-descriptor), helps this file self-identify as an RO-Crate Metadata Document, which is conforming to (`conformsTo`) the RO-Crate specification version 1.1. Notice that the `conformsTo` URL corresponds to the `@context` URL version-wise, but they have two different functions. The context brings the defined terms into the metadata document, while the conformance declares which RO-Crate conventions of using those terms are being followed.


> <tip-title>RO-Crate versions</tip-title>
> This tutorial is written for RO-Crate 1.1, the RO-Crate website will list the [current specification version](https://www.researchobject.org/ro-crate/specification.html) -- RO-Crates can generally be upgraded to newer versions following [semantic versioning](https://semver.org/) conventions, but check the [change log](https://www.researchobject.org/ro-crate/1.1/appendix/changelog.html) for any important changes. The next development version of the specification, indicated with a `-DRAFT` status, may still be subject to changes and should only be used with caution.
{: .tip}


## RO-Crate Root

Next we'll add another entity to the `@graph` array, to describe the [RO-Crate Root](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#direct-properties-of-the-root-data-entity):


```json
{
    "@id": "./",
    "@type": "Dataset",
    "hasPart": [

    ]
}
```

By convention, in RO-Crate the `@id` value of  `./` means that this entity describes the folder in which the RO-Crate metadata file is located. This reference from `ro-crate-metadata.json` is therefore semantically marking the `crate1` folder as being the RO-Crate Root.


{: .note }
This example is a folder-based RO-Crate stored on disk, and therefore absolute paths are avoided, e.g. in case the root folder is moved or archived as a ZIP file. If the crate is being served from a Web service, such as a data repository or database where files are not organized in folders, then the `@id` might be an absolute URI instead of `./` -- this is one reason why we point to the root entity from the metadata descriptor, see section [Root Data Entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html) for details.

## Describing the root entity

When describing the [root entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#direct-properties-of-the-root-data-entity), the properties generally apply to the whole of the crate. For instance it is a good idea to give a description of why these resources are gathered in a crate, as well as giving the crate a name and license for FAIR reuse and citation.

> <question-title></question-title>
>
> Try to add the `name`, `description` and `datePublished` properties, and for `license` as a cross-reference, use [SPDX](https://spdx.org/licenses/) license list to find the identifier for Creative Commons Zero or another license of your choice:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@id": "./",
> >   "@type": "Dataset",
> >   "hasPart": [ ],
> >   "name": "Example crate",
> >   "description": "I created this example by following the tutorial",
> >   "datePublished": "2023-05-22T12:03:00+0100",
> >   "license": { "@id": "http://spdx.org/licenses/CC0-1.0"}
> > }
> > ```
> {: .solution}
{: .question}


> <tip-title>License identifiers</tip-title>
> In the above solution, the identifier for CC0-1.0 <http://spdx.org/licenses/CC0-1.0> is slightly
> different from their listed web page URI <https://spdx.org/licenses/CC0-1.0.html> -- the former
> is chosen to align with [SPDX JSON-LD identifiers](https://github.com/spdx/license-list-data/tree/main/jsonld),
> which unfortunately are not shown directly on their website as _permalinks_.
> It is not a requirement in RO-Crate to use permalinks for `@id` of entities like licenses,
> it is nevertheless best practice to propagate permalinks where known.
{: .tip}

> <tip-title>Choosing a license</tip-title>
> Choosing a license appropriate for your dataset can be non-trivial, particularly if third-party data/software and multiple organizations are involved. See [FAIR Cookbook on licensing](https://faircookbook.elixir-europe.org/content/recipes/reusability/ATI-licensing.html). It is worth noting that an RO-Crate permits data entities to have a `license` different from the overall Crate license. It is still recommended to choose an overall Crate license that can legally apply across all the content in the RO-Crate Root.
{: .tip}


## About cross-references

In a RO-Crate Metadata Document, entities are cross-referenced using `@id` reference objects, rather than using deeply nested JSON objects.
In short, this _flattened JSON-LD_ style (shown below) allows any entity to reference any other entity, and RO-Crate consumers can directly find all the descriptions of a given entity as a single JSON object.


![JSON block with id `ro-crate-metadata.json` has some attributes, `conformsTo` RO-Crate 1.2, and `about` referencing id `./`. In second JSON block with id <code>./</code> we see additional attributes such as its name and description.](../../images/ro-crate-intro/introduction-figure-1.svg "showing RO-Crate Metadata descriptor's <code>about</code> property pointing at the RO-Crate Root entity with matching <code>@id</code>")


> <question-title></question-title>
>
> Consider the root Data Entity `./`, and add such a cross-reference to the file `data.csv` using the _property_ called `hasPart`:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "./",
> >     "@type": "Dataset",
> >     "hasPart": [
> >       {"@id": "data.csv"}
> >     ],
> >     "…": "…"
> > }
> > ```
> {: .solution}
{: .question}


The RO-Crate root is always typed `Dataset`, though `@type` may in some cases have additional types by using a JSON array instead of a single value.  Most entities can have such more specific types, e.g. chosen from [schema.org type list](https://schema.org/docs/full.html).

> <question-title></question-title>
> 1. Navigate the schema.org type list to find a subtype of `CreativeWork` that is suitable for a learning resource..
> 2. Modify the root entity's `@type` to be an array.
> 3. Add the type name for learning resource at the end of the array.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "./",
> >     "@type": ["Dataset", "LearningResource"],
> >     "hasPart": [
> >       {"@id": "data.csv"}
> >     ],
> >     "…": "…"
> > }
> > ```
> {: .solution}
{: .question}

The root has several metadata properties that describe the RO-Crate as a whole, considering it as a Research Object of collected resources. The section on [root data entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html) details further the required and recommended properties of the root `./`.




## Data entities

A main type of resources collected in a Research Object is _data_ -- simplifying, we can consider data as any kind of file that can be opened in other programs. These are aggregated by the Root Dataset with the `hasPart` property. In this example we have an array with a single value, a reference to the entity describing the file `data.csv`.

> <tip-title>Referencing external resources</tip-title>
> RO-Crates can also contain data entities that are folders and Web resources, as well as non-File data like online databases -- see section on [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html).
{: .tip}

We should now be able to follow the `@id` reference for the corresponding _data entity_ JSON block for our CSV file, which we need to add to the `@graph` of the RO-Crate Metadata Document.


> <question-title></question-title>
>
> 1. Add a declaration for the CSV file as new entity with `@type` declared as `File`.
> 2. Give the file a human-readable `name` and `description` to detail it as _Rainfall data for Katoomba in NSW Australia, captured February 2022_.
> 3. To add this is a CSV file, declare the `encodingFormat` as the appropriate [IANA media type](https://www.iana.org/assignments/media-types/#text) string.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "data.csv",
> >     "@type": "File",
> >     "name": "Rainfall Katoomba 2022-02",
> >     "description": "Rainfall data for Katoomba, NSW Australia February 2022",
> >     "encodingFormat": "text/csv"
> > },
> > ```
> >
> {: .solution}
{: .question}


It is recommended that every entity has a human-readable `name`; as shown in the above example, this does not need to match the filename/identifier. The `encodingFormat` indicates the media file type so that consumers of the crate can open `data.csv` in an appropriate program, and can be particularly important for less common file extensions frequently encounted in outputs from research software and instruments.

For more information on describing files and folders, including their recommended and required attributes, see section on [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html).

> <question-title></question-title>
>
> 1. Consider if the file content of `data.csv` is not covered by our overall license (CC0), but [Creative Commons BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) (which only permits non-commercial use)
> 2. To override, add an  `license` cross-reference property on this particular data entity
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "data.csv",
> >     "@type": "File",
> >     "name": "Rainfall Katoomba 2022-02",
> >     "description": "Rainfall data for Katoomba, NSW Australia February 2022",
> >     "encodingFormat": "text/csv",
> >     "license": { "@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/" }
> > },
> > ```
> {: .solution}
{: .question}



## Contextual entities

Entities that we have added under `hasPart` are considered _data entities_, while entities only referenced from those are considered _contextual entities_ -- they help explain the crate and its content.

> <tip-title>Adding entities to the JSON array</tip-title>
> You may have noticed we're adding incrementally to the `@graph` array. It is important to remember the `,` between each entity, **except** for the final entity in the JSON array; and likewise for the properties within the JSON object for each entity. This is an artefact of the strict [JSON](https://www.json.org/) file format rules to simplify parsing. The order of the entities within the `@graph` JSON-LD array and the order of the keys within a JSON object is _not significant_. The _graph_ content is given by the `@id` cross-references.
{: .tip}

You may notice the subtle difference between a _data entity_ that is conceptually part of the RO-Crate and is file-like (containing bytes), while a _contextual entity_ is a representation of a real-life organization that can't be downloaded: following the URL, we would only get its _description_. The section [contextual entities](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html) explores several of the entities that can be added to the RO-Crate to provide it with a **context**, for instance how to link to authors and their affiliations.  Simplifying slightly, a data entity is referenced from `hasPart` in a `Dataset`, while a contextual entity is referenced using any other defined property.

## Detailing licenses

We have previously declared two different `license` cross-references. While following the URLs in this case explain the licenses well, it is also best practice to include a very brief summary of contextual entities in the RO-Crate Metadata Document. This is more important if the cross-reference do not use a permalink and may change over time. As a minimum, each referenced entity should have a `@type` and `name` property. It is also possible to add `url` for more information

> <question-title></question-title>
>
> Add a contextual entity for each of the two licenses, see the [licensing](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html#licensing-access-control-and-copyright) section for details:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
> >     "@type": "CreativeWork",
> >     "name": "CC BY-NC-SA 4.0 International",
> >     "description": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International"
> > },
> > {
> >     "@id": "http://spdx.org/licenses/CC0-1.0",
> >     "@type": "CreativeWork",
> >     "name": "CC0-1.0",
> >     "description": "Creative Commons Zero v1.0 Universal",
> >     "url": "https://creativecommons.org/publicdomain/zero/1.0/"
> > },
> > ```
> {: .solution}
{: .question}


An additional exercise is to try to unify the two entites so that both use spdx identifiers, remembering to update the corresponding `license` cross-references when changing the `@id`. However, not all licenses have a direct SPDX identifier.


## Author

Moving back to the RO-Crate root `./`, let's specify who are the authors of the crate.

> <question-title></question-title>
>
> 1. Add yourself as an [`author`](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html#people) of the crate using the type `Person`
> 2. Include your preferred name.
> 3. If you don't have an [ORCID](https://orcid.org/), you may use either the URL of your main home page at your institution, or a crate-local identifier like `#alice`.
> 4. Include your `affiliation` as a string value.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@id": "./",
> >   "@type": "Dataset",
> >   "author": {"@id": "https://orcid.org/0000-0002-1825-0097"},
> >   "…": "…"
> > },
> > {
> >   "@id": "https://orcid.org/0000-0002-1825-0097",
> >   "@type": "Person",
> >   "name": "Josiah Carberry",
> >   "affiliation": "Brown University"
> > }
> > ```
> {: .solution}
{: .question}


> <tip-title>Who can be authors of an RO-Crate?</tip-title>
> When we say someone is an author of a crate, it means they have contributed something substansively to its content (typically the data). Agreement on what is considered authorship on a dataset can be tricky; you may decide some people would be better represented as `contributor`. One advantage of RO-Crate is that authorship can be declared explicitly also on each data entity, so it can be clearer where each person have contributed (e.g. a statistician is author of an R script).  This means that generally the authors of the crate can be a broader, more inclusive list than perhaps traditionally recognized as academic authorship.
{: .tip}

> <question-title></question-title>
>
> 1. "Unroll" your `affiliation` of the person as cross-reference to another contextual entity, typed as an `Organization`.
> 2. You can use [ROR](https://ror.org/) to find an identifier for most educational/research institutions, or you can use the main web page of your organization as its `@id`.
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@id": "https://orcid.org/0000-0002-1825-0097",
> >   "@type": "Person",
> >   "name": "Josiah Carberry"
> > },
> > {
> >   "@id": "https://ror.org/05gq02987",
> >   "@type": "Organization",
> >   "name": "Brown University",
> >   "url": "http://www.brown.edu/"
> > }
> > ```
> {: .solution}
{: .question}



The reuse of existing identifiers is important for both persons and organization from a FAIR perspective, as their names may not be globally unique.

> <question-title></question-title>
>
> 1. Now imagine you are going to publish the RO-Crate on your institution's web pages.
> 2. Cross-reference the same Organization entity with `publisher` from the RO-Crate Root entitity:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >     "@id": "./",
> >     "@type": "Dataset",
> >     "publisher": {"@id": "https://ror.org/05gq02987"},
> >     "…": "…"
> > }
> > ```
> {: .solution}
{: .question}


## Validating and visualizing

As we made this RO-Crate Metadata File by hand, it's good to check for any JSON errors, such as missing/extra `,` or unclosed `"` quotes. Try pasting the file content into the [JSON-LD Playground](https://json-ld.org/playground/). It should show up any errors, for example:

```
JSON markup - SyntaxError: JSON.parse: expected `','` or `']'` after array element
at line 29 column 5 of the JSON data
```

Modify the JSON file in your editor to fix any such errors. You can also use editor commands such as _Format Document_ to ensure you have consistent spacing, indentation and brackets.

If the document passes without errors in the JSON-LD Playground, you should see output under _Expanded_ looking something like:

```json
[
  {
    "@id": "ro-crate-metadata.json",
    "@type": [
      "http://schema.org/CreativeWork"
    ],
    "http://schema.org/about": [
      {
        "@id": "./"
      }
    ],
    "http://purl.org/dc/terms/conformsTo": [
      {
        "@id": "https://w3id.org/ro/crate/1.1"
      }
    ]
  },
  {
    "@id": "./",
    "@type": [
      "http://schema.org/Dataset"
    ],
    "…": "…"
  }
]
```

This verbose listing of the JSON-LD shows how the `@context` has correctly expanded the keys, but is not particularly readable. Try the _Visualized_ tab to see an interactive rendering of the entities:

![Visualized in the JSON-LD Playground](../../images/ro-crate-intro/jsonld-playground-visualized.png)

As the RO-Crate Metadata Document is valid JSON-LD it is also possible to process it using Linked Data technologies such as triple stores and SPARQL queries. It is beyond the scope of this tutorial to explain this aspect fully, but interested readers should consider how to [handle relative URI references](https://www.researchobject.org/ro-crate/1.1/appendix/relative-uris.html).
As an example, try the _Table_ button and notice that the entities with relative identifiers are not included. This is because when converting to RDF you need absolute URIs which do not readily exist when a crate is stored on disk, we've not decided where the crate is to be published yet.

## Advanced: Converting JSON-LD to triples

Try modify the graph's `@context` within the [JSON-LD Playground](https://json-ld.org/playground/) (don't modify the `ro-crate-metadata.json` on disk), and revisit the _Table_ rendering.

```json
{
  "@context": [
    "https://w3id.org/ro/crate/1.1/context",
    { "@base": "arcp://uuid,deffa754-c764-4e04-aabf-e600c6200553/" }
  ],
  "…": "…"
}
```

![Triples table in the JSON-LD Playground](../../images/ro-crate-intro/jsonld-playground-table.png)

Above `arcp://uuid,deffa754-c764-4e04-aabf-e600c6200553/` is a randomly generated identifier to represent the RO-Crate root, and now the JSON-LD Playground can show all the triples from the metadata file. You can likewise use the _N-Quads_ button to convert the metadata file to the [RDF N-Quads](http://www.w3.org/TR/n-quads/) format. Most RDF libraries and stores have JSON-LD support, but may need to specify a base URI as we did above, making a new UUID for each imported RO-Crate.


## HTML preview

An RO-Crate can be distributed on disk, in a packaged format such as a zip file or disk image, or placed on a static website. In any of these cases, an RO-Crate can have an accompanying HTML version (`ro-crate-metadata.html`) designed to be human-readable.

![Example dataset for RO-Crate specification](../../images/ro-crate-intro/ro-crate-preview-example.png)

> <question-title></question-title>
>
> Try navigating the [preview of the running example](rainfall-1.2.1/ro-crate-preview.html) and find:
>
> 1. What is the license of the rainfall CSV?
> 2. What is the affiliation of the crate's author?
> 3. What does the Validity Check inspect
> 4. What is not covered by this check?
>
> > <solution-title></solution-title>
> > 1. CC BY-NC-SA 4.0 International
> > 2. Brown University
> > 3. The context, and for root dataset: existance, valid identifier, name, description, license and date published.
> > 4. The other entities were not checked, e.g. the `affiliation` of the author.
> {: .solution}
{: .question}


## Advanced: Generating HTML preview

If you have [Node JS](https://nodejs.org/) installed, you can install and try the [ro-crate-html](https://www.npmjs.com/package/ro-crate-html) tool on your `crate1` folder:

```bash
npm i -g ro-crate-html
rochtml crate1/ro-crate-metadata.json
```

Or if you have [Docker](https://www.docker.com/), something like:

```bash
docker pull simleo/rochtml
docker run -v $(pwd):/crate -it simleo/rochtml -c ro-crate-preview_files /crate/ro-crate-metadata.json
```

The above will generate a `ro-crate-preview.html` file within your RO-Crate Root. Experiment with modifying the Metadata file and re-generating the HTML preview to see how your rendering differs from the [running example](rainfall-1.2.1/ro-crate-preview.html).

## Next steps

You have completed making a basic RO-Crate. You may try any of the following:

- Add and describe additional data entities, adding additional properties for [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html)
- Follow the RO-Crate specification for additional [contextual entities](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html) you can add to the crate
- Try briefly describing [provenance](https://www.researchobject.org/ro-crate/1.1/provenance.html) or [software](https://www.researchobject.org/ro-crate/1.1/workflows.html) for any additional data entities you have added.

> <question-title>Complete RO-Crate Metadata Document</question-title>
>
> The final RO-Crate Metadata Document constructed in this tutorial should look something like:
>
> > <solution-title></solution-title>
> > ```json
> > {
> >   "@context": "https://w3id.org/ro/crate/1.1/context",
> >   "@graph": [
> >     {
> >       "@id": "ro-crate-metadata.json",
> >       "@type": "CreativeWork",
> >       "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"},
> >       "about": {"@id": "./"}
> >     },
> >     {
> >       "@id": "./",
> >       "@type": ["Dataset", "LearningResource"],
> >       "hasPart": [
> >         {"@id": "data.csv"}
> >       ],
> >       "name": "Example dataset for RO-Crate specification",
> >       "description": "Official rainfall readings for Katoomba, NSW 2022, Australia",
> >       "datePublished": "2023-05-22T12:03:00+0100",
> >       "license": {"@id": "http://spdx.org/licenses/CC0-1.0"},
> >       "author": { "@id": "https://orcid.org/0000-0002-1825-0097" },
> >       "publisher": {"@id": "https://ror.org/05gq02987"}
> >     },
> >     {
> >       "@id": "data.csv",
> >       "@type": "File",
> >       "name": "Rainfall Katoomba 2022-02",
> >       "description": "Rainfall data for Katoomba, NSW Australia February 2022",
> >       "encodingFormat": "text/csv",
> >       "license": {"@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/"}
> >     },
> >     {
> >       "@id": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
> >       "@type": "CreativeWork",
> >       "name": "CC BY-NC-SA 4.0 International",
> >       "description": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International"
> >     },
> >     {
> >       "@id": "http://spdx.org/licenses/CC0-1.0",
> >       "@type": "CreativeWork",
> >       "name": "CC0-1.0",
> >       "description": "Creative Commons Zero v1.0 Universal",
> >       "url": "https://creativecommons.org/publicdomain/zero/1.0/"
> >     },
> >     {
> >       "@id": "https://orcid.org/0000-0002-1825-0097",
> >       "@type": "Person",
> >       "name": "Josiah Carberry",
> >       "affiliation": {
> >         "@id": "https://ror.org/05gq02987"
> >       }
> >     },
> >     {
> >       "@id": "https://ror.org/05gq02987",
> >       "@type": "Organization",
> >       "name": "Brown University",
> >       "url": "http://www.brown.edu/"
> >     }
> >   ]
> > }
> > ```
> {: .solution}
{: .question}



# Copyright and license

© Copyright 2021-2023 University of Technology Sydney, The University of Manchester UK and RO-Crate contributors.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

## Changes

This tutorial has been adapted from the revised [RO-Crate introduction for 1.2](https://www.researchobject.org/ro-crate/1.2-DRAFT/introduction.html) by Stian Soiland-Reyes. Changes include:
* Rewriting to tutorial style with exercises
* Adaptation to Galaxy Training Material rendering
* Explicit links to RO-Crate specifications
* Example changed to different organization and license


topic: fair
tutorial: data-management
title: FAIR data management solutions
difficulty: intermediate
time_estimation: 10M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: f51bd9e2be825906b9bd9971bf228996
last_modified: 2026-02-26T21:12:15.932833
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/data-management/tutorial.html

content:
The FAIR (Findable, Accessible, Interoperable, Reusable)  data stewardship created the foundation for sharing and publishing digital assets, especially data. This apply to machine accessibility and emphasize that all digital assets should share data in a way that will enable maximum use and reuse.

This tutorial is a short introduction to the FAIR data management framework. You can find out more at the [FAIR Pointers](https://elixir-uk.github.io/FAIR-Pointers/ep1/index.html) online course.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data management planning
In recent years we have notice a data explosion. Number of sequence records in each release of [GenBank](https://www.ncbi.nlm.nih.gov/genbank/statistics/), from 1982 to the present, doubled in size approximately every 18 months. Great amounts of data available followed by expanding range of tools and computational solutions result in reproducibility crisis. Having **data management plan (DMP)** in place is essential to achieve FAIR data management. DMPs are often described as living documents and should be updated according to changing circumstances.

There are several ways to set up FAIR (Findable, Accessible, Interoperable, Reusable) data management plans (DMPs) :
  - Findable (F): Data description and collection or reuse of existing data
  - Accessible (A): Standardised authentication or authorisation (e.g. HTTP, HTTPS)
  - Interoperable (I): Documentation and data quality
  - Reusable (R): Storage and backup supported by legal and ethical requirements

## Data description and collection or reuse of existing data
Reusing legacy datasets from institutional repositories or the digital libraries data collections can be FAIRified retrospectively. Support for data collection and development, throughout the life cycle can be provided and followed by change management and capacity improvement.

Multi-part FAIR research need a way of wrapping up, describing and sharing to promote the reuse of data. **Data sharing agreements** define the purpose of data sharing. Reference roles and responsibilities; specifies the purpose and legal requirements, e.g. for data security.

An institutional aim should be to create an integrated view and context over fragmented resources using their **persistent identifiers (PIDs)** and **metadata**. To make datasets findable, these metadata need to be as widely available as possible.

Enhancing reproducibility, quality and transparency by ensuring information flow and showcasing secondary use is also a part of data management. Promoting hands-on data experience and events activities built an collaborative environment for reproducible science.


## Documentation and data quality
Having access to local knowledge and encouraging best practises at the departmental level is a smart way to offer direction on a variety of standards and methods. In order to implement FAIR data practises within an institution, resources and infrastructure are needed. To increase the possibility of data reuse, several FAIR requirements can be satisfied using freely available guidelines e.g. [RDMkit](https://rdmkit.elixir-europe.org/), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html), [ELIXIR-UK DaSH Fellowship](https://sites.google.com/view/navigation-portal-fellowship/home?authuser=0) initiative and repositories e.g. [Zenodo](https://zenodo.org/), [Harvard Dataverse](https://dataverse.harvard.edu/) and [figshare](https://figshare.com/).

## Storage and backup
Systems for storage, backup and collaboration depend upon technical infrastructure.The **'3-2-1 rule'**, a recommendation for saving three copies of the research data—two locally and one off-site—is a standard backup strategy for research data. Data, metadata, and other research artefacts, such as ontologies, software, documentation, and papers, must all be kept in locations where they are adequately safeguarded, backed up, and accessible to maximise their potential for reuse. Appropriate access management is essential, in addition to backup and restoration services that protect researchers against data loss, theft, malfunctioning computers or storage media, and accidental deletion or inadvertent alterations to the data.

The fundamental component of infrastructure required for the FAIR research data lifecycle are repository services. They allow access to the data, a persistent identifier, and the descriptive metadata that support interoperability. Repositories can include basic data storage, resource finding, managing access and use of private information, facilitating peer review of information related to publications or services requiring digital preservation, and more.

The [OpenAIRE](https://www.openaire.eu/opendatapilot-repository-guide) repository guide advises users to check the availability of a suitable repository in following order:

1. The most effective option (if available) is to maintain the data in accordance with acknowledged discipline-specific criteria using an established, dedicated (external) data archive or repository that caters specifically to the study domain.
2. Making use of institutional data repositories is the second-best option.
3. If none of those options is practical, a free data repository should be used.

Up-to-date lists of available registered data repositories can be found at [re3data](https://www.re3data.org/) and [FAIRsharing](https://fairsharing.org/search?fairsharingRegistry=Database).

## Legal and ethical requirements
Institutional support network (data stewards, ethics boards, IP, legal and financial offices) need to guide researchers in safeguarding data management responsibilities and resources.

# Conclusion
You will have the advantage of saving time and resources by planning how to FAIRify your data in the early phases of your research endeavour. To put this into action, a data management strategy, or DMP, must be written. A DMP is also where you outline your data collection, storage, processing, sharing, and disposal procedures. Planning the management and FAIRification of your data reduces the possibility of issues down the road, whether they be practical, legal, or technical.

Keep in mind that creating FAIR data is a complex process. Consider how you can make your data FAIR one step at a time at each stage of the creation, collection, documentation, storage, sharing, archiving, and preservation processes. The framework for the rest of your study planning is laid by incorporating your data documentation. Imagine you want to use a dataset that was generated by another researcher and how you would like it to be found. Hope this quick introduction to FAIR data management solutions will help you improve not only your experience with data but also influence others by using your guidance and FAIRified resources.


topic: fair
tutorial: fair-rna
title: FAIRification of an RNAseq dataset
url: https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-rna/tutorial.html
difficulty: intermediate
time_estimation: 1H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 0005cf43f641a1ed9b1d8d5abee1afe8
last_modified: 2026-02-26T21:12:15.971431
workflows_json: []
content:
[RNA sequencing](https://en.wikipedia.org/wiki/RNA-Seq) is chosen here as an example of how to FAIRify data for a popular assay in the Life Sciences. RNAseq data can be shared and curated in designated public repositories using established ontologies (and controlled vocabularies) for describing protocols and biological material (metadata).

Two international repositories are commonly used to locate and download RNAseq (meta)data: [ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress) and [GEO](https://www.ncbi.nlm.nih.gov/geo/).  Other repositories for raw sequence data exist (e.g. SRA, ENA, DDBJ), but ArrayExpress and GEO specifically house and index expression data , including rich metadata detailing samples, data processing and final results files such as gene expression matrices.

By submitting data to a public repository, it becomes openly accessible, searchable and annotated with rich metadata, by the submitter and curation team. Note, both repositories belong to the [FAIRsharing database registry](https://fairsharing.org/search?fairsharingRegistry=Database), which can help you find public repositories for all types of Life Science data.

This lesson will take you through a publicly available RNAseq dataset in ArrayExpress and show you how it meets FAIR principles using the checklist published in 2016 {% cite Wilkinson2016 %}.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Example of FAIRification when data is uploaded to a public repository

We will use the following human RNAseq dataset for this learning material [ArrayExpress:E-MTAB-8316](https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-8316). This link hosts all metadata and links to downloadable raw and transformed data, shown as a schematic below.

![A schematic of RNAseq data and metadata within ArrayExpress](../../images/fair_rna_1.png "A schematic of RNAseq data and metadata within ArrayExpress. Data shown are raw FASTQ files and summary gene transcript count matrices.")


# Finding and accessing an RNAseq dataset
In this section, four of the FAIR Guiding Principles are put into practice:
- **(F1) (Meta)data are assigned a globally unique and persistent identifier**
- **(F4) (Meta)data are registered or indexed in a searchable resource**
- **(A1) (Meta)data are retrievable by their identifier using a standardised communications protocol**
- **(A1.1) The protocol is open, free, and universally implementable**


## (F1) (Meta)data are assigned a globally unique and persistent identifier

> <question-title></question-title>
>
>  The RNAseq dataset we are looking at belongs to a study published in Nature Medicine {% cite Alivernini2020 %}.  Can you find the globally unique and persistent identifier for the RNAseq data and data descriptions (metadata), within the text of the publication? 
>
> > <solution-title></solution-title>
> >
> > Under "Data and code availability', towards the end of the manuscript, we are told the following. 
> >
> > ![The data availability section within the RNAseq publication](../../images/fair_rna_2.png "The data availability section within the RNAseq publication showing links to access primary data.")
> >
> {: .solution}
{: .question}



Note that the dataset's unique and persistent (unchanging) identifier is E-MTAB-8316, (F1) and could also be expressed as the full [URL](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316. Note that there are two other identifiers given to 2 single cell RNAseq datasets published as part of the same study. For this lesson, we are only considering the bulk RNAseq dataset.

Note also, the same record can be accessed using a resolution service, such as [identifiers.org](https://identifiers.org/),  that allows URLs to be regularised using a namespace (arrayexpress) and the identifier local to the database ([E-MTAB-8316](https://identifiers.org/arrayexpress:E-MTAB-8316))



## (A1) (Meta)data are retrievable by their identifier using a standardised communications protocol (A1.1) The protocol is open, free, and universally implementable 

> <question-title></question-title>
>
>  Use the following [URL](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316) to access this dataset. The URL, in this case, is a concatenation of the URL of the data repository and the unique identifier: 
>
> > <solution-title></solution-title>
> >
> > You should see the following webpage hosted at the EBI (www.ebi.ac.uk). In the context of FAIR, you have now accessed this RNAseq data using a weblink employing https, where https is a standardised communications protocol or “data getting method” that is open, free and universally implementable (A1, A1.1).  Note that the title of this RNAseq record and the title of the companion publication are different, which means the two cannot be linked by title.  Instead, the accession (persistent ID) is used to connect the data and companion publication.
> >
> > Many databases, including ArrayExpress, also have programmatic ways to access and download data.  For advanced users, ArrayExpress [host a REST server](https://www.ebi.ac.uk/biostudies/arrayexpress/help#programmatic) for this purpose.  Other packages written by third parties, such as [ffq](https://github.com/pachterlab/ffq), also permit this function.
> > ![RNAseq data record](../../images/fair_rna_3.png "The RNAseq data record showing the Primary ID circled in red.")
> >
> {: .solution}
{: .question}


## (F4) (Meta)data are registered or indexed in a searchable resource

> <question-title></question-title>
>
>  Location the dataset using the persistent ID in a different way.  Use the search menu in ArrayExpress to find the dataset again.   Use the following link to (access ArrayExpress)[https://www.ebi.ac.uk/biostudies/arrayexpress/studies/] and then type E-MTAB-8316 into the search bar.
>
> > <solution-title></solution-title>
> >
> > Note here that data access is gained through searching the database whereas previously we used a direct URL.  We are able to search ArrayExpress with the persistent ID  because the RNAseq dataset and its metadata are indexed (F4).
> >
> > ![ArrayExpress/BioStudies search bar](../../images/fair_rna_4.png "The ArrayExpress/BioStudies search bar with the query circled in red.")
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> Alternatively, use the same search menu in ArrayExpress to search again using the words "macrophage rheumatoid arthritis" and selecting "rna-seq of coding rna" on the left-hand search bar. 
>
> > <solution-title></solution-title>
> >
> > Here, the dataset we want is not the first in the list but appears later in the search results. In this example, data access is gained through searching metadata (data about the experiment) (F4), and not the persistent ID as we did previously.
> >
> > ![ArrayExpress/BioStudies search bar for the metadata query](../../images/fair_rna_5.png "The ArrayExpress/BioStudies search bar showing a metadata query circled in red.")
> >
> {: .solution}
{: .question}


# Learning about the experiment and dataset from reading the metadata

In this section, the Findability Principles are put into practice:

- **(F1) (Meta)data are assigned a globally unique and persistent identifier**
- **(F2) Data are described with rich metadata**
- **(F3) Metadata clearly and explicitly include the identifier of the data they describe**

## (F1) (Meta)data are assigned a globally unique and persistent identifier (F3) Metadata clearly and explicitly include the identifier of the data they describe

> <question-title></question-title>
>
> Start looking at the metadata given on dataset webpage. Try to find the unique, persistent identifier in the [record](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316) (F1) (Meta)data are assigned a globally unique and persistent identifier.  
>
> > <solution-title></solution-title>
> >
> > The persistent identifier is circled in red and is the first thing we see in the record (F1). All metadata (descriptions about the data) and the actual raw data files are linked from this page. Here metadata clearly and explicitly include the identifier of the data they describe (F3).
> >
> > ![RNAseq data record showing the Primary ID (persistent identifier)](../../images/fair_rna_6.png "The RNAseq data record showing the Primary ID (persistent identifier) circled in red.")
> >
> > Here data are described with rich metadata (F2), which allows a person to reuse data appropriately by reducing ambiguity around what the data mean or how they are derived. This principle applies also to other principles in  FAIR such as (R1) Meta(data) are richly described with a plurality of accurate and relevant attributes.
> >
> {: .solution}
{: .question}

## (F2) Data are described with rich metadata
Indexing rich metadata allows a person to easily locate a dataset of interest, since it is made searchable within ArrayExpress. Metadata is added by the person submitting the data and is further curated by the ArrayExpress database. Metadata curation is performed via a web-based submission interface called [Annotare](https://www.ebi.ac.uk/fg/annotare/login/) which aids rich curation through using community ontologies and controlled vocabularies.


> <question-title></question-title>
>
> Familiarise yourself with the [page layout](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316): there are links to all protocols, data, sample metadata and assay type. How many samples are in this dataset?
>
> > <solution-title></solution-title>
> >
> > There are 12 samples in this dataset.
> >
> > ![The RNAseq data record showing the number of samples (assays)](../../images/fair_rna_7.png "The RNAseq data record showing the number of samples (assays) circled in red.")
> >
> {: .solution}
{: .question}



> <question-title></question-title>
>
> What data provenance can you find? i.e. what processes have been performed to create the data linked from this page? Are there any metadata which might be absent?
>
> > <solution-title></solution-title>
> >
> > The final 2 protocols within the protocol table drawdown detail all data transformations for the raw and transformed data. 
> >
> > ![The protocol table from the RNAseq data record](../../images/fair_rna_8.png "The protocol table from the RNAseq data record showing the final 2 data protocols circled in red.")
> >
> {: .solution}
{: .question}


When recording data provenance, ambiguities can be introduced.  True provenance can be achieved though through publishing scripts and software versions, enabling other researchers to repeat all data transformation processes. 


# Metadata and community standards

In this section, we will consider Interoperability and data Reuse:

- **(I1) (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.**
- **(I2) (Meta)data use vocabularies that follow FAIR principles.**
- **(R1) (Meta)data are richly described with a plurality of accurate and relevant Attributes.**
- **(R1.3) (Meta)data meet domain-relevant community standards.**


Interoperability is mentioned in our companion (FAIR Pointers course)[https://fellowship.elixiruknode.org/latest/carpentries-course-fair-pointers]  where we consider merging datasets using a common controlled vocabulary.  

## (I1) (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.
Commonly when adding metadata to a dataset a published ontology is used.  In its simplest form, an ontology is a dictionary of terms you use to annotate a piece of data. For example, the NCBI taxonomy database is something you may have used to annotate species within a dataset. An ontology, though, will also define relationships between terms. So in the taxonomy example, the term "Homo sapiens" will belong to parent terms such as Primate, Mammal and so on. Importantly, by using an ontology you can ensure you are using interoperable and searchable terms with your data.

## (I2) (Meta)data use vocabularies that follow FAIR principles.  (R1) (Meta)data are richly described with a plurality of accurate and relevant Attributes.
Published ontologies are linked from the ArrayExpress data submission tool, [Annotare](https://www.ebi.ac.uk/fg/annotare/login/), so this work is done for you. Although, if you wish to start using ontologies to annotate your data at the point it is produced, the [FAIRsharing Standards registry](https://fairsharing.org/search?fairsharingRegistry=Standard) is a place to start. Importantly, when an ontology is published it too becomes findable and reusable in the context of the FAIR principles **(I2)**.
Metadata using published ontologies permit interoperability since you can match identical annotations across data and databases **(I1 & I2)**. Additionally, they are created, refined and used by communities of practice **(R1, R1.3)**.


> <question-title></question-title>
>
> Look at the page again. Identify any metadata that belongs to an existing, published ontology. Note, we have mentioned one already: "Homo sapiens" as part of [taxonomy](https://www.ncbi.nlm.nih.gov/taxonomy).
>
> > <solution-title></solution-title>
> >
> > There are many.  There are ontologies used for Source Characteristics such as ‘Developmental stage’,  ‘Disease’, ‘Organism part’, ‘Cell type’, and so on. These annotations are curated at the point of data submission.
> >
> > ![published ontologies metadata](../../images/fair_rna_9.png "Metadata within the record using controlled terms from published ontologies.")
> >
> > There are also others that are not so obvious. Under "Protocols', a protocol ontology is used under the column ‘Type’.  By selecting the EFO link-and-arrow following the protocol annotation, you are taken to the FAIR (Experimental factor ontology)[https://www.ebi.ac.uk/ols/ontologies/efo] at the EBI. 
> >
> >![Experimental Factor Ontology protocol table from the RNAseq data record](../../images/fair_rna_10.png "The protocol table from the RNAseq data record showing the final 2 data protocols circled in red with links to the Experimental Factor Ontology.")
> >
> {: .solution}
{: .question}

## (R1.3) (Meta)data meet domain-relevant community standards.


![fair RNAseq MINSEQ](../../images/fair_rna_11.png "The MINSEQ Score table from the RNAseq data.")

Notice that the "MINSEQE" star rating on the left hand banner, indicates a level of Community Compliance in terms of (meta)data submission.   The community  standard in this case is MINSEQE (Minimum Information about a Sequencing Experiment Checklist) which is recorded in FAIRsharing {% cite https://doi.org/10.5281/zenodo.5706412 %}.  This example receives a maximum 5-star score demonstrating that data and metadata are available for experiential design, protocols and  variables as well as processed and raw data.

At a more advanced level of community compliance, the ArrayExpress database uses a domain specific metadata format: MAGE-TAB, also identified in [FAIRsharing](https://fairsharing.org/FAIRsharing.ak8p5g).  MAGE-Tab is a tab-delimited file containing all relevant metadata for the experiment.  This is a machine-readable version of the webpage we are looking at.



> <hands-on-title>Locate the MAGE-TAB file containing sample metadata, and open in Excel.</hands-on-title>
>
> Notice how all data on the webpage are also contained in this machine-readable format.  You are looking to download the SDRF file (Sample and Data Relationship Format).
>
> ![fair RNAseq links](../../images/fair_rna_12.png "Links to machine-readable metadata in the RNAseq data record circled in red.")
{: .hands_on}


Finally, this ArrayExpress record links to the raw sequence data housed in the ENA (European Nucleotide archive).  We see this in an exercise later in this course.  The ENA record in turn links to ENA SRA XML format (community recognised), which gives another machine-readable interoperability format.


> <hands-on-title>Machine Readable RNA XML formats</hands-on-title>
> 
> Have a quick look at the machine readable RNA XML formats.  Notice how data are contained within tags very similar to HTML.
>
> - [ENA:PRJEB34761](https://www.ebi.ac.uk/ena/browser/api/xml/PRJEB34761)
> - [ENA:ERP117716](https://www.ebi.ac.uk/ena/browser/api/xml/ERP117716)
>
{: .hands_on}


# Downloading data and metadata for reuse
- **(R1.1) (Meta)data are released with a clear and accessible data usage license.**
- **(R1.2) (Meta)data are associated with detailed provenance.**

## (R1.1) (Meta)data are released with a clear and accessible data usage license
Data licensing makes it clear to any person looking to reuse a dataset, who the data belongs to and how it can be used.  Open licences (available to all) are often used to maximise data reuse, though different stipulations can be applied to these licences to restrict/enforce certain downstream activities, such as reuse of data for commercial purposes.

The [Creative Commons](https://creativecommons.org/) licences are used commonly for life science data, of which there are 7: **CC0, CC BY, CC BY-SA, CC BY-NC, CC BY-ND, CC BY-NC-SA, CC BY-NC-ND**

CC0 is the most permissive licence where "no rights are reserved'.  This means the data creator releases the data without restriction to the public domain.

The second CC BY licence is often written as CC BY 4.0 to reflect the most current (4th) version of the licence.  Here, anyone can share and adapt data for any purpose, including commercial CC BY.

CC BY 4.0 is virtually identical to CC0, except by using CC BY, legal rights are retained to acknowledge the originator of the data.


> <question-title></question-title>
>
> Look at the following site detailing the (Creative Commons licences)[https://creativecommons.org/share-your-work/cclicenses/].  What is the effect of adding the following letters to the CC BY licence; SA, NC and ND?
>
> > <solution-title></solution-title>
> >
> > ![fair RNAseq licences](../../images/fair_rna_13.png "Screenshots from the Creative Commons licences definitions pages for SA, NC and ND.")
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> Look for the licensing options for [arrayexpress](https://www.ebi.ac.uk/biostudies/arrayexpress) at the bottom of the page under [Licensing](https://www.ebi.ac.uk/licencing).  Which licence is used?
>
> ![fair RNAseq licence footer](../../images/fair_rna_14.png "The footer from the RNAseq data record showing the link to the Licensing conditions circled in red.")
>
> > <solution-title></solution-title>
> >
> > CC0 is used.
> >
> {: .solution}
{: .question}


> <question-title></question-title>
>
> How could I download the data or metadata  from this page?
>
> > <solution-title></solution-title>
> >
> > Machine-readable metadata can be downloaded from the “MAGE-TAB Files” link.  There are 2 files: Investigation Design Format (experimental overview) and the Sample/Data Relationship Format (sample/data descriptions).  Note this information can be accessed on the right hand banner as well.
> >
> > Processed data can be downloaded also, in this case there is a link to a read-count matrix.  Sequencing data from the NEA.
> >
> > Access to raw FASTQ data is given via the reference to the ENA database on the right hand banner under “Linked Information”.
> >
> > ![fair RNaseq metadata and rawdata](../../images/fair_rna_15.png "Links to metadata, raw data and processed data in the RNAseq record, circled in red.")
> >
> {: .solution}
{: .question}


## (R1.2) (Meta)data are associated with detailed provenance.
The term "data provenance" relates to information about how and why a piece of data was produced. Often the individual producing the data and details of its production (protocols) are recorded as part of provenance.

> <question-title></question-title>
>
> What provenance metadata can you find on [Arrayexpress:E-MTAB-8316](https://www.ebi.ac.uk/biostudies/ArrayExpress/studies/E-MTAB-8316)?
>
> > <solution-title></solution-title>
> >
> > We are looking for information that could help us understand how data have been created.  We have been given the name of the data creator noting that there is an option too for the author to submit an ORCID.  Data creation and transformations are detailed under Protocols.
> >
> > ![fair RNaseq submitter](../../images/fair_rna_16.png "Showing submitter and data provenance metadata in the RNAseq record, circled in red.")
> >
> {: .solution}
{: .question}


topic: fair
tutorial: fair-persistent-identifiers
title: Persistent Identifiers
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 33a6e09eedc6813f9e319f79d56321c5
last_modified: 2026-02-26T21:12:15.971033
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-persistent-identifiers/tutorial.html

content:
The persistent identifier is defined.  Learners will be able to define the structure of an identifier and explain its importance.  

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Persistent identifiers and the FAIR Principles
Data Identifiers relate to the following 5 FAIR Principles (Table 5.1).  We will discuss and signpost these in this episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | **F1. (meta)data are assigned a globally unique and persistent identifier**<br>F2. data are described with rich metadata (defined by R1 below)<br>**F3. metadata clearly and explicitly include the identifier of the data it describes** <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | **A1. (meta)data are retrievable by their identifier using a standardized communications protocol** <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>**A2. metadata are accessible, even when the data are no longer available** |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>**I3. (meta)data include qualified references to other (meta)data**                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. (meta)data are released with a clear and accessible data usage license<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        


Table 5.1: The 15 FAIR Guiding Principles.  Principles relating to data identifiers are highlighted in **black**.

# Using persistent identifiers (PIDs)

Identifiers are an important theme within the FAIR principles, arguably being foundational; they are considered two of the pillars for the FAIR principles since they are crucial for Findable (F) and Accessible (A) principles.

Identifiers are an eternal reference to a digital resource such as a dataset and its metadata. They provide the information required to reliably identify, verify and locate your research data. 

Commonly, a persistent identifier is a unique record ID in a database, or a unique URL that takes a researcher to the data in question in a database. Persistent identifiers (PIDs) have to be unique so that only one dataset can be identified by this identifier. In addition to the identifier being unique, it needs to be persistent. When depositing or hosting data, you should ensure the longevity of this persistence meets your requirements, which may require reading specific database policies regarding identifier policy.  

Since FAIR permits the withdrawal of data, the FAIR Principles combat the potential for broken URLs by stating:  _“Metadata are accessible, even when the data are no longer available.”_  (FAIR Principle A2).  This means the link (PID) remains valid, displaying all the original metadata of the record even though the data is no longer available.

**It is important to note that when you upload your data to a public repository, the repository will create this ID for you automatically**.

Based on [how to FAIR](https://howtofair.dk/how-to-fair/persistent-identifiers/), many resources can help you find databases to assign PIDs to your data. One of these resources is [FAIRsharing](https://fairsharing.org/) something we’ve already encountered in the previous episodes. [FAIRsharing](https://fairsharing.org/), provides a list of databases grouped by domains and organisations.

# The structure of persistent identifiers

To ensure that a PID is globally unique, commonly it is supplied as a unique URL. For the case of a record in a biological database, the use of a URL ensures that the database record ID is associated with the database name or often some derivation of this. This is often enough to ensure the uniqueness of the PID for any future scenario. 

Commonly, for things like publications, a DOI is used for the PID, where DOI stands for Digital Object Identifier. An example is shown below where the PID is constructed from 3 pieces of information: the resolver service, the prefix (namespace) and the suffix (dataset ID).

**Resolver service**: the domain/service/institution hosting the PID e.g. [https://www.doi.org] (doi.org)

**Prefix**: a unique number referring to the publisher. This is also known as the namespace.

**Suffix**: the unique dataset number

![DOI_structure](../../images/figure5-1.png "The structure of a DOI")

For biological data, PIDs usually require a resolver that can deal with multiple resolving locations, which means that if a database changes its name or internal structure, the new and old variations of the PID remain valid and take the user to the (meta)data.  One commonly used resolver service is called [identifiers.org](https://docs.identifiers.org/) which maintains a list of database namespaces (prefix’s) as a persistent record.  If a database changes its name, it keeps the original namespace operational or alternatively arranges for redirection from the original.

Examples of using identifiers.org to construct a PID are given below for 2 different databases, [Ensembl](https://www.ensembl.org/Mus_musculus/Gene/Summary?g=ENSMUSG00000031887;r=8:105984918-105991241) and [WikiPathways](https://www.wikipathways.org/pathways/WP1243.html), respectively. The **namespace** is given as the database name in these examples.

![PID_structure](../../images/figure5-2.png "The structure of a PID using the identifiers.org resolver service")

> <question-title></question-title>
>
> Access the preprint for [Identifiers for the 21st century: How to design, provision, and reuse persistent identifiers to maximize utility and impact of life science data](https://www.google.com/url?q=https://doi.org/10.1101/117812&sa=D&source=docs&ust=1681921501619473&usg=AOvVaw0iHyEgvEzDniMbIcvhokQk).  This paper makes 10 recommendations for PID best practice. 
>
> <p> Locate the lesson for “Do not reassign or delete identifiers”.  Which PID is used as an example of a “tombstone page”?  Which FAIR Principle does this relate to? </p>
>
> > <solution-title></solution-title>
> >
> > Tombstone PID: [https://www.uniprot.org/uniprotkb/A0AV18](https://www.uniprot.org/uniprotkb/A0AV18/history)
> > <p> </p> This relates directly to the FAIR Principle A2. _“Metadata are accessible, even when the data are no longer available”_ Note that other FAIR Principles are also illustrated.
> > <p>  </p> The Tombstone page is _“retrievable by their identifier using a standardised communications protocol”_(FAIR Principle A1), which in this case is http(s).  The page contains metadata which _“are assigned a globally unique and persistent identifier”_ (FAIR Principle F1).  Also the metadata _“clearly and explicitly includes the identifier of the data they describe”_ (FAIR Principle F3) noting that the identifier itself is featured on the webpage.
> > <p>  </p> Additionally, the webpage features a link to the updated UniProt record thereby _“metadata include qualified reference to other (meta)data”_ (FAIR Principle I3)
> >
> {: .solution}
{: .question}


# Useful Resources
- More on identifiers: [RDMkit](https://rdmkit.elixir-europe.org/identifiers.html) and [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/findability/identifiers.html)
- Nick Juty, Sarala M. Wimalaratne, Stian Soiland-Reyes, John Kunze, Carole A. Goble, Tim Clark; Unique, Persistent, Resolvable: Identifiers as the Foundation of FAIR. Data Intelligence 2020; 2 (1-2): 30–39. {% cite Juty2020 %}


topic: fair
tutorial: fair-origin
title: FAIR and its Origins
difficulty: intermediate
time_estimation: 40M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 3bbde779f497947ed103a051e85a7c1a
last_modified: 2026-02-26T21:12:15.970560
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-origin/tutorial.html

content:
FAIR as an acronym, its origins and its guiding principles are introduced.  Learners will be able to explain the difference between FAIR and open data and contextualise the main principles of FAIR around the common characteristics of identifiers, access, metadata and registration.

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}



# FAIR

The concepts underlying the FAIR principles are intuitively grounded in good scientific practice, though formalising them as easily measurable is not straightforward.  Being completely ‘FAIR’ is somewhat of an ideal, where it is often more pragmatic to be ’FAIR enough’ for a particular purpose or use case.

FAIR’s primary goal is to maximise data reuse by researchers.  The FAIR principles enable reuse by helping researchers share and manage their data, though FAIR is not limited to data alone and can be applied to services, software, training and workflows.  The word FAIR is an acronym, derived from its major components: ‘F’indable, ‘A’ccessible, ‘I’nteroperable, and ‘R’eusable which form the foundation of the FAIR Guiding Principles. 

**Findable** means that data and its metadata can be found/discovered by humans and computers.  Part of this is making rich metadata and keywords available to search engines and data repositories, so that companion data can be discovered.

**Accessible** means that once discovered, data and metadata can be accessed/downloaded by humans and computers.  Typically this means the commitment of the resource to its long term hosting and availability, with a suitable licence, and in appropriate format.

**Interoperable** means that data and metadata are supplied in formats that can be easily used and interpreted by humans and computers. The file formats and terms (vocabularies) used can be integrated easily with other datasets and software.

**Reusable** means that metadata is rich, enabling appropriate reuse.  Commonly FAIR will encourage the use of community standards for data curation.

These 4 major components form headings for the 15 FAIR Guiding Principles shown in **Table 1.1**

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a globally unique and persistent identifier<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>R1.1. (meta)data are released with a clear and accessible data usage license<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards                                        |

Table 1.1: The FAIR guiding principles as described in Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) {% cite Wilkinson2016 %}. 


> <question-title></question-title>
>
> Look at the wording of the FAIR principles in **Table 1.1**. Which terms are used more than once?  Which terms are you seeing for the first time?
>
> > <solution-title></solution-title>
> >
> > Shared terms are: **data** (F1, F2, F4, A1, A2, I1, I2, I3, R1, R1.1, R1.2, R1,3), **metadata** (F1, F2, F3, F4, A1, A2, I1, I2, I3,  R1, R1.1, R1.2, R1,3), **identifier** (F1, F3, A1), **protocol** (A1, A1.1, A1.2). <br> Not so obviously shared terms are: **vocabularies** (I1, I2), **access** (A1, A1.1, A1.2, R1.1) <br> Terms which you may be seeing for the first time are: metadata, persistent identifier, searchable resource, standardised communications protocol, authentication and authorisation procedure, knowledge representation, vocabularies, qualified references, data usage license, provenance, domain-relevant community standards.
> >
> {: .solution}
{: .question}


The aim of this course is to put these principles into context and familiarise learners with  terms used commonly around FAIR.  A glossary of terms appears at the bottom of this episode as a point of reference.

# Open data and FAIR 

![FAIR_is_not_the_same_as_Open_data](../../images/figure1-2_fair-vs-open.png "FAIR is not the same as Open data")
 

Since FAIR promotes data sharing, it is often misunderstood as Open data.
  
The [Open Data handbook](https://opendatahandbook.org/) defines Open data as **“data that can be freely used, reused and redistributed by anyone.”**  Commonly, FAIR data is open however FAIR compliance does not mandate access without restriction. Instead, for instances where FAIR data is subject to restricted access, the conditions of access need to be stated to be compliant with FAIR, for example access around sensitive data.  Often in FAIR data, people adhere to the philosophy of **“as open as possible, and as closed as necessary”** thereby maximising opportunities to reuse data.

> <question-title></question-title>
>
> Under which circumstances would restricted or closed  access to FAIR data be advisable?
>
> > <solution-title></solution-title>
> >
> > Where data is sensitive or subject to intellectual property.  Protecting sensitive data overrules mandating research data should be open access. <br> Note though that in most cases, people following FAIR principles will be looking to share their data openly. Also note that sensitive data can be released through anonymisation and in many cases subject to controlled access by the authority of the principal investigator or data access committee. 
> >
> {: .solution}
{: .question}


# What is meant by FAIRification and FAIRness of data? 
  
**FAIRification** is the process of making your data FAIR compliant by applying the 15 Guiding Principles shown in **Table 1.1**.  The extent to which you apply these principles defines the **FAIRness** of your data.  In other words, FAIRness refers to the extent by which your data is FAIR and implies some implicit means of measuring its compliance.

# FAIR’s origins

A [report from the European Commission Expert Group on FAIR data](https://zenodo.org/record/1285272#.ZD_9KhXMIqt) describes the origins of FAIR and its development in 2014-2015 by a FORCE11 Working Group.  The following exercise dips into this report and asks you to investigate some of FAIR’s history and foundation. 


> <question-title></question-title>
>
> Read page 11 of [the European Commission report](https://zenodo.org/record/1285272#.ZD_9QxXMIqu), under the heading “Origins and definitions of FAIR”. What benefit did the FORCE11 Working Group see to coining the word FAIR?
>
> > <solution-title></solution-title>
> >
> > The report states: “a FORCE11 Working Group coined the FAIR data definition, latching onto an arresting and rhetorically useful acronym. The wordplay with fairness, in the sense of equity and justice, has also been eloquent in communicating the idea that FAIR data serves the best interests of the research community, and the advancement of science as a public enterprise that benefits society.”
> >
> {: .solution}
{: .question}


# Course structure
  
During the remainder of this course we will put the FAIR Guiding Principles into context using the 4 FAIR characteristics of **metadata**, **data registration**, **access** and **identifiers**, and devote an episode to each of these.  Whilst we teach, we will map content to the appropriate FAIR principle and define all relevant terms identified.
 
# Glossary of FAIR terms
  
All terms in this glossary are mentioned in the FAIR Guiding Principles (Table 1.2) and referenced in the following episodes of this course.
  
  
  | The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a **globally unique and persistent identifier**<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the **identifier** of the data it describes <br>F4. (meta)data are registered or **indexed in a searchable resource**                                   |
| To be Accessible:           | A1. (meta)data are retrievable by their **identifier** using a **standardized communications protocol** <br>A1.1 the protocol is open, free, and **universally implementable**<br>A1.2 the protocol allows for an **authentication and authorization procedure**, where necessary <br>A2. metadata are accessible, even when the data are no longer available |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable **language for knowledge representation**. <br>I2. (meta)data use **vocabularies** that follow FAIR principles<br>I3. (meta)data include **qualified references** to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br> R1.1. (meta)data are released with a clear and accessible **data usage license**<br>R1.2. (meta)data are associated with detailed **provenance**<br>R1.3. (meta)data meet domain-relevant **community standards**            

Table 1.2: Terms used by the FAIR Principles that appear in this glossary are highlighted in **black**.

**Globally unique and persistent identifier**: a reference to a digital resource such as a dataset, a document, a database, etc, usually given as a URL, that takes the user to that resource.  The persistent identifier (PID) is unique, globally, in the sense that it is not used to identify any other digital resource.  The PID is persistent, in that it enables a resource to be located long term, preferably permanently.  For more see: [How to FAIR](https://howtofair.dk/how-to-fair/persistent-identifiers/) and [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/findability/identifiers.html), episode 5 (identifiers).

**Indexed in a searchable resource**: repositories and catalogues that can be queried (search box) are examples of searchable resources.  Indexing refers to processes within the architecture of the data repository where (meta)data are organised so that they can be queried based on defined fields.  Indexing by an internet search engine (for example, Google) is another example of an indexed and searchable resource.  For more see: [GO FAIR](https://www.go-fair.org/fair-principles/f4-metadata-registered-indexed-searchable-resource/).
  
**Standardised communications protocol**: a method that connects two computers and ensures secure data transfer. Examples of this include the hypertext transfer protocol ([http](https://en.wikipedia.org/wiki/HTTP)(s)) and the file transfer protocol ([ftp](https://en.wikipedia.org/wiki/File_Transfer_Protocol)) that permit data to be requested and downloaded by selecting a link on a webpage, or launched from within a script, for example using an (application programming interface) API. For more see the [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/).
  
**Universally implementable**: in the context of a standardised communications protocol such as http(s), universally implementable means it can be used by a number of resources and software, for example Firefox, Chrome and Unix commands such as wget.  For more see the [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/).
  
**Authentication and authorisation procedure**: data repositories with authentication and authorisation procedures generally require a login (and password) to access (meta)data.  For more see the [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/).

**Language for knowledge representation**: in the context of (meta)data exchange between computers, (meta)data should be in formats that are universally recognised (interoperable standards).  For more see [GO FAIR](https://www.go-fair.org/fair-principles/i1-metadata-use-formal-accessible-shared-broadly-applicable-language-knowledge-representation/) and [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/introduction/FAIR-and-knowledge-graphs.html).
  
**Vocabularies**: (or **controlled vocabulary**) is a dictionary of terms you can use when producing (meta)data.  Controlled vocabularies are often shared between databases and communities so by using them you can allow data from different sources to be merged (interoperable), based on a shared understanding of the concepts.   Ontologies are related to vocabularies, where terms in the vocabulary are organised by relations between them.  A commonly used ontology is the [NCBI taxonomy](https://www.ncbi.nlm.nih.gov/taxonomy) where the term ‘Homo sapiens’ belongs to a hierarchy of parent terms such as ‘Primates’ and ‘Mammalia’.   The ontology defines the vocabularies and the parent/child relationships.   For more see Ten simple rules for making a vocabulary FAIR {% cite Cox2021 %}.
  
**Qualified references**: are terms used to describe relationships to pieces of (meta)data.  For more see [GO FAIR](https://www.go-fair.org/fair-principles/i3-metadata-include-qualified-references-metadata/).
  
**Data usage license**: describes the legal rights on how others use your data.  For more details on considerations in selecting a licence, or to find out more about types of licence that are available, see [RDMkit](https://rdmkit.elixir-europe.org/licensing). 
  
**Data provenance**: refers to metadata describing the origin of a piece of data, including information such as version, original location of the data, and usually an audit trail up to the current version.  For more see [RDMkit](https://rdmkit.elixir-europe.org/data_provenance#how-to-document-and-track-your-data-provenance).
  
**Community standards**: are standard guidelines used to structure and exchange data, usually supported by community-developed resources and/or software.  In the context of (meta)data, community standards relate often to the standardised ontologies used by a domain of research, and minimum information guidelines allowing data to be interoperable.  For more see [FAIRDOM](https://fair-dom.org/community_standards).
  
**Machine-readable**: though this term is not referenced in the FAIR principles, it is often discussed within the FAIR context. Machine-readable (meta)data is supplied in a structured format that can be read by a computer.  For more see [Open Data Handbook](https://opendatahandbook.org/glossary/en/terms/machine-readable/) and [RDMkit](https://rdmkit.elixir-europe.org/machine_actionability).
  
**(Meta)data**: is shorthand for ‘metadata and data’.

# Useful Resources
- The published FAIR Guiding Principles: Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) {% cite Wilkinson2016 %}.
- Recipes for data FAIRification, written by domain experts giving real-world examples: [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html)
- Documentation and frameworks for data FAIRification.  Each of the 15 FAIR principles is put into context with real data examples: [GO FAIR](https://www.go-fair.org/fair-principles/)
- FAIR walkthrough using examples from across all academic disciplines: [How to FAIR](https://howtofair.dk/)


topic: fair
tutorial: ro-crate-submitting-life-monitor
title: Submitting workflows to LifeMonitor
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["ro-crate"]
content_hash: 35746fc56bb520b1fa027ec57cfdc799
last_modified: 2026-02-26T21:12:15.984242
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-submitting-life-monitor/tutorial.html

content:
[{LM}](https://www.lifemonitor.eu/) is a service to support the
sustainability and reusability of published computational workflows. The
LifeMonitor helps alleviate the burden of maintaining workflows over time
through **automation**.

This document will show you **how to use the
[LifeMonitor](https://www.lifemonitor.eu/) service and its [GitHub
app](https://github.com/apps/lifemonitor)** on a workflow in a GitHub
repository.

> <agenda-title></agenda-title>
>
> In this tutorial, you will learn how to create a git repo, and begin working with it.
>
> 1. TOC
> {:toc}
>
{: .agenda}

## Why use the LifeMonitor?

Keeping workflows reusable takes work! Like all software, even the best
workflows can break over time if left unmaintained: for instance, there could be
a regression or an API change in an unpinned dependency, or an external resource
that the workflow relies upon might be moved to a different URL. Periodic
automated **testing** is a fundamental practice that helps expose such problems,
giving workflow maintainers a chance to intervene and fix them.  The
[LifeMonitor](https://www.lifemonitor.eu/) helps you monitor and periodically
execute the automated tests for your workflow.

Moreover, workflows should be findable to be reusable.
[LifeMonitor](https://www.lifemonitor.eu/) helps automate the generation of
[RO-Crate metadata](https://www.lifemonitor.eu/workflow_testing_ro_crate) as
well as registering new releases of your workflow with the
[WorkflowHub](https://workflowhub.eu).

Finally, [LifeMonitor](https://www.lifemonitor.eu/) can apply automated checks
to your workflow repository to help you follow [community best
practices]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}).


## Pre-conditions

We assume you **published a Galaxy workflow** to a GitHub repository following the
[*Galaxy Community best
practices*]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#community-best-practices).

In addition, we assume you have followed the same best practices to [**create
tests** for your
workflow]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#generating-tests-for-your-workflow) using
[Planemo](https://planemo.readthedocs.io/en/latest/best_practices_workflows.html#tests),
as well as configuring a [GitHub Actions
workflow]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#adding-a-github-workflow-for-running-tests-automatically) to automatically
run those tests.  Don't worry if this sounds like a lot: the [best
practices]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}) document provides straightforward
instructions.

### Running example

As a running example, we're going to use a simple "sort and change case"
workflow: it merely takes a text file, sorts the lines and swaps the case of all
the letters. You can access and inspect this example workflow, its tests and
its GitHub Action through [this link](https://github.com/by-covid/gtn/tree/main/docs/submitting-life-monitor/sort-and-change-case-workflow).


## Installing the LM GitHub App

The [{LM} GitHub App](https://github.com/apps/lifemonitor) provides the most
straightforward way to use {LM} with your GitHub-hosted workflow. The app can do
the following things.

* Examine your repository and apply a series of automated *checks*.
* Suggest pull requests (PRs) to make changes or additions to bring the workflow
  repository closer to conforming to best practices.
* Open issues to let you know about problems detected by the checks:
  * in some cases, you can interact with the LifeMonitor bot through the issues,
      to provide information or issue commands.
* Register new releases/versions of the workflow with both the [LifeMonitor
  service](https://app.lifemonitor.eu/) (for test monitoring) and the
  [WorkflowHub](https://workflowhub.eu/) workflow registry.


Follow the instructions provided by the [LifeMonitor
documentation](https://www.lifemonitor.eu/lm_wft_best_practices_github_app#installation)
to complete the app installation on your repository, and:

* we recommend that you enable the LifeMonitor "Issue Checks";
* we also recommend that you enable the automatic upload to the
    [WorkflowHub](https://workflowhub.eu/).

> <tip-title>Accounts Required</tip-title>
> Note that completing the installation steps may require you to create an
> account with both LifeMonitor and WorkflowHub.
> We recommend **using [LS Login](https://lifescience-ri.eu/ls-login/)** to authenticate with both
> services.
{: .tip}

> <tip-title>GitHub Connection Required</tip-title>
> You will also need to connect your GitHub account to your LifeMonitor account.
{: .tip}

## Configure your LifeMonitor account

To fully enable the features provided by LifeMonitor you need to enable some
optional settings in the LM configuration.

### Enable e-mail notifications

LifeMonitor can send you emails to notify you of important events related to
your workflows or the workflows you've [subscribed
to](https://www.lifemonitor.eu/lm_dashboard#subscribing-to-workflows).  We
recommend performing this step to -- among other things -- receive notifications
if your workflow tests start failing, or when a release of your workflow has
been registered with the LifeMonitor test monitoring service and with the
WorkflowHub registry.

[Click
here](https://www.lifemonitor.eu/lm_test_monitoring#enabling-e-mail-notifications)
to access the LifeMonitor documentation on how to perform this step.


### Configure GitHub integration

If you [installed the LM GitHub App](#installing-the-lm-github-app) on your
workflow repository your GitHub account should already be connected to
LifeMonitor. Verify this by accessing your account's [profile
page](https://app.lifemonitor.eu/account/profile) and checking that the
**Connected Accounts** table includes an entry for GitHub and it says
**CONNECTED**; if it says "CONNECT", then click on the button and follow the
procedure.  See the [LifeMonitor
documentation](https://www.lifemonitor.eu/faq#which-external-accounts-are-linked-with-my-lifemonitor-account)
for more details.

If you want, you can now check the LifeMonitor [GitHub integration
settings](https://app.lifemonitor.eu/account/profile?currentView=githubSettingsTab)
for your account. These are the default settings which will be applied unless
overridden with a repository-specific configuration file (we'll see how in a
later section).


### Connect your WorkflowHub account to LifeMonitor

Connecting your WorkflowHub identity to LifeMonitor allows LifeMonitor to
perform specific WorkflowHub actions on your behalf (using your identity).
These actions consist in registering and updating workflow entries as well as
querying the WorkflowHub for the workflows you have previously registered.

Check the *Account* tab on your [LifeMonitor account
profile](https://app.lifemonitor.eu/account/profile): if the buttons shown
in the image below say "**connected**", you're ok; else, click on the
appropriate "**connect** button and log into the WorkflowHub with the account
you'd like LifeMonitor to use.
![Connect WorkflowHub buttons](./images/account_connections_whub-dev-disconnected.png)
The LifeMonitor documentation has [a more detailed explanation](https://www.lifemonitor.eu/faq#which-external-accounts-are-linked-with-my-lifemonitor-account).


### Enable WorkflowHub Integration for your account

Enabling registry integration for WorkflowHub enables the LM automations related
to workflow registry activities.


Go to the *Registry Integration* tab on your [LifeMonitor account
profile](https://app.lifemonitor.eu/account/profile?currentView=registrySettingsTab).
Enable the switches for both "wfhub" and "wfhubdev".  When you flip the switch,
you may be taken to the WorkflowHub authentication page; in that case, log into
the WorkflowHub with the same account credentials that you connected in the
previous step.

![LM config registry integration](./images/lm_config_registry_integration.png)


## LifeMonitor Checks

Once the LM GitHub app is installed, it starts working for you by running its automated
checks. The checks generally report one problem at a time, by opening an issue
or opening a PR with a suggested fix.  In the following subsections we'll show
the sequence of PR's and issues that are opened in our Sort and Change Case
workflow repository.

### RO-Crate metadata

LM might notice that we don't have an RO-Crate:
![Missing RO-Crate PR](./images/missing_ro_crate_pr.png)

The LM app **automatically** executes
[repo2crate](https://github.com/crs4/repo2rocrate) as recommended by [best
practices]({% link topics/fair/tutorials/ro-crate-galaxy-best-practices/tutorial.md %}#best-practice-repositories-and-ro-crate) and
opens a pull request to propose the resulting RO-Crate for integration in our
repository.  Review the RO-Crate and integrate additional metadata or correct
it as necessary. Finally, merge the changes into your repository.  LifeMonitor with
automatically delete the PR branch once merged, to keep your repository tidy.
![Merge PR](./images/merge_pr.png)


### LifeMonitor configuration file

Next, LM suggests that we add a [LifeMonitor configuration
file](https://www.lifemonitor.eu/lm_wft_best_practices_github_app#configuration-file)
to our repository.  Through this file in YAML format, we can customize some aspects of
LifeMonitor's behavior and provide it with additional metadata.
![LM config PR](./images/lm_config_pr.png)

This time, we're going to make some changes to the default configuration before
accepting it.  The PR's "Conversation" tab on GitHub tells us to push changes to
a specific branch to make changes to the file:
![LM config PR branch](./images/lm_config_pr_branch.png)

So we'll go to the local copy of our workflow repository, fetch the remote and
checkout this branch:

```console
$ cd sort-and-change-case-workflow
$ git fetch origin
$ git checkout lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349
Branch 'lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349' set up to track remote branch 'lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349' from 'origin'.
Switched to a new branch 'lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349'
```

Now we can edit the file locally with our favorite text editor:

```console
$ nvim .lifemonitor.yaml
```

We're going to set the workflow name and make it public (which means that anyone
will be able to see it on LifeMonitor):

```yaml
name: "Sort and change case"
public: true
```

We're also going to enable automatic registration of workflow releases in
WorkflowHub and LifeMonitor.  We're going to edit the configuration so that:

1. workflow tag names that follow the [Semantic Versioning](https://semver.org/)
   convention are recognized as releases;
2. the WorkflowHub record is updated for each new release.

```yaml
    push:
      tags:
        - name: "*.*.*"  #  pattern to identify a release tag
          update_registries: ["wfhub"]  # Registry to be updated. "wfhub" == WorkflowHub
```

If you're just testing things, avoid dirtying your WorkflowHub collection.
Instead, point the LM app to the development instance of WorkflowHub by
specifying:

```yaml
update_registries: ["wfhubdev"]  # Registry to be updated. "wfhubdev" == WorkflowHub dev
```

Commit your changes now and push them to the repository:

```console
$ git commit -a -m "Update LifeMonitor settings"
[lifemonitor-issue-0d71f7fa25ea5d7c5b9f63e27b08b59fbc1b5349 0924ad7] Update LifeMonitor settings
1 file changed, 22 insertions(+), 52 deletions(-)
rewrite .lifemonitor.yaml (63%)
$ git push origin
```

Now you can go back to the Pull Request and merge it.
![LM config PR merged](./images/lm_config_pr_merged.png)

### New and changing checks

The set of automated checks provided by LifeMonitor is under active development.
This means that the exact sequence of issue and PR's you see may change over
time: this is normal.  Do feel free to [open an
issue](https://github.com/crs4/life_monitor/issues) if you notice something that
doesn't work as you expect or if you have a suggestion for a new automated
check.

## Automatic workflow registration to WorkflowHub


To have your workflow's WorkflowHub entry automatically updated, you
**must** [connect your WorkflowHub (or dev.WorkflowHub)
account](#connect-your-workflowhub-account-to-lifemonitor) to LifeMonitor
and [enable WorkflowHub (or dev.WorkflowHub)
integration](#enable-workflowhub-integration-for-your-account).

## Make a workflow release

Now you can make a release of your workflow. The LifeMonitor
[configuration](#lifemonitor-configuration-file) we specified for the workflow
repository specifies that we identify release by git tags matching the
expression.  You can tag the workflow as release "0.1.0" with the git CLI:

```console
$ git tag 0.1.0
$ git push --tags origin
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:crs4/sort-and-change-case-workflow.git
* [new tag]         0.1.0 -> 0.1.0
```

The LifeMonitor now start working and automatically register the workflow with
both the LifeMonitor test monitoring functionality and the WorkflowHub registry
you enabled and selected.

> <tip-title>E-mail notifications</tip-title>
> If you enabled e-mail notifications, you should receive an email telling
> you that the registration happened and providing you with a direct link to the
> workflow on LifeMonitor (from which you'll be able to access the corresponding
> WorkflowHub entry).
> ![LM new registration](./images/lm_new_registration.png)
{: .tip}

The workflow should also appear on your [LifeMonitor
dashboard](https://app.lifemonitor.eu/dashboard).  If the GitHub Action that
tests your workflow has not been executed yet, then LM will report that the test
status is "unavailable" -- but this is normal.  After you execute your action
then the system will report its passing or failing status.

The workflow entry in LifeMonitor also has a link to the corresponding
WorkflowHub entry.  Follow the link to reach a page with your workflow's
metadata:
![LM new registration workflowhub](./images/lm_new_registration-workflowhub.png)

## Profit

Now that you've completed the setup, you'll be notified if your workflow's tests
start to fail. {LM} will also periodically try to trigger the execution of your
testing GitHub Action should there be no commits on the repository that would
trigger the tests on their own.  Also, any new workflow releases that you create
will be automatically be propagated to both LifeMonitor and WorkflowHub, keeping
your registry up to date and your workflow a little more FAIR.


topic: fair
tutorial: ro-crate-in-python
title: RO-Crate in Python
difficulty: intermediate
time_estimation: 30M
tools_json: []
requirements_json: []
tags_json: ["ro-crate"]
content_hash: 05c3024bd0a16d5d3fd86a493d4f3098
last_modified: 2026-02-26T21:12:15.974708
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/ro-crate-in-python/tutorial.html

content:
This tutorial will show you how to manipulate [RO-Crates](https://w3id.org/ro/crate/) in Python using the [ro-crate-py](https://github.com/ResearchObject/ro-crate-py) package. It is based on the [ro-crate-py documentation](https://github.com/ResearchObject/ro-crate-py/blob/e1218fbca595f4c33059cfe15849ee2ae9e6896b/README.md).

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

Let's start by installing the library via [pip](https://docs.python.org/3/installing/). Note that the name of the package is `rocrate`.

```bash
pip install rocrate
```

# Creating an RO-Crate

In its simplest form, an RO-Crate is a directory tree with an `ro-crate-metadata.json` file at the top level. This file contains metadata about the other files and directories, represented by [data entities](https://www.researchobject.org/ro-crate/1.1/data-entities.html). These metadata consist both of properties of the data entities themselves and of other, non-digital entities called [contextual entities](https://www.researchobject.org/ro-crate/1.1/contextual-entities.html). A contextual entity can represent, for instance, a person, an organization or an event.

Suppose Alice and Bob worked on a research project together, and then wrote a paper about it; additionally, Alice prepared a spreadsheet containing experimental data, which Bob then used to generate a diagram. For the purpose of this tutorial, you can just create placeholder files for the documents:

```python
import os

data_dir = "exp"
os.mkdir(data_dir)

for filename in ["paper.pdf", "results.csv", "diagram.svg"]:
    with open(os.path.join(data_dir, filename), "w") as file:
        pass
```

Let's make an RO-Crate to represent this information:

```python
from rocrate.rocrate import ROCrate

crate = ROCrate()
paper = crate.add_file("exp/paper.pdf", properties={
    "name": "manuscript",
    "encodingFormat": "application/pdf"
})
table = crate.add_file("exp/results.csv", properties={
    "name": "experimental data",
    "encodingFormat": "text/csv"
})
diagram = crate.add_file("exp/diagram.svg", dest_path="images/figure.svg", properties={
    "name": "bar chart",
    "encodingFormat": "image/svg+xml"
})
```

We've started by adding the data entities. Now we add contextual entities representing Alice and Bob:

```python
from rocrate.model.person import Person

alice_id = "https://orcid.org/0000-0000-0000-0000"
bob_id = "https://orcid.org/0000-0000-0000-0001"
alice = crate.add(Person(crate, alice_id, properties={
    "name": "Alice Doe",
    "affiliation": "University of Flatland"
}))
bob = crate.add(Person(crate, bob_id, properties={
    "name": "Bob Doe",
    "affiliation": "University of Flatland"
}))
```

At this point, we have a representation of the various entities. Now we need to express the relationships between them. This is done by adding properties that reference other entities:

```python
paper["author"] = [alice, bob]
table["author"] = alice
diagram["author"] = bob
```

You can also add whole directories together with their contents. In an RO-Crate, a directory is represented by the `Dataset` entity:

```python
logs_dir = os.path.join(data_dir, "logs")
os.mkdir(logs_dir)

for filename in ["log1.txt", "log2.txt"]:
    with open(os.path.join(logs_dir, filename), "w") as file:
        pass

logs = crate.add_dataset("exp/logs")
```

Finally, we serialize the crate to disk:

```python
crate.write("exp_crate")
```

This should generate an `exp_crate` directory containing copies of all the files we added and an `ro-crate-metadata.json` file containing a [JSON-LD](https://json-ld.org) representation of the metadata. Note that we have chosen a different destination path for the diagram, while the paper and the spreadsheet have been placed at the top level with their names unchanged (the default).

Some applications and services support RO-Crates stored as archives. To save the crate in zip format, you can use `write_zip`:

```python
crate.write_zip("exp_crate.zip")
```

> <comment-title>How `rocrate` handles the contents of `exp/logs`</comment-title>
>
> Exploring the `exp_crate` directory, we see that all files and directories contained in `exp/logs` have been added recursively to the crate. However, in the `ro-crate-metadata.json` file, only the top level Dataset with `@id` `"exp/logs"` is listed. This is because we used `crate.add_dataset("exp/logs")` rather than adding every file individually. There is no requirement to represent every file and folder within the crate in the `ro-crate-metadata.json` file - in fact, if there were many files in the crate it would be impractical to do so.
>
> If you do want to add files and directories recursively to the metadata, use `crate.add_tree` instead of `crate.add_dataset` (but note that it only works on local directory trees).
{: .comment}

## Appending elements to property values

What ro-crate-py entities actually store is their JSON representation:

```python
paper.properties()
```

```json
{
  "@id": "paper.pdf",
  "@type": "File",
  "name": "manuscript",
  "encodingFormat": "application/pdf",
  "author": [
    {"@id": "https://orcid.org/0000-0000-0000-0000"},
    {"@id": "https://orcid.org/0000-0000-0000-0001"},
  ]
}
```

When `paper["author"]` is accessed, a new list containing the `alice` and `bob` entities is generated on the fly. For this reason, calling `append` on `paper["author"]` won't actually modify the `paper` entity in any way. To add an author, use the `append_to` method instead:

```python
donald = crate.add(Person(crate, "https://en.wikipedia.org/wiki/Donald_Duck", properties={
  "name": "Donald Duck"
}))
paper.append_to("author", donald)
```

Note that `append_to` also works if the property to be updated is missing or has only one value:

```python
for n in "Mickey_Mouse", "Scrooge_McDuck":
    p = crate.add(Person(crate, f"https://en.wikipedia.org/wiki/{n}"))
    donald.append_to("follows", p)
```

## Adding remote entities

Data entities can also be remote:

```python
input_data = crate.add_file("http://example.org/exp_data.zip")
```

By default the file won't be downloaded, and will be referenced by its URI in `ro-crate-metadata.json`:

```json
{
  "@id": "http://example.org/exp_data.zip",
  "@type": "File"
}
```

If you add `fetch_remote=True` to the `add_file` call, however, the library (when `crate.write` is called) will try to download the file and include it in the output crate.

Another option that influences the behavior when dealing with remote entities is `validate_url`, also `False` by default: if it's set to `True`, when the crate is serialized, the library will try to open the URL to add / update metadata such as the content's length and format.

## Adding entities with an arbitrary type

An entity can be of any type listed in the [RO-Crate context](https://www.researchobject.org/ro-crate/1.1/context.jsonld). However, only a few of them have a counterpart (e.g., `File`) in the library's class hierarchy, either because they are very common or because they are associated with specific functionality that can be conveniently embedded in the class implementation. In other cases, you can explicitly pass the type via the `properties` argument:

```python
from rocrate.model.contextentity import ContextEntity

hackathon = crate.add(ContextEntity(crate, "#bh2021", properties={
    "@type": "Hackathon",
    "name": "Biohackathon 2021",
    "location": "Barcelona, Spain",
    "startDate": "2021-11-08",
    "endDate": "2021-11-12"
}))
```

Note that entities can have multiple types, e.g.:

```
    "@type" = ["File", "SoftwareSourceCode"]
```

# Consuming an RO-Crate

An existing RO-Crate package can be loaded from a directory or zip file:

```python
crate = ROCrate('exp_crate')  # or ROCrate('exp_crate.zip')
for e in crate.get_entities():
    print(e.id, e.type)
```

```
./ Dataset
ro-crate-metadata.json CreativeWork
paper.pdf File
results.csv File
images/figure.svg File
https://orcid.org/0000-0000-0000-0000 Person
https://orcid.org/0000-0000-0000-0001 Person
...
```

The first two entities shown in the output are the [root data entity](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html) and the [metadata file descriptor](https://www.researchobject.org/ro-crate/1.1/metadata.html), respectively. The former represents the whole crate, while the latter represents the metadata file. These are special entities managed by the `ROCrate` object, and are always present. The other entities are the ones we added in the [section on RO-Crate creation](#creating-an-ro-crate). As shown above, `get_entities` allows to iterate over all entities in the crate. You can also access only data entities with `crate.data_entities` and only contextual entities with `crate.contextual_entities`. For instance:

```python
for e in crate.data_entities:
    author = e.get("author")
    if not author:
        continue
    elif isinstance(author, list):
        print(e.id, [p.get("name") for p in author])
    else:
        print(e.id, repr(author.get("name")))
```

```
paper.pdf ['Alice Doe', 'Bob Doe']
results.csv 'Alice Doe'
images/figure.svg 'Bob Doe'
```

You can fetch an entity by its `@id` as follows:

```python
article = crate.dereference("paper.pdf")  # or crate.get("paper.pdf")
```

# Command Line Interface

> <comment-title>Jupyter Notebook users: switch to a terminal</comment-title>
> The code cells in this section use Unix shell commands, which can't be run within a notebook. Open a Unix/Linux terminal to follow along.
{: .comment}

`ro-crate-py` includes a hierarchical command line interface: the `rocrate` tool. `rocrate` is the top-level command, while specific functionalities are provided via sub-commands. Currently, the tool allows to initialize a directory tree as an RO-Crate (`rocrate init`) and to modify the metadata of an existing RO-Crate (`rocrate add`).

```console
$ rocrate --help
Usage: rocrate [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  add
  init
  write-zip
```

## Crate initialization

The `rocrate init` command explores a directory tree and generates an RO-Crate metadata file (`ro-crate-metadata.json`) listing all files and directories as `File` and `Dataset` entities, respectively.

```console
$ rocrate init --help
Usage: rocrate init [OPTIONS]

Options:
  --gen-preview
  -e, --exclude CSV
  -c, --crate-dir PATH
  --help                Show this message and exit.
```

The command acts on the current directory, unless the `-c` option is specified. The metadata file is added (overwritten if present) to the directory at the top level, turning it into an RO-Crate.

## Adding items to the crate

The `rocrate add` command allows to add files, datasets (directories), workflows, and other entity types (currently [testing-related metadata](https://crs4.github.io/life_monitor/workflow_testing_ro_crate)) to an RO-Crate:

```console
$ rocrate add --help
Usage: rocrate add [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  dataset
  file
  test-definition
  test-instance
  test-suite
  workflow
```

Note that data entities (e.g., workflows) must already be present in the directory tree: the effect of the command is to register them in the metadata file.

## Example

To run the following commands, we need a copy of the ro-crate-py repository:

```bash
git clone https://github.com/ResearchObject/ro-crate-py
```

Navigate to the following directory in the repository we just cloned:

```bash
cd ro-crate-py/test/test-data/ro-crate-galaxy-sortchangecase
```

This directory is already an RO-Crate. Delete the metadata file to get a plain directory tree:

```bash
rm ro-crate-metadata.json
```

Now the directory tree contains several files and directories, including a Galaxy workflow and a Planemo test file, but it's not an RO-Crate anymore, since there is no metadata file. Initialize the crate:

```bash
rocrate init
```

This creates an `ro-crate-metadata.json` file that lists files and directories rooted at the current directory. Note that the Galaxy workflow is listed as a plain `File`:

```json
{
  "@id": "sort-and-change-case.ga",
  "@type": "File"
}
```

To register the workflow as a `ComputationalWorkflow`, run the following:

```bash
rocrate add workflow -l galaxy sort-and-change-case.ga
```

Now the workflow has a type of `["File", "SoftwareSourceCode", "ComputationalWorkflow"]` and points to a `ComputerLanguage` entity that represents the Galaxy workflow language. Also, the workflow is listed as the crate's `mainEntity` (this is required by the [Workflow RO-Crate profile](https://w3id.org/workflowhub/workflow-ro-crate/1.0), a subtype of RO-Crate which provides extra specifications for workflow metadata).

To add files or directories after crate initialization:

```bash
cp ../sample_file.txt .
rocrate add file sample_file.txt -P name=sample -P description="Sample file"
cp -r ../test_add_dir .
rocrate add dataset test_add_dir
```

The above example also shows how to set arbitrary properties for the entity with -P. This is supported by most `rocrate add` subcommands.


topic: fair
tutorial: fair-access
title: Access
difficulty: intermediate
time_estimation: 50M
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["fair", "dmp", "data stewardship"]
content_hash: 0aed678f65b1e64de2a3ee9b3fb68c29
last_modified: 2026-02-26T21:12:15.967296
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/fair-access/tutorial.html

content:
Data access including levels of access are described.  Learners will be able to illustrate data access in terms of the FAIR Principles using companion terms including communications protocol and authentication.  Learners will also be able to interpret the data usage licence associated with different data sets

> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}

# Data access and the FAIR Principles
Data access relates to the following 5 FAIR Principles (Table 4.1).  We will discuss and signpost these in this episode.

| The FAIR Guiding Principles |                                                                                                                                                                                                                                                                                                                                       |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| To be Findable:             | F1. (meta)data are assigned a globally unique and persistent identifier<br>F2. data are described with rich metadata (defined by R1 below)<br>F3. metadata clearly and explicitly include the identifier of the data it describes <br>F4. (meta)data are registered or indexed in a searchable resource                                   |
| To be Accessible:           | **A1. (meta)data are retrievable by their identifier using a standardized communications protocol <br>A1.1 the protocol is open, free, and universally implementable<br>A1.2 the protocol allows for an authentication and authorization procedure, where necessary <br>A2. metadata are accessible, even when the data are no longer available** |
| To be Interoperable:        | I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. <br>I2. (meta)data use vocabularies that follow FAIR principles<br>I3. (meta)data include qualified references to other (meta)data                                                                                         |
| To be Reusable:             | R1. meta(data) are richly described with a plurality of accurate and relevant attributes <br>**R1.1. (meta)data are released with a clear and accessible data usage license**<br>R1.2. (meta)data are associated with detailed provenance<br>R1.3. (meta)data meet domain-relevant community standards        


Table 4.1: The 15 FAIR Guiding Principles.  Principles relating to data access are highlighted in **black**.

# What is data access?
Making data accessible means it can be made available to use by both humans and computers, though FAIR data does not necessarily mandate that all data is openly accessible and we discuss this in a minute.

As outlined in the Accessibility Principles (meta)data must be retrievable by their identifier using a standardised communication protocol (FAIR Principle A1) which is open, free and universally implementable (FAIR Principle A1.1).

A standardised communication protocol is something like http(s) or FTP that allows data to be requested and downloaded, for example, by clicking on a link on a webpage.  Simply put, **a protocol** is a method that connects two computers and ensures secure data transfer.  Web browsers such as Firefox and Chrome can use the http(s) communication protocol since it is universally implementable, open and free.  

It is important to note that (meta)data access is not limited to humans clicking links on webpages. For a machine ‘user’, examples of accessing data include the use of an ‘application programming interface’ (API), and Unix command line, using  wget and [curl](https://curl.se/docs/manpage.html).

We mentioned above that FAIR data does not necessarily mandate open data.  Commonly, this relates to controlled access, which is discussed later in this Episode, but in the exercise below we investigate a dataset that has been deleted where it has been previously accessible.  FAIR states: _“metadata are accessible even when the data are no longer available”_ (FAIR Principle A2) which is one of the few principles that relate solely to metadata (and not data).  This means data can be deleted from its original online location at any time, but its original metadata must remain accessible.  PIDs (usually connected through URLs) to the original data must remain live, though the record may, for example, change to display metadata only. This is useful where originally referenced data changes over time, or becomes obsolete or deprecated; a record of the original metadata, and where appropriate reasons for its removal, or redirection to updated records, provide a provenance trail from the original data, which may have been referenced, for example, in publications, where it is not possible to update the PID.  


> <question-title></question-title>
>
> The [Protein Data Bank in Europe](https://www.ebi.ac.uk/pdbe/) (PDBe) is a searchable repository of biological macromolecular structures.  Please take a look at the following record that has been retired: [1ins](https://www.ebi.ac.uk/pdbe/entry/pdb/1ins).  You will see that the crystal structure is no longer available, but what metadata is available?
>
> > <solution-title></solution-title>
> >
> > Metadata includes the original citation, an explanation of why the record is no longer available and a redirection to the replacement  entry.
> >
> {: .solution}
{: .question}

# Types of data access

Where restricted access is required (for example, sensitive data or data subject to intellectual property) generally only the recommended parts are restricted, whereas associated metadata and non-sensitive data are openly accessible.  As part of the FAIR Principles, terms of access need to be stated, usually as part of a data licence (FAIR Principle R1.1).  Remember, FAIR data is **as open as possible, and as closed as necessary**.
Resources for working with sensitive data are given at the end of this Episode.

There are four types of data access as described in [RDMkit](https://rdmkit.elixir-europe.org/sharing#what-should-be-considered-for-data-sharing):

**Open access**: Anyone can access the data, and use it for any purpose.

**Registered access or authentication procedure**: researchers are required to register and authenticate to have the right to access the data (login and password)

**Controlled access or Data Access Committees (DACs)**: researchers will apply for access, and their application reviewed by a data access committee

**Access upon request (not recommended)**: a researcher provides their contact details for access. Contact details should be provided in the metadata which will be publicly available.

Any access requiring login and password makes use of _“[a protocol] allowing for an authentication and authorisation procedure, where necessary”_ (FAIR Principle A1.2).  Commonly for authentication, a researcher will be assigned a unique ID, or the system may support sign-in with an ORCID ID, which is the case of many data repositories including [Zenodo](https://zenodo.org/).  In some cases, there may be an option to use a Google account or institutional email to sign in, and many infrastructures also support ‘single sign in’.

# Data usage licence

A data usage licence describes the legal rights on how others use your data.  When you publish your data, you should describe clearly in what capacity your data can be used. 

There are many types of licences that can be used, including the  [MIT licence](https://opensource.org/license/mit/) (for software) or the commonly used [Creative Commons licences](https://creativecommons.org/licenses/) (a selectable collection of licences). These licences provide precise descriptions of the rights to data use, where the latter defines rights for sharing, adapting and commercialisation. Open access data usually carries the [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) or [CC0](https://creativecommons.org/share-your-work/public-domain/cc0/) licence permitting open sharing and adaptation, even for commercial purposes.  The licence is applied by adding the licence declaration to the data similar to this training page.  Take a look at the banner at the bottom of this page.  It states: Licensed under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) 2018–2023 by [The Carpentries](https://carpentries.org/). 

# Making sensitive data accessible

Controlled access is often afforded to sensitive data or commonly any data that could potentially harm, see [RDMkit](https://rdmkit.elixir-europe.org/sensitive_data#is-your-data-sensitive) for a full definition.

Where access is granted, sensitive data is often de-identified, meaning that identifying (meta) data is removed or reassigned, leaving the analytics-appropriate component.

For example, a participant’s name can be removed from a questionnaire, and their home address can be substituted for the name of the town they live in.  This anonymises the data since the participant can no longer be located.  Note though that other information in the questionnaire could compromise this.  If other data reveal that the participant won the town’s 10K road race in 2023, we could potentially identify the individual using the name of the town and an online search.  If more information in the questionnaire states that the participant has a rare disease, we are broaching disclosure of sensitive, personal data.

Though people refer to anonymisation when de-identifying (meta)data, often they mean pseudonymisation.  Data anonymisation and pseudonymisation are slightly different.

**Data anonymisation** is the process of irreversibly de-identifying personal data such that an individual cannot be identified by anyone, including the study team and the individual themselves.  If data are anonymised, no one can link data back to the subject. 

**Pseudonymisation** is more commonly used.  It is a process where identifying-fields are replaced by artificial identifiers called pseudonyms or pseudonymised IDs.
Commonly, a person’s name or medical ID will be replaced with a unique participant ID within the study.
Pseudonymisation ensures no one can link data back to the individual, apart from nominated members of the study team who will be able to link pseudonyms to identifying information, such as medical records.


> <question-title></question-title>
>
> Use RDMkit's guidelines on [sensitive data](https://rdmkit.elixir-europe.org/sensitive_data) to familiarise yourself further on de-identification of data. What further training can you identify?
>
> > <solution-title></solution-title>
> >
> > At the bottom of the page, under “Training”, useful resources are given.  The [TeSS training portal](https://tess.elixir-europe.org/) permits users to search for courses, events, videos and other learning material for data in the life sciences.
> >
> {: .solution}
{: .question}


# Useful resources
- More about Data Access and Sharing: [RDMkit](https://rdmkit.elixir-europe.org/sharing), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/accessibility.html)
- Definition and examples of Standardised Communication protocol using authentication: [Australian Research Data Commons](https://ardc.edu.au/resource/standardised-communications-protocols/)
- More about Data licensing: [RDMkit](https://rdmkit.elixir-europe.org/licensing), [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/recipes/reusability/ATI-licensing.html)
- The [Creative Commons licences](https://creativecommons.org/choose/)
- Working with sensitive data: [EBI training](https://www.ebi.ac.uk/training/events/working-sensitive-data/)


topic: fair
tutorial: dataplant-arcs
title: DataPLANT ARCs
difficulty: intermediate
time_estimation: 2H
tools_json: []
requirements_json: ["type: \"internal\""]
tags_json: ["plants", "fair", "data stewardship"]
content_hash: 0e826ef4d89389dd0d3395175c4fc69c
last_modified: 2026-02-26T21:12:15.954674
zenodo_link: https://zenodo.org/records/13970369
workflows_json: []
url:
https://training.galaxyproject.org/training-material/topics/fair/tutorials/dataplant-arcs/tutorial.html

content:
In this tutorial we will guide you through the process of creating your ARC (Annotated Research Context)
ARCs are a part of your Research Data Management (RDM) strategy, and are aimed specifically at plant research
data. ARCs are an [RO-crates](https://www.researchobject.org/ro-crate/) implementation developed by
[DataPLANT (NFDI4Plants)](https://www.nfdi4plants.org/). ARCs allow you to bundle and annotate
your research data in a standardized and [FAIR](https://www.go-fair.org/fair-principles/) (Findable, Accessible,
Interoperable, Reusable) manner.

![ARC concept showing bundling of experimental data, computation, and annotations](images/arcs-intro.png)


> <agenda-title></agenda-title>
>
> In this tutorial, we will cover:
>
> 1. TOC
> {:toc}
>
{: .agenda}


# Getting set up for the tutorial

For this tutorial you need three things:
 1. An account on [DataHUB](https://git.nfdi4plants.org)
 2. The [ARCitect](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/) tool installed
 3. Example data to fill your ARC with

Below we will walk you through each of these 3 setup steps.

## Create a DataHUB account

DataHUB is a GitLab instance designed to hosts ARCs. Here you can work collaboratively to create your ARC, and once you are ready to publish your ARC, you can do this from here as well.

> <hands-on-title> Create a DataHUB account </hands-on-title>
>
> 1. Already have an account? Please [log in](https://git.nfdi4plants.org/])
> 2. New to DataHUB? Create an account: [https://git.nfdi4plants.org/](https://git.nfdi4plants.org/)
>    - If you are not familiar with Git or GitLab that is ok, ARCitect will take care of syncing to DataHub behind the scenes
>
{: .hands_on}


## Download demo data

In this tutorial we will create an ARC for an example research data. In this example investigation we grew Talinum plants under
drought conditions, with watered plants as control. Under these conditions the plants switch their type of photosynthesis
(water: C3, drought: CAM). To study this change, we profiled gene expression (Transcriptomics) using RNASeq, and metabolites
(Metabolomics) via GC-MS (Gas-Chromatography Mass Spectrometry)


> <hands-on-title> Download example ARC data </hands-on-title>
>
> 1. Download the zip file from Zenodo (<2MB) to your machine
>
>    ```
>    https://zenodo.org/records/13970369/files/GTN_ARC_tutorial_data.zip
>    ```
>
> 2. Unzip the file
>    - You can put the files wherever is most convenient for you
>
> 3. Have a quick look at the files and folders
>    - This example data mimics how you might have your project data organised on your computer
>    - In the next part we will show how to organise this data in an ARC-compatible way
>
{: .hands_on}


## Install ARCitect

Now we will install the ARCitect tool. This tool will help you create and fill your ARC.


> <hands-on-title> Install the ARCitect tool on Linux  </hands-on-title>
>
> 1. Follow the installation instructions on the [DataPLANT knowledgebase](https://nfdi4plants.org/nfdi4plants.knowledgebase/) for your OS:
>    - [Windows Installation Instructions](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/installation/#tab-21)
>    - [macOS Installation Instructions](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/installation/#tab-22)
>    - [Linux Installation Instructions](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/arcitect/installation/#tab-23)
>
>    > <comment-title> ARCitect version </comment-title>
>    >
>    > This tutorial uses version {{page.extra.arcitect_version}} of ARCitect. Since this tool is still
>    > under rapid development, a newer version of ARCitect may already be available. Feel free to install
>    > the latest version of ARCitect, but be aware that some screenshots and instructions may be outdated.
>    >
>    {: .comment}
>
>
>
> 2. Start ARCitect to verify that everything is installed correctly. You should see a screen like this:
>
>    ![screenshot of the ARCitect interface after startup](images/arcitect-home.png)
>
{: .hands_on}


# Initialize your ARC structure

Now that you have everything set up for the course, we can start creating our ARC. First we will organize our example data into the ARC structure, then we will add structured metadata describing our experiments and data.


## The ISA data model

ARCs build on the [ISA Abstract Model](https://isa-specs.readthedocs.io/en/latest/isamodel.html)
for metadata annotation.

The ISA model comes with a hierarchy (ISA: Investigation - Study - Assay)
that aligns well with most projects in (plant) biology labs. It allows grouping multiple assays (measurements) to one study,
and multiple studies to one investigation.

![Overview of the ISA model](images/isa-model.png "Image source left panel: [isa-tools.org](https://isa-tools.org/format/specification.html)")

Your ARC has one isa.investigation.xlsx workbook at its root, where metadata about the investigataion is captured. Similarly, each study or assay that you add to your ARC contains one isa.study.xlsx or isa.assay.xlsx, respectively.


> <comment-title> Before you start </comment-title>
>
> Before creating your own ARC, invest some time to think about the following questions.
>
> - What is my **investigation**?
> - What is my **study**?
> - Which **assay** did I perform?
> - What is my (raw) **dataset**?
> - What **protocols** did I use?
>
{: .comment}


## The ARCitect interface

The ARCitect window consists of 4 main parts; on the left is the menu panel, next to it is a panel displaying the ARC folder structure, then the main panel where we will configure our ARC, and on the righ is (optionally) a help panel.

![Screenshot of the ARCitect graphical user interface](images/ARCitect-GUI.png "Overview of ARCitect interface. A) menu panel, B) ARC folder structure, C) Main panel, D) Help panel (off by default).")

Let's have a closer look at the menu panel:

![screenshot of the ARCitect menu panel](images/arcitect-sidebar.png)

This menu bar allows you to:
1. **Login**: Log in to DataHUB (required to sync your ARC to DataHUB)
2. **New ARC**: Create a new ARC
3. **Open ARC**: Open an existing ARC from your machine
4. {% icon download-cloud %} **Download ARC**: From DataHUB to your machine (e.g. from a colleague or published)
5. {% icon save %} **Save ARC**: Save your changes to your machine
6. **Explorer**: Open the local ARC folder on your computer
7. **Commit**: prepare your ARC changes for sync to DataHUB
8. **DataHUB sync**: push your comitted changes to DataHUB
9. **History**: Show previous versions of your ARC
10. **Validation**: Configure validation (checks) for our ARC
11. **Services**: Check on the status of ARC services (e.g. is DataHUB down for maintenance?)
12. **Settings**: For example turn on tool tips or the help panel



## Creating a new ARC

Let's start by creating a new ARC.

> <hands-on-title> Create a new ARC in ARCitect </hands-on-title>
>
> 1. In the left-hand panel of ARCitect, select **New ARC**
>    - Navigate to a location on your machine where you would like to create the ARC folder
>    - Choose a descriptive name for your ARC, e.g. `TalinumPhotosynthesis`
>
>    > <comment-title>What's in a name?</comment-title>
>    >
>    > By default, the name give your ARC here will be used for:
>    >  - the ARC folder on your machine
>    >  - the ARC repository on DataHUB at `https://git.nfdi4plants.org/<YourUserName>/<YourARC>` (see next steps)
>    >  - the identifier for your investigation
>    >
>    > **Note:** Make sure that no ARC exists at `https://git.nfdi4plants.org/<YourUserName>/<YourARC>`. Otherwise you will sync to that ARC.
>    {: .comment}
>
> 2. If all went well, you should see the folder structure that was created for your ARC:
>    ![screenshot of the ARC folder panel after initializing our ARC](images/new-arc.png)
>
> 3. Click on the **Explorer** button in the menu to see the files created on your machine
>
{: .hands_on}

The basic ARC structure is initialized for you, and looks something like this:

```bash
my-arc
├── studies
│   ├── study1              # subfolder for each study
│   │   ├── protocols       # folder for protocol documents
│   │   ├── resources       # folder for any other supporting information and files
│   │   isa.study.xlsx      # study-level metadata (e.g. sample sheet, culturing/harvesting conditions)
│   │   README.md           # Free-text description of your study
│   ├── assay2
│   │   ├── ...
├── assays
│   ├── assay1              # subfolder for each assay
│   │   ├── dataset         # assay result files (e.g. sequencing files)
│   │   ├── protocols       # assay protocol files
│   │   isa.assay.xlsx      # assay-level metadata (e.g. instrument)
│   │   README.md
│   ├── assay2
│   │   ├── ...
├── runs                    # analysis results
│   ├── ...
├── workflows               # analysis workflows (CWL, or any other scripts/code used)
│   ├── ...
├── isa.investigation.xlsx  # investigation-level metadata
└── README.md
```


## Add investigation-level metadata

We will start by adding some basic data about our ARC. We do this in the investigation-level metadata.

> <hands-on-title>Adding investigation information</hands-on-title>
>
> 1. In the ARC folder panel (second panel from the left), click on the investigation
>    - This is the root of the ARC folder structure (e.g. *TalinumPhotosynthesis*)
>    - This will bring up the investigation information in the main panel
>    - Notice that the Identifier has already been filled in for you (but you can still change it)
>    - Behind the scenes this information will be saved as an Excel sheet `isa.investigation.xlsx`
>
>    ![screenshot of the investigation metadata screen](images/investigation-empty.png)
>
> 2. **Add a title** for your investigation
>    - This should be a concise but descriptive title
>    - Tip: think of what you would name your publication based on this investigation
>    - For this tutorial, you could set your title to `Effect of Drought on Talinum Photosynthesis`
>
> 3. **Add a description** for your investigation
>   - Here you can describe your main research questions and methods in a few sentences or paragraphs.
>   - Tip: think of this as a short abstract like you would provide in a publication
>   - For this tutorial you can add any text here
>
> 4. **Add an ARC Contact** (you)
>    - Click on the **{% icon plus %} (plus) icon** under Contacts
>    - Click on the **{% icon dropdown %} (dropdown) icon** on the right of the contact to expand it
>    - Fill in at least your First name, Last name, and email.
>    - **Tip:** if you have an [ORCiD](https://orcid.org) ID, fill that in and hit "Search" to autofill these fields
>
> 5. Your investigation tab should look something like this
>
>    ![screenshot of investigation metadata filled in](images/investigation-filled.png)
>
{: .hands_on}


## Saving your ARC

It is a good idea to frequently save your work. There are 2 levels of saving your ARC; you can:
1. Save it locally (to your computer)
2. Save it online to DataHUB (syncing)

By default your ARC on DataHUB is private to you, but you have the option of sharing it with your collaborators.

> <comment-title>Why and how often to sync to DataHUB?</comment-title>
>
> Syncing your ARC to DataHUB frequently has several advantages:
>  - It makes it easy to collaborate with colleagues on your ARC
>  - It is a backup of your ARC in case anything happens to your computer
>  - It is version controlled. This means you can always go back to any previous version your ARC if needed
>
> We advise you to save your ARC to your local machine as often as you can, and sync to DataHUB every time you have
> completed a unit of work (e.g. "added data files", "added culturing metadata", "added protocol documents" etc), this
> makes it easy for others (and future you) to see what you have changed.
>
{: .comment}

Since we have just completed our first unit of work (initialized our ARC and added basic investigation metadata), let's save and sync our ARC to DataHUB now.

> <hands-on-title>Save and Sync our ARC</hands-on-title>
>
> 1. Save locally simply by hitting the **Save ARC** button on the left
>    - That's it! Too easy!
>    - Do this as often as possible to avoid losing work
>
> 2. Click **Login** at the top of the menu panel
>    - Select `git.nfdi4plant.org` as your host
>    - Click the **Login** button
>    - Since you already logged into DataHub via your browser earlier, ARCitect should now automatically log you in
>
> 3. To check that you are successfully logged in, the Login button should now be replace with your DataHub username
>    ![screenshot of the menu panel as it looks when you are logged in](images/arcitect-loggedin.png)
>
> 4. Click on **Commit** in the menu
>    - This screen will show you all files that have chanced since your last sync
>    - You can choose whether to sync all files, or only some
>    - Choosing which files to sync is called "committing" those file.
>    - In our case only one file has changed, `isa.investigation.xlsx`. By default it is already selected to be committed
>
> 5. Add a **commit message**
>    - in a few words, describe your changes
>    - this will help you and your colleagues
>    - For example, for this changes you can write `added investigation metadata`
>    - You can leave the rest of the fields to the default values
>
>    ![screenshot of the commit menu](images/arcitect-commit.png)
>
> 6. Click the **Commit** button
>    - There will be a confirmation screen like the one below
>    - Click the **Ok** button
>
>    ![Screenshot of the commit confirmation dialogue window](images/arcitect-commit-confirmation.png)
>
> 7. Click on **DataHUB Sync** in the menu
>
>    ![Screenshot of the DataHUB sync menu](images/arcitect-sync.png)
>
> 8. Click on **+ (plus)** to add a 'remote' (DataHUB location to save your ARC)
>    - you only have to do this once
>    - you can keep all the default settings
>    - Click **Add**
>
>    ![screenshot of the remote adding dialogue window](images/arcitect-add-remote.png)
>
> 9. Click **Push** to sync from your computer to DataHUB
>    - **Pull** will do the reverse, it will 'pull' in changes from DataHUB into your local copy of the ARC,
>      for example if a collaborator made changes, or you worked on it from a different machine.
>
> 10. Check that the push was successfull
>     - Navigate to `https://git.nfdi4plants.org/YourUsername/TalinumPhotosynthesis`
>     - Replace 'YourUsername' with your DataHUB username in the URL
>     - If you used a different Identifier for your ARC, replace 'TalinumPhotosynthesis' with your ARC Identifier in the URL
>     - You should see your ARC on DataHUB:
>
>     ![screenshot of your new ARC on DataHUB](images/datahub-firstpush.png)
>
{: .hands_on}

Congrats! You have initialized, saved, and synced your ARC! Now let's continue with the good part, actually filling your ARC.


# Structure your data in the ARC

## Think about your data

Next we will divide our research into studies and assays. As a rule of thumb, create a study for every research question, and an assay for every measurement you performed (e.g. sequencing, mass-spec, imaging). An investigation can have one study, or several. A study can have one or more assays, and each assay can be associated with one or more studies.


> <hands-on-title> Evaluate the example data </hands-on-title>
>
> 1. Recall our example research:
>
>    *In this example investigation we grew Talinum plants under drought conditions, with watered plants as control.
>    Under these conditions the plants switch their type of photosynthesis (water: C3, drought: CAM). To study this
>    change, we profiled gene expression (Transciptomics) using RNASeq, and metabolites (Metabolomics) via GC-MS
>    (Gas-Chromatography Mass Spectrometry)*
>
> 2. Have a look at the demo data we downloaded
>
> > <question-title> Which studies and assays do we have? </question-title>
> >
> > 1. Which studies do we have?
> > 2. Which assays did we perform?
> >
> > > <solution-title></solution-title>
> > > 1. We had one research question, the effect of drought on photosynthesis, so we will create 1 study
> > > 2. We performed 2 measurements, RNA sequencing and GC-MS, so we will create 2 assays in our ARC
> > {: .solution}
> {: .question}
{: .hands_on}

## Create a study

Let's create our study now.

> <hands-on-title>Create an ARC study</hands-on-title>
>
> 1. Click on the **{% icon plus %} (plus) icon** next to the studies folder
> 2. Provide a study identifier, e.g. `talinum_drought`
>    ![screenshot of the study creation dialogue window](images/arcitect-new-study.png)
> 3. Click **{% icon plus %} New Study** button to confirm
> 4. **Expand** the new study folder that was created in your ARC
>    - you should see the protocols and resources folders created
>      ![screenshot of the ARC structure after adding the new study](images/arcitect-new-study-files.png)
{: .hands_on}

As we did for the investigation, we can also add some basic study information

> <hands-on-title>Add basic study metadata</hands-on-title>
>
> 1. Click on your study `talinum_drought`
> 2. The main panel will now show study metadata, here you can add basic information about your study
>    such as description, contacts, associated publications.
>
>    ![screenshot of the study metadata screen](images/arcitect-study-metadata.png)
>
> 3. **Add a description** for the study
> 4. **Add yourself** as a contact
>    - Remember that you can use your ORCiD ID for a fast way to fill this data
> 5. **Add a publication**
>    - Normally you would add the paper you published based on the data in this ARC
>    - For this tutorial, add any publication (e.g. one of yours, or one you find interesting)
>    - The easiest way to do this, is to add the **DOI link** or **Pubmed ID**, then the information will be autofilled
>    - If you don't have a publication in mind, try one of the following papers:
>      - Pubmed ID: `7825135`
>      - DOI: `https://doi.org/10.1136/bmj.331.7531.1498`
>
>    ![Example of study metadata filled](images/arcitect-study-metadata-filled.png)
{: .hands_on}

Since we have once again completed a unit of work, let's save and sync our ARC again!

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
>    - **Save ARC** button in menu
> 2. Sync your ARC to DataHUB
>    - **DataHub Sync** button in menu
>    - Commit all changed files
>    - Provide a good commit message (e.g. `added drought study`)
{: .hands_on}


## Create an assay

Next, we will create our 2 assays. Recall that we performed two measurements/assays, RNA sequencing, and metabolomics.

> <hands-on-title>Create the RNA-seq assay</hands-on-title>
>
> 1. Click on the **{% icon plus %} (plus) icon** next to the assays folder
> 2. Give your assay a name, e.g. `rnaseq`
> 3. In **measurement type** field, choose `mRNA Sequencing`
>    - this field is **ontology based**
>    - as soon as you start typing, suggetions for known terms will pop up
>    - Notice the TSR and TAN fields are automatically filled in once you select a term. These fields help make your ARC machine readable.
>
> 4. Similarly, set the  **Technology Type** to `Next-generation Sequencing`
>
>    > <question-title></question-title>
>    > Look through our demo data, can you find out which sequencing machine was used?
>    > > <solution-title></solution-title>
>    > > In the example data, we've got a **methods** folder where we've stored notes on our assays,
>    > > In the file called **illumina_libraries** we see that the sequencer we used was a `Illumina HiSeq 2000`
>    > {: .solution}
>    {: .question}
>
> 5. Under **Technology platform**, fill in `Illumina HiSeq 2000`
>
>    ![screenshot of assay metadata filled](images/arcitect-assay.png)
>
{: .hands_on}


Awesome, now let's repeat the process for our metabolomics assay.

> <hands-on-title>Create the metabolomics assay</hands-on-title>
>
> 1. Create a new assay called `metabolomics`
> 2. Under **measurement type** choose `Gas Chromatography Mass Spectrometry`
> 3. We can leave the Technology type/platform fields empty for now
>
>   ![screenshot of metabolomics assay](images/arcitect-assay2.png)
{: .hands_on}


And once again, let's save our ARC and sync it to DataHUB.

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
>    - **Save ARC** button in menu
> 2. Sync your ARC to DataHUB
>    - **DataHub Sync** button in menu
>    - Commit all changed files
>    - Provide a good commit message (e.g. `added rnaseq and metabolomics assays`)
{: .hands_on}



# Adding files to your ARC

As our next step, let's add the files we have in our demo folder to our ARC.

## Add protocol documents

> <hands-on-title>Adding protocols</hands-on-title>
>
> 1. Open the folder with demo data on your computer
> 2. Look through the **methods** folder
>    - This contains files with notes about methods we used in our research
>    - In our ARC, these usually go in **protocols** folders
>    - Studies and assays both have protocols folders
>
>    > <question-title></question-title>
>    >
>    > Where should each of the 4 files in the methods folder go in our ARC?
>    >
>    > > <solution-title></solution-title>
>    > > - `plant_material.txt`: contains a general sample sheet, and should go in the study-level protocol folder
>    > > - `RNA_extraction.txt` and `Illumina_libraries.txt` are specific to the RNA sequencing assay, so should go in the protocols folder for the rnaseq assay
>    > > - `metabolite_extraction.txt` is specific to our metabolomics assay, so should go in the metabolomics protocol folder
>    > {: .solution}
>    {: .question}
>
> 3. There are two ways to add files to your ARC, we will do both now, in the future you can choose whichever option you prefer
>
> 4. **Adding files via ARCitect**
>    - Expand the `studies` folder
>    - Expand the `talinum_drought` folder
>    - Right-click on the `protocols` folder
>    - Choose `Import files` option
>    - Browse to the demo data and choose the `methods/plant_material.txt` file
>
>    ![screenshot of menu to add documents](images/arcitect-import-files.png)
>
> 5. **Adding files via file explorer**
>    - Click on `Explorer` in the left-hand menu
>    - Open the `assays/metabolomics/protocols` folder of your ARC
>    - In another window, open the demo data folder
>    - Copy or drag the `metabolite_extraction.txt` file from the demo data folder to your ARC folder
>
> 6. **Add the two remaining protocol files** to the ARC in your preferred way
>    - Files `illumina_libraries.txt` and `RNA_extraction.txt`
>    - Both should go to `assays/rnaseq/protocols/` folder in your ARC
>
{: .hands_on}

Note that adding files via ARCitect import option will create a copy of the file, rather than moving it, so if you have large data files you might want to move the files via the explorer option.

## Add assay files

Next, let's add the RNA-seq data we have to the ARC structure.

> <hands-on-title>Add RNA-seq data</hands-on-title>
>
> 1. Look in the `rnaseq_data` folder of the demo data
>    - Files with the `.fastq.gz` suffix are raw sequencing files
>    - Raw data goes in the `assays/rnaseq/dataset/` folder
>    - `NGS_samplesheet.xlsx` can go into the assay's protocol folder
>
> 2. Move the files to the ARC in your preferred way
>
>    ![screenshot of arc explorer panel after adding files](images/arcitect-assay-files-rnaseq.png)
>
{: .hands_on}

Finally, let's do the same for our metabolomics data

> <hands-on-title>Add Metabolomics data</hands-on-title>
>
> 1. Look in the `metabolomics_data` folder of the demo data
>    - Folders with the `.D` suffix are raw data files
>    - The rest of the files can go to the protocols folder
>
> 2. Move all the files to the ARC metabolomics assay folder
>    - Note that if you move them via ARCitect, you should use the **Import Directories** option for the raw data folders
>
>    ![screenshot of arcitect files structure after adding metabolomics files](images/arcitect-assay-files-metabolomics.png)
>
{: .hands_on}



Congrats! you have added data to your ARC! As always, let's make sure to save & sync our work before we continue.

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
> 2. Sync your ARC to DataHUB
>    - Provide a good commit message (e.g. `added protocols and raw data files`)
{: .hands_on}


# Add your experimental metadata (SWATE)

Now that we have all our data in the ARC, we need to add metadata about our research in a structured way.
We already put a lot of information about our samples and experiments in the protocols folders,
but these are just free-text notes, like we would put in a methods section of a paper, or in our lab notebook.
The information is there but it is not **structured** yet.

In this section, we will show you how you can add structured metadata to your ARC. We do this by using a
standard format ([isa-tab](https://isa-specs.readthedocs.io/en/latest/isatab.html)), combined with
ontology-based annotations. We call these **annotation tables**, and these tables with metadata can be created
directly within the ARCitect tool.

> <comment-title>SWATE</comment-title>
>
> If you have previously already looked into ARCs, you may remember a tool called SWATE.
> SWATE helps you create these annotation tables. SWATE has now been integreted into ARCitect
>
{: .comment}

## Add sample information to the study metadata

First, we will add information about our samples (plants) to our study.

We took some notes about our samples in `plant_material.txt` file (now in our protocols folder)

```
Plant Material and Growth Conditions

Plants:
  - Species: Talinum fruticosum
  - Age: 28-d-old

Growth Chamber
  - 12 h light (30°C, 37% relative humidity)/12 h dark (22°C)
  - 425 µEinstein

Drought treatment:
  - no water for 12 days
  - re-water for 2 days

Sampling:
  - Leaves: topmost mature unshaded leaves, ca. 3–4.5 cm (see pictures)
  - shock-frozen in liquid N2

drought samples (CAM)
  - DB_097
  - DB_099
  - DB_103

re-watered samples (C3)
DB_161
DB_163
DB_165
```

> <tip-title> Viewing and editing text documents in ARCitect</tip-title>
>
> Text files that we added to our ARC, can also be viewed and edited from within ARCitect
>
> 1. Expand the talinum_drought study
> 2. Expand the protocols folder
> 3. Click on the `plant_material.txt` file
>
>    ![screenshot of markdown editor in ARCitect](images/arcitect-markdown-editor.png)
> 4. From this screen you can edit the text as needed.
>    - text files support [Markdown syntax](https://www.markdownguide.org/cheat-sheet/)
{: .tip}

### The ISA-Tab format

We will now take this free-text description of our samples, and turn them into a structured annotation table
using ARCitect.

At it's core, an annotation table often looks like:

Source Name | .. | .. | Sample Name
------------|----|----|-----------
plant 1     | .. | .. | sample1
plant 1     |    |    | sample2
plant 2     |    |    | sample3
...         |    |    | ...

The **Source Name** column and lists the sources (plants) you used in your experiments.
The final column, **Sample Name** describes the samples derived from the sources (plants).
Then there can be any number of columns in between describing characteristics of the plants (e.g. species, location, conditions, etc).
Which columns appear here depends on your experimental setup, and can be chosen by you.

These sample names you provide here can be referenced in other annotation tables in the study, or in assay-level annotation tables.
ARCitect uses these sample identifiers to put information from different annotation tables together, so make sure you use
consistent spelling and capitalization.

### Create an annotation table from scratch

There are 2 ways to add annotation tables, you can either:
1. build them up by hand, choosing the columns ("Building Blocks") yourself.
2. use a pre-defined template. These templates are either provided by the DataPLANT core team, or by other communities that use ARCs (or you!).

We will show both ways of adding annotation tables in this section of the tutorial.

First, let's add our sample information as an annotation table the manual way. Since this information pertains to the study as a whole,
not just a single assay, we will add our sample sheet information at the study level.

> <hands-on-title>Initialize smple information annotation table </hands-on-title>
>
> 1. Click on your study name in the ARC structure
>    - This should bring up the study metadata we filled in previously
> 2. At the bottom of the main panel, click on the **{% icon plus %} (plus) icon**
>    - the icon is located next to the current tab named *Metadata*
>
>    ![screenshot showing the button for adding an annotation table](images/arcitect-annotation-table-add.png)
>
>    - You should then see a new tab named **NewTable0** appear
>
>    ![screenshot after adding the new annotation table](images/arcitect-annotation-table-new.png)
>
> 3. **Rename** the table to **Sample Information**
>    - by right-clicking on the tab name **NewTable0**
>    ![screenshot of renaming option](images/arcitect-annotation-table-rename.png)
>
> 4. In the center panel, choose **Or start from Scratch!**
>
> 5. We fill first add our **Source Name** column
>    - In the dropdown, select **Input**
>    - Next select **Source Name**
>    - Click **Add Column**
> 6. Next, add an output column **Sample Name**
>    - Click on **Add building block** icon at the top of the table
>    - In the dropdown, select **Output**
>    - Next select **Sample Name**
>    - Click **Add Column**
>
>    ![screenshot of the two added columns](images/arcitect-annotation-table-start.png)
>
> 7. Let's add a column for the species as well
>    - Click on the **Add building block** button
>    - Change the dropdown to **Characteristic**
>    - In the search bar next to it, type **Species**
>    - Select the **NCIT** ontology term
>
> 8. The column probably appear on the right
>    - It is customary to keep the output column on the right
>    - Right-click on the species column
>    - Select **Move column**
>    ![screenshot of the move column option](images/arcitect-annotation-table-move-column.png)
>    - Change the column to `1`
>    - Click **Apply** to get a preview of the new order
>    - Click **Submit** to apply the change
>    ![screenshot of final move dialogue window](images/arcitect-annotation-table-move-column2.png)
>    - You should now have 3 columns, with species column in the middle
>
> 9. To add data, we have to **add rows**
>    - We had 6 samples, so we will add 6 rows
>    - At the bottom, change the 1 to a 6, and click on the {% icon plus %} (plus) button to add the rows
>    ![screenshot of the add rows button](images/arcitect-annotation-table-add-rows.png)
>    - You should now have 6 rows in your table
>    ![annotation table with rows added](images/arcitect-annotation-table-withrows.png)
>
{: .hands_on}

Now that we have our basic table initialized, let's start adding some data. For our example study, we grew 6 plants
(we will label them `DB001` to `DB006`). We grew 3 plants in drought conditions, and 3 controls. From our `sample_information.txt`
file we see that the samples we derived from these plants had IDs DB_097, DB_099, DB_103 (drought/CAM), and DB_161, DB_163, DB_165
(controls/re-watered/C3). All plants are species *Talinum fruticosum*.

Let's fill all this information in our annotation table.

> <hands-on-title>Fill the annotation table</hands-on-title>
>
> 1. Fill the first column (**Source Name**) with our 6 plant names
>    ```
>    DB001
>    DB002
>    DB003
>    DB004
>    DB005
>    DB006
>    ```
>    > <tip-title>Copying a whole column at once </tip-title>
>    > You can copy the values one by one, but you can also copy the whole column at once.
>    > - Copy the IDs above
>    > - In your ARC table, Select the first cell of the *Source Name* column
>    > - Hold down the shift key
>    > - Select the last (6th) cell of your column
>    > - Paste the IDs (`CTRL+V`)
>    {: .tip}
>
> 2. Fill the last column (**Sample Name**) with our 6 sample names
>    ```
>    DB_097
>    DB_099
>    DB_103
>    DB_161
>    DB_163
>    DB_165
>    ```
> 3. Now we will fill in the Species column
>    - Select the first cell in the Species column
>    - Start typing the species name `Talinum fruticosum`
>    - Since this is an ontology column, you will get suggestions as you type.
>    - In our case, we see NCBI ontology term for our species, please select this value.
>    ![screenshot of ncbi term in search](images/arcitect-species-search.png)
>
> 4. Copy the species value to the enitre column
>    - Right-click on the species term you just added in the firs row
>    - Select **Fill column** to apply this term to all 6 rows.
>    ![screenshot of our table after these steps](images/arcitect-annotation-sample.png)
{: .hands_on}

This is the basic process of building our annotation table. We can now add more building blocks (columns) as needed,
for all the information we have about our plants that we wish to describe. In the next section, we will add more columns.

But first, let's have a closer look at our annotation table.

> <comment-title> Why do we call them Building Blocks?</comment-title>
>
> 1. Click on the (expand) icon next to the species column name
>    - You will now see additional hidden columns
>    - These columns provide information about the ontology term you chose (TSR and TAN)
>    - There is also a hidden unit column. (For species this column is unused)
>
>
> ![screenshot of the hidden columns](images/arcitect-hidden-columns.png)
>
> This is why we use the term **Building Block** instead of **column**; behind the scenes there are actually more columns.
> We hide these columns in ARCitect, because most of the time these columns are not so interesting to humans, but they are
> what makes your ARC machine-readable.
>
> This machine readability is essential for example for ARC searching tools so that you can perform advanced
> searches such as *"Find all ARCs with data of Talinum Fruticosum that performed an RNA-seq assay"* and facilitate re-use
> of data. This will help people find your research, and will help you find data for your next study.
>
{: .comment}


Let's add some more building blocks for other information we have about our plants.
Based on our methods document, we know a few more characteristics of our plants:

- Light intensity of 425 µEinstein
- Plant age of 28 days
- Watering exposure of 12 days drought (for 3 of the samples) or 12 days drought followed by 2 days rewatering (for the other 3 samples)

Notice that the first two of these characteristics include a value (425, 28) and a unit (µEinstein, days).
In the next step we will show you how to deal with units in ARCitect

> <hands-on-title>Add building blocks with units</hands-on-title>
>
> 1. **Add a new building block** for plant age
>    - Select **Characteristic** from the dropdown
>    - Type **Plant Age**
>    - Select the DPBO term *Plant age*
>    - In the results, you can always click on the {% icon dropdown %} (dropdown) icon to get more information about the term
>    ![screenshot of ontology term information](images/arcitect-plant-age.png)
>
> 2. Before you add the column to the table, you can also **set the unit** for the column
>    - Click the button labelled **Unit**
>    - Search for **Day**
>    - Find an ontology term with a good description
>    ![screenshot of window adding building block with unit selected](images/arcitect-building-block-add-with-unit.png)
>    - Click **Add Column**
>    - Notice that the column is now already filled with the unit
>
> 3. The column will appear to the right of the column you were last on
>    - If you want your column in a different place, move it now in the same way we did before (right click the column)
>
> 4. Add the value **28** to the age column for all 6 samples
>    - You can see that your chosen unit is automatically added to the value you provide in the cell
>    - remember, you can fill in one cell, then use the "Fill Column" option from the right-click menu to apply the same value to all rows.
>
>    ![screenshot of our annotation table after adding plant age column](images/arcitect-annotation-table-age-added.png)
{: .hands_on}


> <comment-title>Characteristic vs Factor vs Parameter vs Component</comment-title>
>
> When adding building blocks, you may have noticed that a distinction is made between *characteristic*, *parameter*,
> *factor* and *component*. But when to use which?
>
> Have a look at this [Annotation Principles Guide](https://nfdi4plants.github.io/nfdi4plants.knowledgebase/docs/guides/isa_AnnotationPrinciples.html)
> For a more precise description, but in short:
>
> - **Characteristics columns** describe inherent properties of the source material, e.g., a certain strain or ecotype,
>   but also the temperature an organism was exposed to.
> - **Factor columns** describe independent variables that determine the specific output of your experiment when process and analysis were identical.
>   Most of the time, Factors are the most important building blocks for downstream computational analysis.
> - **Parameter columns** describe steps in your experimental workflow, e.g., the temperature or extraction buffer used for your assay.
>   Multiple Parameter columns form a protocol.
> - **Component columns** are used to list anything physical of a protocol, e.g. instrument names, software names, or reagents names.
>
{: .comment}

Keeping this in mind, let's add more building blocks

> <hands-on-title>Add more buiding blocks</hands-on-title>
>
> 1. Add a building block for **light intensity exposure** (PECO:0007224)
>    - Add it as a **Parameter column**
>    - Set the unit to *microeinstein per square meter per second* (UO:0000160)
>    - Fill the column with a value of 425 for all samples
>
> 2. Add a building block for **Watering exposure** (PECO:0007383)
>    - Add it as **Factor column**
>    - set the value to "12 days drought" for the drought samples (`DB_097,DB_099,DB_103`)
>    - set the value to "12 days drought + 2 days rewatered"* for the other 3 samples
>
>    ![screenshot of the annotation table at this point](images/arcitect-annotation-table-complete.png)
>
{: .hands-on}

As a final step, we can associate our protocol document (`plant_material.txt`) with our annotation table.

> <hands-on-title>Add more buiding blocks</hands-on-title>
>
> 1. Add a new building block
>    - Add it as a **ProtocolREF** column (under *More*)
>    - Click **Add column**
> 2. For the value, set the file name `plant_material.txt` for all columns
>    - ARCitect will know to look in the protocols folder for this file
>    ![screenshot of the ProtocolREF column](images/arcitect-annotation-table-protocolref.png)
{: .hands_on}

Well done! You have added your first annotation table! You can add as many columns as you need in your annotation tables. The aim is always to capture as much information as you can in this way.

While we succeeded in creating our annotation table, it was quite a bit of work adding in our building blocks one by one. Luckily, there are a lot of templates available that can make your life easier! In the next section, we will show you how to do this.

## Add assay annotation

Building annotation tables from scratch can be a bit time consuming, especially if we have to repeat it for every ARC we create
Therefore, DataPLANT has also a way to share templates for annotation tables. For example, the metadata for RNA sequencing will be very similar for
everybody. So instead of creating this annoation table from scratch, we can load a pre-existing template. This is just a starting point, we can still
add and remove building blocks as needed, but it will save us a lot of time.

### Create RNA-Seq assay annotation tables from template

> <hands-on-title></hands-on-title>
>
> 1. Click on the RNA-seq assay in your ARC
> 2. Add a new annotation table
>    - {% icon plus %} (plus) icon at the bottom
> 3. **Rename** the table to a better name
>    - e.g. "RNA extraction"
> 4. Choose **Start from an existing template!** in the center of the screen
> 5. Search for **RNA**
>    - The results will show a few options from the official DataPLANT templates
>    - You can also search other community templates by changing the **Select community** dropdown to **All** (or a specific community)
>    ![template search results for term RNA](images/arcitect-template-rnaseq.png)
>
> 6. Choose the **RNASeq extraction** template
>    - Click **Select** to confirm
>    - This will give you a preview of the building blocks in the template
>    - Click **Add template**
> 7. Add 6 rows to the table
>    ![empty template file after importing](images/arcitect-template-rna-extraction.png)
> 8. Try to add RNA extraction information to this table
>    - Open the `RNA_extraction.txt` file in your protocols folder to view the information to add.
>
> > <question-title></question-title>
> >
> > 1. How does your final annotation table look?
> >
> > > <solution-title>Some Hints</solution-title>
> > > 1. The **biosource amount** column does not have a unit set yet. To do so now, please:
> > >    - Search for *milligram* in the column
> > >    - Right-click on the term, and select `as unit cell`.
> > >    - Choose "Fill column" from the right-click menu to apply this to all rows
> > > 2. Set the unit for **buffer volume** column to **microliter** in the same way
> > {: .solution}
> >
> > > <solution-title> Full solution </solution-title>
> > > Below are screenshots of a filled annotation table. Note that there are often multiple ways to fill an ARC.
> > >
> > > ![first half of the final annotation table](images/arcitect-annotation-table-rna-extraction-solution-part1.png)
> > > ![second half of the final annotation table](images/arcitect-annotation-table-rna-extraction-solution-part2.png)
> > >
> > {: .solution}
> {: .question}
{: .hands_on}


As always, let's save and sync our ARC

> <hands-on-title>Save & Sync</hands-on-title>
>
> 1. Save your ARC locally
> 2. Sync your ARC to DataHUB
>    - Provide a good commit message
{: .hands_on}

# Adding computational workflows to your ARC

We have now added raw data, protocols, and annotation tables to our ARC. We can publish this *Data ARC* as-is,
but ARCs can also support addition of analysis workflows and results.

ARCs offer integrations for [CWL](https://www.commonwl.org/) (Common Workflow Language) workflows, but this is out of scope for this tutorial.
You can read more about adding CWL workflows to your ARC in the [DataPLANT knowledgebase](https://nfdi4plants.org/nfdi4plants.knowledgebase/docs/guides/ComputationalWorkflows/cwl_examples.html)

If you do not have CWL workflows, you can put any other scripts or code you used in the `workflows` folder, and
any analysis results in the `runs` folder. This will not be structured or statndardized, but better than nothing.

# Publishing your ARC

Once you have finished your ARC, you can choose to publish it. It will then become public for anybody to find and re-use.
Before you can do this, your ARC must pass some basic validation. This process checks if your ARC is in valid ARC format,
and if any files you reference are really included, etc.


## ARC validation

By default, the basic ARC validation is enabled in DataHUB

> <hands-on-title></hands-on-title>
>
> 1. Open your ARC repository on DataHUB. Tip: a quick way to get there is
>    - click on **DataHUB sync** in the ARCitect menu
>    - click on the **globe icon** to open the DataHUB repository in the browser
> 2. On the right of your DataHUB ARC repository, you should see a badge labelled **pipeline**
>    with either a green *passed* label or a red *failed* label
>
>    ![example of ARC pipeline badge](images/datahub-pipeline-passed.png)
>
> 3. You can click on this badge to get more information about the tests
>    - you can also click on the green checkmark or red cross next to your latest commit see validation pipeline details
>    - if your tests are failing, this can give you more information about what is wrong
>    - if it is green, you usually don't need to look at the test details.
>    - We will cover testing in more detail in a different tutorial, for now, just know that it exists.
{: .hands_on}

This test performs some basic checks, but in order to publish our ARC, we need additional validation. This is not
enabled by default.

## Enabling Invenio testing

Publishing of ARCs uses [Invenio](https://inveniosoftware.org/). Before you can publish your ARC,
we must enable the *Invenio validation package*.

> <hands-on-title> Invenio Validation </hands-on-title>
>
> 1. Click on **Validation** in the ARCitect menu panel
> 2. Under **Custom Validation Packages**, select **Invenio**
>    ![Screenshot of menu adding invenio](images/arcitect-validation-invenio-add.png)
> 3. Click **Save**
> 4. Sync the change to DataHUB
>    - Remember to **commit** first
> 5. Open your ARC repository on DataHUB again
>    - You should see additional badges
>    - Note that it may take a few minutes for these badges to show up
>    ![screenshot of invenio badges on DataHUB](images/datahub-validation-invenio-badges.png)
{: .hands_on}


## Publishing Process

Once all of the test pass, you can start the submission process for publishing your ARC.

> <hands-on-title></hands-on-title>
>
> 1. Open your ARC repository on DataHUB
> 2. Click on the invenio testing badge
>    - This can only be done if the badge is green
> 3. This will open a screen where you can preview the basic ARC information
>    ![Screenshot of invenio publishing page](images/datahub-invenio-publish.png)
> 4. We will not publish this training ARC, but for your real ARC you would now check the boxes to agree to the terms, and then hit the **Publish** button.
{: .hands_on}


## Where is your ARC published to?

After publishing, your ARC will be public, and automatically included in several locations.

### ARChive

The DataPLANT [ARChive](https://archive.nfdi4plants.org/) will list all published ARCs

For example [this ARC](https://archive.nfdi4plants.org/records/9e5jx-75d83)

![Example of a published ARC on ARChive](images/archive-example.png)

You see this looks a bit like a publication. Notice the DOI on the right. Every ARC will receive a DOI.
You can use this DOI in your publication to point readers to your ARC.

> <comment-title> Updating a published ARC </comment-title>
> Note that you can still update your ARC after publishing, and you can publish a newer version. You will
> receive a new DOI for the updated ARC, and the original version of your ARC will remain accessible
> with the original DOI.
{: .comment}

### ARC Registry

The [ARC registry](https://arcregistry.nfdi4plants.org/isasearch) is another place where your ARC will be listed.
The ARC registry can search across different institution's DataHUBs, and supports advanced searching of ARC metadata.

![screenshot of ARC registry homepage](images/arcregistry.png)

Many of these ARC searching and aggregation portals are still under active development, and may change/improve a lot in
the coming years.

# Next steps on your ARCification journey

You should now know the basics needed to create an ARC for your own data using the ARCitect tool.

For support during the process of creating an ARC for your own project(s), have a look at:
- The [DataPLANT knowledgebase](https://nfdi4plants.org/nfdi4plants.knowledgebase/).
- The [DataPLANT helpdesk](https://helpdesk.nfdi4plants.org/?topic=Documentation_KnowledgeBase)



# Conclusions

Congrats! You have now created your first ARC using ARCitect, you synced it to DataHUB, and saw the procedure for publishing
your ARC.


